Claim Missing Document
Check
Articles

Found 1 Documents
Search

Multi-Stage Hybrid Retrieval with Neural Re-Ranking and Rule-Based Context Validation for Domain-Specific Retrieval-Augmented Generation Suharyadi; Irwansyah Saputra
Journal of Novel Engineering Science and Technology Vol. 5 No. 03 (2026): In Press - Journal of Novel Engineering Science and Technology
Publisher : The Indonesian Institute of Science and Technology Research

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.56741/jnest.v5i03.2329

Abstract

Retrieval-Augmented Generation (RAG) has emerged as a widely adopted approach for grounding responses generated by large language models (LLMs) in external document corpora. However, conventional single-stage retrieval pipelines frequently fail to surface the most relevant context at top-ranked positions and lack explicit mechanisms to verify evidence adequacy prior to generation. This paper proposes a multi-stage RAG pipeline integrating three sequential components: hybrid retrieval combining BM25 and dense vector search via Reciprocal Rank Fusion (RRF), neural re-ranking using a cross-encoder transformer model, and a rule-based context validation gate that classifies retrieved context into three categories OK, WEAK, or ABSTAIN before answer generation. The system is evaluated on a domain-specific administrative corpus from the Indonesian AID Scholarship (TIAS) program using 50 annotated queries across four question types. Results demonstrate that hybrid retrieval improves recall at rank 10 (Recall@10) from 0.6391 to 0.8097 compared with BM25 alone, while neural re-ranking further increases mean reciprocal rank at rank 10 (MRR@10) from 0.6702 to 0.8040 (p = 0.0043, Wilcoxon signed-rank test). The complete pipeline achieves token-level F1 scores of 0.4429 and 0.4999 for the 4-billion-parameter (4B) and 12-billion-parameter (12B) model configurations, respectively, with a faithfulness score of 1.00. In comparison, the LLM-only baselines obtain token-level F1 scores below 0.09. The context validation mechanism correctly triggers ABSTAIN on 90% of out-of-domain queries, confirming its utility as a reliability control layer. These findings demonstrate that multi-stage retrieval with explicit context validation substantially improves both ranking quality and answer reliability in domain-specific RAG deployments.