<![CDATA[Retrieval-Augmented Generation (RAG) incorporates externally retrieved information as contextual knowledge to improve the quality of responses generated by language models. However, a RAG system's effectiveness depends heavily on the retrieval method's ability to identify and rank relevant documents. This study compares BM25, Multilingual-E5, and a hybrid approach based on Reciprocal Rank Fusion (RRF) using a product and service catalog corpus consisting of 145 documents and 40 benchmark queries. The evaluation was conducted in two stages: retrieval performance was assessed using Hit@1, Hit@3, Hit@5, and MRR@5, while RAG answer quality was evaluated using Faithfulness, Answer Relevancy, and Answer Accuracy. The results show that BM25 achieved the highest Hit@1 score (0.775) and MRR@5 (0.8279), whereas Multilingual-E5 achieved the highest Hit@5 score (0.950). For answer quality, Multilingual-E5 obtained the highest Faithfulness score (0.9667), while the hybrid RRF approach achieved the highest Answer Relevancy score (0.8451); both achieved an Answer Accuracy score (0.7938). Statistical tests revealed no significant differences among the retrieval methods. Answer Relevancy and Answer Accuracy were descriptively lower when the target document was absent from the Top-5 context, whereas the target's exact position within ranks 1-5 showed no significant correlation with answer quality. A context-size robustness analysis further showed descriptive improvements in Answer Relevancy and Answer Accuracy from Top-1 to Top-3 and Top-5, although none of the nine context-size effects remained statistically significant after Holm correction. Under a criterion emphasizing Top-5 target coverage and downstream answer quality, Multilingual-E5 showed a favorable operational profile under the evaluated configuration.]]>
Copyrights © 2026
