Suherwin Suherwin
Universitas Pejuang Republik Indonesia, Indonesia

Published : 1 Documents Claim Missing Document
Claim Missing Document
Check
Articles

Found 1 Documents
Search

Comparing FAQ-Based and Retrieval-Augmented Generation Chatbots for Academic Service Question Answering Suherwin Suherwin; Moh. Zainuddin; Rachmat Rachmat
G-Tech: Jurnal Teknologi Terapan Vol 10 No 3 (2026): G-Tech, Vol. 10 No. 3 July 2026
Publisher : Universitas Islam Raden Rahmat, Malang

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.70609/g-tech.v10i3.10019

Abstract

Academic service units need question-answering systems that respond quickly while remaining aligned with institutional regulations. This study provides a controlled, deployment-oriented comparison of a lightweight FAQ chatbot and a Retrieval-Augmented Generation (RAG) chatbot for academic-service question answering. The novelty lies in evaluating both prototypes against the same version-controlled official-document scope, the same 60-question dataset, category-level service cases, paired statistical tests, inter-rater reliability, and RAGAS retrieval-quality metrics. The questions covered registration, KRS/KHS, schedules, tuition payment, thesis or final project, and administrative letters. Three validators assessed accuracy, relevance, response effectiveness, and response time using official documents as the reference standard. RAG achieved 88.33% accuracy, a mean relevance score of 4.48/5, and 86.67% response effectiveness; FAQ achieved 68.33%, 3.46/5, and 65.00%, respectively. Paired tests confirmed significant advantages for RAG in accuracy, relevance, and effectiveness, while FAQ was significantly faster (0.62 s vs 2.41 s). RAGAS evaluation showed context precision of 0.86, context recall of 0.82, faithfulness of 0.89, and answer relevancy of 0.91. The findings support a cautious hybrid strategy: FAQ for stable repetitive requests and RAG for contextual, document-dependent questions, subject to broader testing with live queries and stronger retrieval baselines.