Article Per Year (5 Year)
p-Index From 2021 - 2026
0.23
P-Index
This Author published in this journals
All Journal Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi)
Selfina Pare
Universitas Musamus
Computer Science & IT Engineering

Published : 1 Documents Claim Missing Document
Claim Missing Document
Check
Articles

Found 1 Documents
Search

 1 
Reliable Indonesian Academic Service Chatbot Using Retrieval-Augmented Generation and Adaptive Refusal Jarot Budiasto; Tatik Melinda Tallulembang; Selfina Pare; Muhammad Hasbi
Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi) Vol 10 No 5 (2026): October 2026 (In progress)
Publisher : Ikatan Ahli Informatika Indonesia (IAII)

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.29207/resti.v10i5.8018

Abstract

Academic service information is often stored in lengthy, formal, and distributed university documents, making accurate access difficult. Generative chatbots can improve access but may answer unsupported questions unless retrieval and refusal are controlled. This study proposes a reliability-oriented Retrieval-Augmented Generation (RAG) chatbot for Indonesian academic services integrating BGE-M3, FAISS, adaptive refusal thresholding, rule-based guardrails, Qwen2.5, and retry-on-refusal. A controlled experiment used 12 official documents and 160 Indonesian questions across eight categories. The final configuration achieved Precision@5 of 0.7250, Recall@5 of 0.9125, MRR of 0.8940, nDCG@5 of 0.8888, and mean semantic similarity of 0.8516. All 16 expected-refusal questions were refused and none of the 144 answerable questions was falsely refused, yielding observed refusal accuracy of 1.0000 and a false refusal rate of 0.0000. Retry-on-refusal reduced the false refusal rate from 0.0556 to 0.0000 and increased answerable-only similarity from 0.8150 to 0.8392. These results provide initial evidence that combining retrieval confidence, guardrails, and retry supports more reliable behavior within the controlled benchmark. Because the refusal subset was small and institution-specific, larger evaluations using naturally occurring student queries are required before inferring operational reliability.
Co-Authors JAROT BUDIASTO Muhammad Hasbi Tatik Melinda Tallulembang