Ikhlasul Amal
IPB University

Published : 1 Documents Claim Missing Document
Claim Missing Document
Check
Articles

Found 1 Documents
Search

Design and Evaluation of a Route-Aware, Evidence-Grounded RAG Chatbot for Accessing Village Monographs and Census Data within Data Desa Presisi Ikhlasul Amal; Irman Hermadi; Yani Nurhadryani
Journal of Information System and Informatics Vol 8 No 5 (2026): October
Publisher : Asosiasi Doktor Sistem Informasi Indonesia

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.63158/journalisi.v8i5.1856

Abstract

Data Desa Presisi integrates narrative village monographs and structured census records, creating challenges for contextual, numerical, and cross-source information access. This study designed and evaluated a locally deployed, route-aware, evidence-grounded retrieval-augmented generation (RAG) chatbot that combines deterministic DuckDB queries, document retrieval, provenance, and privacy-aware routing. Evaluation used 100 ground-truth questions from Setu and Gunung Putri. Eighty-four database, monograph, and mixed-evidence questions were assessed using RAGAS and ROUGE-L, whereas 16 conversational and policy questions were evaluated through deterministic checks. Elasticsearch at a temperature of 0.1 produced the highest composite score (0.592018), although post hoc Wilcoxon tests found no statistically significant superiority over alternative configurations. The selected configuration achieved a 98.0% functional pass rate and a 97.92% numeric-match rate. Public-mode usability evaluation involving 21 village officials and staff yielded 96.03% effectiveness. Scalability testing covered pipeline construction and retrieval across 30 census inputs and 28 monographs, but did not assess concurrent users or full end-to-end production workloads. These findings provide preliminary evidence that route-aware RAG can improve transparent, evidence-grounded access to heterogeneous village data while preserving deterministic handling of numerical queries and privacy-sensitive information.