<![CDATA[The advancement of artificial intelligence has driven LLM-based chatbot implementation in education, particularly for New Student Admission (PMB) services. This research analyzes and evaluates an LLM-based PMB chatbot using Retrieval-Augmented Generation (RAG) at Politeknik Krakatau. The system integrates three LLM models (GPT-4 Turbo, Mistral Devstral, Xiaomi Mimo-v2) with FAISS and LangChain. Comprehensive evaluation uses eight standard metrics: four for retrieval (Recall@3, Precision@3, MRR, NDCG@3) and four for generation (BERTScore, ROUGE-1, ROUGE-L, METEOR), with 100 questions across six categories. Results show the retrieval component achieves excellent performance with Recall@3 of 1.000 (perfect), MRR of 0.756, and NDCG@3 of 0.864, indicating effective document finding and ranking. For generation, Mistral Devstral demonstrates best performance with BERTScore of 0.755, ROUGE-1 of 0.604, and METEOR of 0.427, followed by GPT-4 Turbo (BERTScore 0.723) and Xiaomi Mimo-v2 (BERTScore 0.718). These comprehensive results enable evidence-based model selection, producing a chatbot delivering accurate, contextually relevant, and consistent responses to prospective students. This directly addresses slow and inefficient admission services by reducing administrative workload through automated, high-quality information provision while improving response speed and reliability. Compared to previous studies, this research provides the most comprehensive evaluation of RAG-based PMB chatbots in Indonesia, with retrieval performance surpassing prior studies and offering actionable insights bridging technical metrics and real-world service improvement in higher education institutions.]]>
Copyrights © 2026
