Retrieval Augmented Generation-Based Chatbot for Prospective and Current University Students

Hartono, Luluk Setiawati
Setiawan, Esther Irawati
Singh, Vrijraj

International Journal of Engineering, Science and Information Technology
Vol 5, No 3 (2025)

Universities utilize chatbots as assistants for users, especially prospective and current students, to access information and answer questions with relevant answers. This study introduces a new approach to an open-source model-based QA system using Gemma2-2b-it by combining Retrieval Augmented Generation (RAG) and Fine-tuning (FT) techniques. Previously, some studies have focused on only one approach, but this study will combine and compare both methods separately. Raw conversation data from WhatsApp, the main university website, and university PDF documents are used. The Retrieval Augmented Generation Assessment (RAGAS) framework will be used to evaluate the performance of the RAG model. In contrast, precision, recall, and similarity are used to assess the comparative performance of RAG and fine-tuning. The results of the RAGAS show that RAG using the base model is better than RAG using a fine-tuned model, which has 0.78 faithfulness, 0.64 answer relevancy, 0.81 context precision, and 0.68 context recall, so the overall RAGAS Score is 0.72. The comparison of precision and recall of fine-tuning are higher than those of using RAG, but the similarity score is not much different. Furthermore, the potential improvement for RAG of this study can be increased by adding a reranking process in the retrieved context, and fine-tuning of the embedding model can also be added to increase the retrieval process's performance. In addition, further experiments on various datasets and the challenge of overfitting in fine-tuning must be overcome so that the model can also perform better generalization.