Development and Post-Hoc Evaluation of a Transformer-Based Mental Health Chatbot Using the ESPRT Framework - David S; Muhammad Syafiq Akmal; Aulia Faqih Rifa’i (Faculty of Science and Technology, Universitas Islam Negeri (UIN) Sunan Kalijaga Yogyakarta, 2026)
Mental health disorders are an increasing global concern, yet access to professional care is still limited due to resource shortages and persistent social stigma. Artificial intelligence–based chatbots have therefore emerged as a potential tool for providing early support. However, developing systems that are both computationally efficient and psychologically appropriate remains challenging. This study developed a chatbot using the FLAN-T5-base architecture, fine-tuned on the MentalChat16K dataset, comparing Full Fine-Tuning (FFT) and Low-Rank Adaptation (LoRA). Quantitative evaluation yielded objectively low n-gram scores (e.g., ROUGE-L of 17.81 for FFT and 17.17 for LoRA). Although these scores may suggest weak generative performance, they were largely affected by the brevity penalty because the models generated much shorter responses than the reference answers. Thus, to further evaluate response quality, an LLM-as-a-judge qualitative assessment was conducted, confirming that both models produced safe, contextually relevant, and reasonably empathetic responses. Notably, LoRA achieved comparable qualitative performance while updating only 0.35% of parameters, reducing GPU memory usage by 21% and training time by about 24%. Interpreted through the Eco-Socio-Psycho-Religio-Technic (ESPRT) post-hoc discussion framework, the findings demonstrate that LoRA is a practical and sustainable approach for mental health support systems in resource-limited environments.