IJID (International Journal on Informatics for Development)
Vol. 15 No. 1 (2026): IJID JUNE

Development and Post-Hoc Evaluation of a Transformer-Based Mental Health Chatbot Using the ESPRT Framework

David S (Informatika UIN Sunan Kalijaga Yogyakarta)
Muhammad Syafiq Akmal (Informatika UIN Sunan Kalijaga Yogyakarta)
Aulia Faqih Rifa’i (Informatika UIN Sunan Kalijaga Yogyakarta)



Article Info

Publish Date
02 Jul 2026

Abstract

Mental health disorders are an increasing global concern, yet access to professional care is still limited due to resource shortages and persistent social stigma. Artificial intelligence–based chatbots have therefore emerged as a potential tool for providing early support. However, developing systems that are both computationally efficient and psychologically appropriate remains challenging. This study developed a chatbot using the FLAN-T5-base architecture, fine-tuned on the MentalChat16K dataset, comparing Full Fine-Tuning (FFT) and Low-Rank Adaptation (LoRA). Quantitative evaluation yielded objectively low n-gram scores (e.g., ROUGE-L of 17.81 for FFT and 17.17 for LoRA). Although these scores may suggest weak generative performance, they were largely affected by the brevity penalty because the models generated much shorter responses than the reference answers. Thus, to further evaluate response quality, an LLM-as-a-judge qualitative assessment was conducted, confirming that both models produced safe, contextually relevant, and reasonably empathetic responses. Notably, LoRA achieved comparable qualitative performance while updating only ~0.35% of parameters, reducing GPU memory usage by 21% and training time by about 24%. Interpreted through the Eco-Socio-Psycho-Religio-Technic (ESPRT) post-hoc discussion framework, the findings demonstrate that LoRA is a practical and sustainable approach for mental health support systems in resource-limited environments.

Copyrights © 2026






Journal Info

Abbrev

ijid

Publisher

Subject

Computer Science & IT

Description

One important point in the accreditation of higher education study programs is the availability of a journal that holds the results of research of many investigators. Since the year 2012, Informatics Department has English language. Journal called IJID International Journal on Informatics for ...