Article Per Year (5 Year)
p-Index From 2020 - 2025
6.356
P-Index
This Author published in this journals
All Journal Jurnal Teknologi dan Manajemen Informatika Sinkron : Jurnal dan Penelitian Teknik Informatika JURNAL LENTERA : Kajian Keagamaan, Keilmuan dan Teknologi JOURNAL OF APPLIED INFORMATICS AND COMPUTING Dinamisia: Jurnal Pengabdian Kepada Masyarakat Jurnal Sisfokom (Sistem Informasi dan Komputer) Technomedia Journal Jurnal Nasional Pendidikan Teknik Informatika (JANAPATI) Capture : Jurnal Seni Media Rekam Building of Informatics, Technology and Science TEPIAN Jurnal Teknik Informatika (JUTIF) Business Economic, Communication, and Social Sciences Journal (BECOSS) Jurnal ABDIMAS (Pengabdian kepada Masyarakat) UBJ Jurnal Pendidikan dan Teknologi Indonesia Jurnal Locus Penelitian dan Pengabdian Eduvest - Journal of Universal Studies Jurnal Indonesia : Manajemen Informatika dan Komunikasi The Indonesian Journal of Computer Science
Santoso, Handri
Unknown Affiliation
Religion Aerospace Engineering Arts Humanities Automotive Engineering Computer Science & IT Control & Systems Engineering Decision Sciences, Operations Research & Management Economics, Econometrics & Finance Education Electrical & Electronics Engineering Engineering Environmental Science Health Professions Industrial & Manufacturing Engineering Languange, Linguistic, Communication & Media Law, Crime, Criminology & Criminal Justice Library & Information Science Mathematics Neuroscience Social Sciences Other

Published : 48 Documents Claim Missing Document
Claim Missing Document
Check
Articles

Found 1 Documents
Search
Journal : Jurnal Sisfokom (Sistem Informasi dan Komputer)

 1 
Comparative Analysis of RAG-Based Open-Source LLMs for Indonesian Banking Customer Service Optimization Using Simulated Data Lijaya, Hendra; Ho, Patricia; Santoso, Handri
Jurnal Sisfokom (Sistem Informasi dan Komputer) Vol. 14 No. 3 (2025): JULY
Publisher : ISB Atma Luhur

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.32736/sisfokom.v14i3.2383

Abstract

In the digital era, banks face challenges in delivering fast, accurate, and efficient customer service, especially for frequently asked simple questions. This study evaluates the effectiveness of three open-source Large Language Models (LLMs), namely Gemma2-9B-Sahabat-AI, Qwen2.5-14B-Instruct, and Mistral-Nemo-Instruct in supporting a Retrieval-Augmented Generation (RAG) question-answering system for the banking sector. Using 12,000 synthetic billing documents indexed with intfloat/multilingual-e5-large-instruct embeddings (1024 dimensions), model performance was assessed via semantic similarity metrics, LLM-as-a-Judge scores (GPT-4o-mini and Gemini 2.0 Flash), and human validation Gemma2-9B-Sahabat-AI achieved the highest semantic similarity score (0.9627), followed by Mistral (0.9614) and Qwen2.5 (0.9284). In LLM-as-a-Judge evaluations, Qwen2.5 ranked highest on GPT-4o-mini (92.2), while Gemma2 led under Gemini 2.0 Flash (88.4). Human evaluators gave perfect scores for factual questions (1–10), but all models struggled with arithmetic in question 13. Gemma2’s average response time was 41 seconds, faster than Qwen2.5’s 72 seconds and Mistral’s 48 seconds, confirming Gemma2’s balanced performance in accuracy, speed, and computational efficiency. These findings underscore the potential of locally operated open-source LLMs for banking applications, ensuring privacy and regulatory compliance. However, limitations include reliance on synthetic data, a narrow question set, and lack of user diversity. Future research should involve broader queries, real user testing, and numeric reasoning modules to ensure robust and scalable deployment in real-world banking customer service environments.
Co-Authors Agus Anwar Akbar, Achmad Fathurrazi Akbariani, Nurhaliza Vania Amanda, Djaja Andrew Andrew, Andrew Andriani, Febri Aprilia, Novia Pramesti Dazki, Erick Devanda, Faustin Eko Hadianto, Eko Firmansah, R Arif Fransisca, Viviana Harlis Setiyowati HARYONO Heni Jusuf Hindarto, Djarot Ho, Patricia Indrajit, Eko ISWAHYUDI ISWAHYUDI Karjadi, Daniel Avian Karjadi, Markus Laurenso, Justin Lijaya, Hendra Lucia Sri Istiyowati Lupitha, Mariska Mareta, Arvin Maria Magdalena Mohammad, Saiful Azhari Nana Sutarna Perdana, Nanda Praharto, Suwandhy Prawira, Khairul Thamrin Prijantono, Sonny Priswanto, Budi Purwadi Purwadi Purwadi Putri, Gladys Rahman, Ben Ramadhoni, Moehammad Richardus Eko Indrajit Rochadiani, Theresia Herlina Ruddin, Isra Sakti, Irwin Sandjaya, Julio Santoso, Billy Cahyo Saputra, Adi Dwifana Sari, Muh Masri Sentosa, Steve Suhandi, Vartin Suherman, Endang Sumargo, Ruly Sze, Edward Tauhid, Tauhid Wedha, Alessandro Enriqco Putra Bayu Wedha, bayu Yasa Widjaja, Herman Wijaya, Oktavianus Hendry