cover
Article Per Year (5 Year)
All
Home Page
OAI Link
Editorial Team
Contact
Reviewer
Google Scholar
Contact Name
Rian Ferdian
Contact Email
rian.ferdian@fti.unand.ac.id
Phone
-
Journal Mail Official
jitce@fti.unand.ac.id
Editorial Address
-
Location
Kota padang,
Sumatera barat
INDONESIA
Journal of Information Technology and Computer Engineering
Published by Universitas Andalas
ISSN : 25991663     EISSN : -     DOI : -
Core Subject : Science, Engineering,
Computer Science & IT Control & Systems Engineering Electrical & Electronics Engineering Engineering
Journal of Information Technology and Computer Engineering (JITCE) is a scholarly periodical. JITCE will publish research papers, technical papers, conceptual papers, and case study reports. This journal is organized by Computer System Department at Universitas Andalas, Padang, West Sumatra, Indonesia.
Arjuna Subject : -
Articles 1 Documents
 1 
Search results for , issue "Vol. 9 No. 2 (2025): Journal of Information Technology and Computer Engineering" : 1 Documents clear
Evaluating IndoGPT for Legal Queries: A Benchmark Against GPT-4 Models irawan, ade; Palupi, Ade Cahyaning
JITCE (Journal of Information Technology and Computer Engineering) Vol. 9 No. 2 (2025): Journal of Information Technology and Computer Engineering
Publisher : Universitas Andalas

Show Abstract | Download Original | Original Source | Check in Google Scholar

Abstract

This study evaluates a chatbot developed with the Large Language Model (LLM) IndoGPT, focusing on its use of Retrieval-Augmented Generation (RAG) to answer questions about university regulations from legal PDF documents in the Indonesian Language. IndoGPT's performance is benchmarked against the more advanced models, GPT-4 and GPT-4o. The chatbot combines RAG techniques with the LangChain framework, and its effectiveness is assessed using the Retrieval-Augmented Generation Assessment (RAGAS) framework. The dataset includes publicly available legal documents from Universitas Pertamina, with test data created by the authors. IndoGPT consistently underperforms compared to GPT-4 and GPT-4o. GPT-4 achieves superior metrics with Context Precision at 0.9027, Context Recall at 0.8693, Faithfulness at 0.8486, and Answer Relevancy at 0.8142. Similarly, GPT-4o delivers strong results with Context Precision at 0.8896, Context Recall at 0.8594, Faithfulness at 0.8804, and Answer Relevancy at 0.8773. In contrast, IndoGPT shows significant deficiencies, with much lower scores: Context Precision at 0.6687, Context Recall at 0.5711, Faithfulness at 0.0738, and Answer Relevancy at 0.1628. These findings highlight IndoGPT's substantial limitations, especially when compared to GPT-4 and GPT-4o, which excel in providing accurate, contextually relevant answers. The GPT-4-based chatbot demonstrates strong capabilities in understanding user queries and delivering accurate responses while effectively reducing hallucinations through the RAG technique.

Page 1 of 1 | Total Record : 1

 1 

Filter by Year

2025 2025


Reset
Filter By Issues
All Issue Vol. 9 No. 2 (2025): Journal of Information Technology and Computer Engineering Vol. 9 No. 1 (2025) Vol. 8 No. 2 (2024) Vol 8 No 2 (2024): Journal of Information Technology and Computer Engineering Vol. 8 No. 1 (2024) Vol 7 No 01 (2023): Journal of Information Technology and Computer Engineering Vol. 7 No. 01 (2023) Vol. 7 No. 2 (2023) Vol 6 No 02 (2022): Journal of Information Technology and Computer Engineering Vol. 6 No. 02 (2022) Vol. 6 No. 01 (2022) Vol 6 No 01 (2022): Journal of Information Technology and Computer Engineering Vol 5 No 02 (2021): Journal of Information Technology and Computer Engineering Vol. 5 No. 02 (2021) Vol. 5 No. 01 (2021) Vol 5 No 01 (2021): Journal of Information Technology and Computer Engineering Vol 4 No 02 (2020): Journal of Information Technology and Computer Engineering Vol. 4 No. 02 (2020) Vol. 4 No. 01 (2020) Vol 4 No 01 (2020): Journal of Information Technology and Computer Engineering Vol 3 No 02 (2019): Journal of Information Technology and Computer Engineering Vol. 3 No. 02 (2019) Vol 3 No 01 (2019): Journal of Information Technology and Computer Engineering Vol. 3 No. 01 (2019) Vol 2 No 02 (2018): Journal of Information Technology and Computer Engineering Vol. 2 No. 02 (2018) Vol 2 No 01 (2018): Journal of Information Technology and Computer Engineering Vol. 2 No. 01 (2018) Vol 1 No 01 (2017): Journal of Information Technology and Computer Engineering Vol. 1 No. 01 (2017) More Issue