Article Per Year (5 Year)
p-Index From 2021 - 2026
0.23
P-Index
This Author published in this journals
All Journal Journal of Intelligent Computing, Systems, and Applications (JICSA)
Angelina Nadeak
Institut Teknologi Del

Published : 1 Documents Claim Missing Document
Claim Missing Document
Check
Articles

Found 1 Documents
Search

 1 
Evaluation of Gen-AI (LLM) Results on Prompts in the Batak Language Ade Siahaan; Angelina Nadeak; Jeremi Samosir; Jesica Siburian; Boy Hutajulu
Journal of Intelligent Computing, Systems, and Applications (JICSA) Vol. 2 No. 1 (2026): September 2026
Publisher : Penerbit Institut Teknologi Del

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.54074/jicsa.v2i1.45

Abstract

The rapid advancement of Large Language Models (LLMs) hassignificantly enhanced natural language processing capabilities;however, their performance on low-resource regional languagesremains underexplored. The Toba Batak language, a regionallanguage spoken in North Sumatera, Indonesia, faces limitedrepresentation in global AI training datasets, raising concerns aboutlinguistic accuracy and cultural preservation in AI-generatedcontent. This study evaluates the performance of four LargeLanguage Models—ChatGPT, Gemini, Perplexity, and Toba-LLM—in generating responses to prompts written in the Toba Bataklanguage. The evaluation follows the CRISP-DM methodology andemploys a structured dataset consisting of categorized prompts andcorresponding ground truth answers. Model outputs are assessedusing six quantitative metrics: lexical similarity, keyword overlap,length ratio, cosine similarity, cultural relevance, and fluency. Theresults indicate that general-purpose models such as ChatGPT andGemini outperform the local Toba-LLM in terms of semanticsimilarity and fluency, while Toba-LLM demonstrates higher lexicalalignment and response conciseness. However, cultural relevanceremains a challenge across all models, highlighting the difficulty ofcapturing culturally grounded knowledge in low-resource languagesettings. These findings underscore the need for culturally awareevaluation metrics and enriched local language datasets to improvethe applicability of LLMs for regional language preservation and AI-based educational applications.
Co-Authors Ade Siahaan Boy Hutajulu Jeremi Samosir Jesica Siburian