<![CDATA[The rapid advancement of Large Language Models (LLMs) hassignificantly enhanced natural language processing capabilities;however, their performance on low-resource regional languagesremains underexplored. The Toba Batak language, a regionallanguage spoken in North Sumatera, Indonesia, faces limitedrepresentation in global AI training datasets, raising concerns aboutlinguistic accuracy and cultural preservation in AI-generatedcontent. This study evaluates the performance of four LargeLanguage Models—ChatGPT, Gemini, Perplexity, and Toba-LLM—in generating responses to prompts written in the Toba Bataklanguage. The evaluation follows the CRISP-DM methodology andemploys a structured dataset consisting of categorized prompts andcorresponding ground truth answers. Model outputs are assessedusing six quantitative metrics: lexical similarity, keyword overlap,length ratio, cosine similarity, cultural relevance, and fluency. Theresults indicate that general-purpose models such as ChatGPT andGemini outperform the local Toba-LLM in terms of semanticsimilarity and fluency, while Toba-LLM demonstrates higher lexicalalignment and response conciseness. However, cultural relevanceremains a challenge across all models, highlighting the difficulty ofcapturing culturally grounded knowledge in low-resource languagesettings. These findings underscore the need for culturally awareevaluation metrics and enriched local language datasets to improvethe applicability of LLMs for regional language preservation and AI-based educational applications.]]>
Copyrights © 2026
