<![CDATA[The development of GenAI has encouraged the use of automated essay scoring technology through various platforms, one of which is the ESAO (Essay Analytic Online) application. Although this LLM-based system is capable of automatically generating assessment feedback narratives, standardizing evaluation methods to measure the reliability of these texts still faces significant challenges. This study aims to test the suitability of the Bilingual Evaluation Understudy (BLEU) and Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metrics as instruments to measure the extratextual performance of the ESAO application. The research method was carried out by comparing feedback texts from ESAO with authentic lecturer assessment drafts on three different characteristics of the exam material: dataset condition analysis, descriptive statistics, and correlation and regression. The test results showed an average value of the BLEU metric of 0.0522 and ROUGE of 0.1255. This study revealed that low scores do not represent a functional failure of the ESAO application, but rather indicate fundamental limitations and shortcomings in using rigid lexical metrics (word-based metrics) in assessing dynamic generative texts. The BLEU and ROUGE metrics rely heavily on rigid n-gram overlap, thus failing to capture the semantic similarity, academic reasoning context, and linguistic variation generated by ESAO. This study concludes that traditional evaluation metrics such as BLEU and ROUGE are inaccurate and incompatible as a single benchmark for Generative AI performance in the context of educational assessment, necessitating a transition to semantic-based metrics in the future.]]>
Copyrights © 2026
