<![CDATA[This study aims to evaluate the functionality and answer quality of a web-based learning application that uses Generative Artificial Intelligence (GenAI) for the Pancasila and Civic Education (PPKN) course. The primary focus of this research lies in the system evaluation process, while the application development was carried out solely as a means of generating test data. The system was evaluated in two stages: functional testing using the black-box testing method and answer quality assessment using the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) method. Black-box testing was conducted to ensure that all core system features operated according to specifications. The results of the black-box testing showed a 100% success rate across all test scenarios. Furthermore, answer quality evaluation was performed on 50 test data pairs consisting of GenAI-generated answers and reference texts (gold standards) prepared by PPKN lecturers using the ROUGE method. The evaluation results showed an average F1-score of 97% on the ROUGE-1, ROUGE-2, and ROUGE-L metrics. A total of 49 out of 50 answers were categorized as “Very Good” (? 0.75), while 1 answer was categorized as “Good.” These findings indicate that the application is capable of generating answers with a very high level of textual similarity to academic references. This study contributes to filling the gap in empirical evidence and provides a standardized evaluation benchmark for web-based GenAI applications in education, while also offering an evaluation approach that integrates system functional testing and ROUGE-based answer quality measurement. However, this evaluation is still limited to linguistic aspects based on n-grams and does not yet fully represent semantic depth.]]>
Copyrights © 2026
