Refania Meilani Firdos
Universitas Pelita Bangsa, Indonesia

Published : 1 Documents Claim Missing Document
Claim Missing Document
Check
Articles

Found 1 Documents
Search

Evaluating Informatics Engineering Learning Outcomes Using Weighted Rubrics to Compare Teacher and LLM Scores Nurhadi surojudin; Antika Zahrotul Kamalia; Sufajar Butsianto; Refania Meilani Firdos; Bagus Tri Ardiawan
Brilliance: Research of Artificial Intelligence Vol. 6 No. 4 (2026): Brilliance: Research of Artificial Intelligence, Article Research November 2026
Publisher : Yayasan Cita Cendekiawan Al Khwarizmi

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.47709/brilliance.v6i4.9858

Abstract

The increasing use of large language models (LLMs) in educational assessment highlights the need to examine scoring consistency between automated and teacher evaluations. This study analyzes assessment-score consistency and compares teacher and LLM evaluations using a weighted-rubric approach. A descriptive-analytical method was applied to the HARMOGEN-R dataset. First, 32,805 evaluation records were examined to assess consistency between recorded total scores and aggregated criterion scores. Second, 345 paired teacher–LLM responses were analyzed across five criteria using equal weights of 0.20. Evaluation metrics included mean absolute error (MAE), root mean squared error (RMSE), mean score difference, Pearson correlation, and Spearman rank correlation. The results showed that 32,618 evaluations (99.43%) had matching recorded totals and summed criterion scores. In the paired analysis, the mean teacher score was 52.583, compared with 49.996 for LLM scores. The MAE was 4.168, the RMSE was 9.328, and the mean difference (LLM minus teacher) was ?2.587. Pearson and Spearman correlations were 0.9781 and 0.9863, respectively (both p < 0.001). Despite these strong positive associations, score differences indicate that correlation does not necessarily establish agreement or interchangeability. The weighted-rubric approach provides a transparent method for score aggregation and identifying assessment discrepancies. However, equal weights were used as a baseline, and the criteria were not validated against official Course Learning Outcomes (CLOs) or Program Learning Outcomes (PLOs). Future research should investigate alternative weighting schemes, validate learning-outcome mapping, and assess score agreement across broader educational contexts.