Addressing the Black-Box Challenge: Confidence Index Approach to Grading of Open-Ended Assessments in Accounting Education

Husain

Journal of Education Technology
Vol. 9 No. 1 (2025): February

Traditional grading systems have long been a common practice in the world of cross-disciplinary education, but they face challenges such as limited scalability, inconsistencies in assessment, and a lack of personalized feedback. This condition encourages the emergence of an alternative in the form of an automatic assessment system based on artificial intelligence (AI). Although promising, this approach is not without problems, especially regarding the transparency and reliability of assessments which are often referred to as the "black box" effect. This study aims to address this effect through the integration of the trust index into the AI assessment system, in order to provide more interpreted assessment results in the context of accounting education. This research is a quantitative study with a supervised machine learning approach. The AI model is trained to evaluate students' open-ended answers on several accounting subjects. Data was collected from students' answer scripts and compared to the results of human assessments. The analysis was conducted to see the relationship between AI scores and human assessment scores based on trust and experience levels. The results showed that the use of trust indexes significantly improved the consistency and reliability of assessment results, with a positive correlation between high trust scores and the suitability of AI assessment results with human assessments. However, variations were also found in some subjects, suggesting that the effectiveness of AI is influenced by the specific characteristics of the material. The conclusions of this study show that the integration of trust indexes into AI scoring systems can improve the transparency and accuracy of assessments. However, a hybrid approach that combines AI and human appraiser engagement is still necessary to ensure efficiency, educational integrity, and fairness in the assessment process. The implications of this study emphasize the need for adaptive automatic appraisal development strategies, especially in the context of accounting education. However, the results of the study cannot be generalized widely because they are limited to the context of accounting education only.