Jurnal Al-Fawa'id : Jurnal Agama dan Bahasa
Vol 16 No 2 (2026): September

An Argument-Based Validation Framework for Automated Scoring of Higher-Order Thinking in Arabic Essays

Rozaanah, Rozaanah (Unknown)
Riyadi, Slamet (Unknown)



Article Info

Publish Date
30 Sep 2026

Abstract

Automated essay scoring (AES) for Arabic has advanced quickly, but most studies report agreement statistics without stating what an automated score is meant to show or which uses it can support. The gap is sharper for higher-order thinking skills (HOTS), where good answers legitimately diverge. This article develops an argument-based validation framework for Taḥkīm, a semantic-similarity engine designed to score Arabic essays written by Indonesian undergraduates at the analysing, evaluating, and creating levels of the revised Bloom taxonomy. Using a conceptual design approach, the study specifies the intended interpretation and uses of Taḥkīm scores, derives an interpretation-use argument from Kane's framework and the Williamson-Xi-Breyer criteria, and synthesizes Arabic AES research to identify threats that arise in Arabic. The framework comprises four inferences (scoring, generalization, extrapolation, and decision), each with a warrant, testable assumptions, required evidence, and a decision rule fixed before data collection. It identifies eight threats, among them orthographic normalization, colloquial vocabulary, incomplete anchor coverage for divergent creating-level responses, and the confounding of reasoning with language proficiency, and pairs each with a design response and an empirical check. A staged evaluation plan sets the minimum evidence for each permitted use, from formative feedback to first-pass scoring under lecturer review, and excludes unsupervised grading until all four inferences hold. The framework gives Arabic departments in Islamic higher education a procedure for stating and testing claims about automated scores before adopting them, and it is the protocol for the empirical validation of Taḥkīm.

Copyrights © 2026






Journal Info

Abbrev

Alfawaid

Publisher

Subject

Religion Languange, Linguistic, Communication & Media

Description

Jurnal al-Fawa’id : Jurnal Agama dan Bahasa (P-ISSN: 2088-9593 dan E-ISSN: 2774-3748) adalah jurnal yang diterbitkan oleh Sekolah Tinggi Agama Islam Ali bin Abi Thalib Surabaya. Sirkulasi penerbitan jurnal ini adalah 2 (dua) kali dalam setahun, yaitu di bulan maret dan september. Artikel yang ...