Andy Dahnial Pratama Mz
Master’s Program in Indonesian Language Education, Universitas Tadulako, Palu

Published : 1 Documents Claim Missing Document
Claim Missing Document
Check
Articles

Found 1 Documents
Search

Developing an Artificial Intelligence (AI)-Assisted Instrument for Assessing the Short Story Writing Skills of Grade XI Students at Senior High School Andy Dahnial Pratama Mz; Ida Nuraeni; Sukma
IDEAS: Journal on English Language Teaching and Learning, Linguistics and Literature Vol. 14 No. 2 (2026): IDEAS: Journal on English Language Teaching and Learning, Linguistics and Lite
Publisher : Universitas Islam Negeri Palopo

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.24256/ideas.v14i2.12887

Abstract

Assessing short story writing is prone to subjectivity when it is not supported by a valid and reliable instrument, while the use of artificial intelligence (AI) in drafting assessment instruments has rarely been documented. This study aims to describe the development process and the quality of an AI-assisted instrument for assessing the short story writing skills of grade XI students at SMA Al-Azhar Mandiri Palu. It followed the 4D development model (define, design, develop, disseminate), with AI (Claude Opus 5.5) serving as a drafting aid rather than as a rater of students’ work. Data were collected through interviews, observation, document analysis, validation by five experts, the scoring of 30 short stories by two teachers, and a practicality questionnaire, and were analyzed using feasibility percentages, the intraclass correlation coefficient (ICC), Cronbach’s alpha, percentage agreement, and practicality percentages. The AI generated 32 rubric descriptors from a single prompt. After the researcher’s review, 12 descriptors were accepted unchanged, 17 were revised to make them observable and consistent with the quantitative requirement, and 3 were rejected because they relied on impressionistic criteria. The 12 descriptors accepted unchanged were later revised on the language validator’s advice, so none of the AI-generated descriptors was used unchanged in the final product. The instrument obtained a feasibility percentage of 92.63% (very valid), a single-measures ICC of 0.758 (good; 95% CI 0.541–0.878), and a practicality percentage of 98.33% (very practical). Cronbach’s alpha was low (−0.224 and 0.251) because scores clustered at the top of the scale, while exact inter-rater agreement reached 94.17%. The instrument is feasible for use, and AI is effective as a drafting aid provided that its output is reviewed by the researcher and validated by experts.