Studies in Learning and Teaching
Vol. 4 No. 2 (2023): August

Antithesis of Human Rater: Psychometric Responding to Shifts Competency Test Assessment Using Automation (AES System)

Mohammad Idhom (Doctoral Program of Vocational Education, Universitas Negeri Surabaya, Indonesia)
I Gusti Putu Asto Buditjahjanto (Doctoral Program of Vocational Education, Universitas Negeri Surabaya, Indonesia)
Munoto (Doctoral Program of Vocational Education, Universitas Negeri Surabaya, Indonesia)
Trimono (Data Science Study Program, UPN "Veteran" Jawa Timur, Indonesia)
Prismahardi Aji Riyantoko (Data Science Study Program, UPN "Veteran" Jawa Timur, Indonesia)



Article Info

Publish Date
31 Aug 2023

Abstract

This research is part of proof tests to a combination of statistical processing methods, collecting assessment rubrics in vocational education by comparing two systems, automated essay scoring and human rater. It aims to analyze the final assessment score of essays in Akademi Komunitas Negeri (AKN) Pacitan (Pacitan’s State Community College) and Akademi Komunitas Negeri (AKN) Blitar (Blitar’s State Community College) in East Java, Indonesia. The provisional assumption is that the results show an antithesis to the assessment of human feedback with an automated system due to the conversion of scores between the rubric and the algorithm design. As the hypothesis, algorithm-based score conversion affects automated essay scoring and human rater methods, which led to antithesis feedback. The validity and reliability of the measurement maintain the scoring consistency between the two methods and the accuracy of the answers. The novelty of this article is comparing between AES system and Human Rater using statistical methods. The research shows that there is a similar result using the psychometrics approach, which indicates different metaphor expressions and language systems. Thus, the objective of this study is to provide assistance in the advancement of an information technology system that utilizes a scoring mechanism merging computer and human evaluations, employing a psychological approach known as psychometric leads.

Copyrights © 2023






Journal Info

Abbrev

SiLeT

Publisher

Subject

Education Other

Description

SiLeT: Studies in Learning and Teaching is an internationally recognized journal in the field of education (Learning and Teaching) and is published three times a year (in April, August, and December). The aim of this journal is to publish high-quality studies in the areas of learning, teaching, ...