Claim Missing Document
Check
Articles

Found 1 Documents
Search
Journal : Journal of Applied Data Sciences

Psychometric Validation of an AI-Based Evaluation System for Identifying Discrepancies in Learning Processes Suyasa, P. Wayan Arta; Pujawan, I Gusti Ngurah; Divayana, Dewa Gede Hendra; Budhyani, I Dewa Ayu Made; Sugiarta, I Made; Candiasa, I Made
Journal of Applied Data Sciences Vol 7, No 2: May 2026
Publisher : Bright Publisher

Show Abstract | Download Original | Original Source | Check in Google Scholar | DOI: 10.47738/jads.v7i2.1168

Abstract

This research advances the field of educational evaluation by designing and psychometrically validating an artificial intelligence (AI)- based diagnostic tool to detect discrepancies in university learning processes. The main novelty is the integration of the Provus Discrepancy Model combined with a forward-chaining inference engine. This research aims to transform evaluation from an administrative activity to an ongoing process of improvement. The tool was developed and validated through a sequential mixed-methods approach with 400 participants from 3 state universities and 8 evaluation experts. Results from the study provide evidence that the validated system created a substantial range of psychometric characteristics. These psychometric characteristics include strong content validity (SD-CVI/Ave = 0.94); high internal consistency and reliability (Cronbach's α = 0.94); solid construct validity as demonstrated through Confirmatory Factor Analysis (CFA) (CFI = 0.94; RMSEA = 0.054) and a substantial range of predictive analytics (diagnostic learning analytics), which the AI learning analytics engine evaluated learning discrepancies with a 92.4% diagnostic accuracy (47.4% more accurate than manual evaluation methods). The system's validated usefulness is demonstrated through high system usability (SUS = 88.2); high practical utility (85% total score on the Pragmatic Utility Assessment); significant utility (real-world) practical utility (detected 45 discrepancy patterns), cost efficiency (73% cost and 67% analysis time compared to traditional methods), and a range of analytics (predictive and learning discrepancy analytics). The significant contribution of this study is the development of the world's first integrated AI evaluation system that meets high methodological and psychometric standards, along with a set of real-time diagnostic analytics. Ultimately, this study developed the first truly integrated, novel paradigm evaluation system that combined the historically established evaluation construct and mechanisms with the most advanced AI capabilities, providing educators and institutions with evaluation tools to deliver data-driven pedagogical strategies and interventions in higher education. 
Co-Authors ., I G. P. Habibi Mudi Sucipta ., I Gede Karya Danu Palguna ., I Nengah Adi Mahendra ., I Nyoman Mei Adi Shantiyana ., Kadek Eva Virga Yuni ., Made Dwi Savitri ., Ni Ketut Arum ., NI WAYAN SANJIWANI UTAMI ., Putu Wisma Artha Diputra A.A. I Ngurah Maya Pradnyani . Budayana, I Made Dewa Gede Hendra Divayana, Dewa Gede Hendra Dewi, Putu Kartika Dharma Tari, I Dewa Ayu Eka Purba Divayana, Dewa Hendra Dr. I Made Sugiarta, M.Si. . Drs.Djoko Waluyo,M.Sc . Durasa, Helfra Evi Marliantini, Made Fitrianiayuningsih, Niputu Bela Gede Suweken Gusti Ayu Mahayukti Helfra Durasa I Dewa Ayu Made Budhyani I Dewa Gede Agung Pratama Putra . I Dewa Gede Agung Pratama Putra ., I Dewa Gede Agung Pratama Putra I G. P. Habibi Mudi Sucipta . I Gede Karya Danu Palguna . I Gede Ratnaya I Gusti Bagus Gending . I Gusti Nyoman Yudi Hartawan I Gusti Nyoman Yudi Hartawan, S.Si . I Kadek Agus Prabawan Saputra I Komang Wahyu Wiguna I Made Ardana I Made Candiasa I Made Suarsana I Made Suarsana I made Yudi Suardi I Nengah Adi Mahendra . I Nyoman Jampel I Nyoman Mei Adi Shantiyana . I Nyoman Sukajaya I PUTU PASEK SURYAWAN I Putu Wisna Ariawan I Wayan Amperawan I Wayan Puja Astawa I Wayan Widiana Ida Ayu Dewi Wulandari Ida Ayu Dewi Wulandari Ida Ayu Putu Mahayani Ida Bagus Putrayasa Jayanta, Kadek Kadek Eva Virga Yuni . Kartini, Siluh Made Surya Komang Setemen Made Dwi Savitri . Made Juniantari, Made Mahayukti, Gst. Ayu Mardika Pranata, Kadek N. N. T Tesashogi Ni Kadek Santiyani Ni Ketut Arum . Ni Ketut Widiartini Ni Made Arri Sudani . Ni Made Arri Sudani ., Ni Made Arri Sudani Ni Made Sri Mertasari Ni Nyoman Parwati Ni Wayan Eka Sentana Wati . Ni Wayan Eka Sentana Wati ., Ni Wayan Eka Sentana Wati Ni Wayan Sanjiwani Utami . Niputu Bela Fitrianiayuningsih Nursuhaida, Muhammad Alwan Oktaviani, Ni Putu Ari P. WAYAN ARTA SUYASA Pramono, Campa Febriana Sanggra Prof. Dr.I Nengah Suparta,M.Si . Putu Wika Purwani . Putu Wika Purwani ., Putu Wika Purwani Putu Wisma Artha Diputra . Santiyani, Ni Kadek Sariyasa . Sepiani, Ni Komang Sinurat, Frisillia Br Suardi, I made Yudi Suardika, I Made Dandi Tesashogi, N. N. T Wargina, I Made Wibawa, Nyoman Arda Widhiasih, I Dewa Ayu Gede Widhiasih, I Dewa Ayu Gede Widiastuti, Ni Putu Kusuma Wulandari, Ida Ayu Dewi Yasa, I Putu Arya Swankastika Yudi Hartawan , I Gusti Nyoman