Analysis of Computerized Adaptive Test to Reveal Misconceptions and Science Literacy in Science Courses - Muhammad Azzarkasyi; Ani Rusilowati; Endang Susilaningsih; Ibrahim; Mohd Isha Awang (Faculty of Mathematics and Natural Sciences, Universitas Negeri Yogyakarta, 2026)
The development of diagnostic instruments that accurately capable in assessing interdisciplinary science literacy and identifying misconceptions represents a strategic imperative in higher education, particularly within integrated science courses that demand concept transfer across physics, chemistry, biology, and earth sciences. The study advanced both theoretical and methodological frontiers by developing and psychometrically validating a Computerized Adaptive Test (CAT) instrument, which is grounded in a Four-Tier Diagnostic Test framework and designed to profile students’ science literacy and misconception patterns in integrated science contexts. The study employed a quantitative instrument development design with 150 students of science education from three universities in Aceh, Indonesia. The study moved beyond Classical Test Theory by utilizing Item Response Theory (IRT), specifically the Rasch model and two-parameter logistic (2PL) model to evaluate item parameters essential for CAT calibration. The findings demonstrated strong psychometric properties, with Rasch infit and outfit statistics within acceptable ranges confirming unidimensional, while item difficulty parameters spanned from 2.1 to 1.8 logits, providing adequate ability continuum coverage. Critically, domain-specific analysis revealed that items requiring cross-disciplinary concept transfer, particularly those integrating physics and biology, exhibited significantly higher discrimination parameters than items confined to isolated disciplinary content, offering a novel theoretical insight that science literacy is inherently an integrative construct rather than a sum of disciplinary knowledge components. Methodologically, this study advances CAT development by demonstrating that rigorous IRT-based calibration and iterative quality control are essential for ensuring measurement accuracy. The identification of invalid items underscores that item attrition is a necessary feature of responsible test development. Generally, these findings contribute to the broader movement toward adaptive, personalised assessment in higher education, providing a replicable model for researchers and practitioners seeking to leverage CAT technologies to enhance diagnostic precision and support targeted remediation in interdisciplinary science education worldwide.