Indonesian Journal of Education Research (IJoER)
Vol. 7 No. 5 (2026): October

What the Content Validity Index Does Not Show: Rater Severity and Cut-Point Sensitivity in a Two-Round Delphi Validation and Implications for Instrument Validation in Education

Nguyen Dinh Tho (Thu Dau Mot University)
Nguyen Phuong Chi (Hanoi National University of Education)
Pham Tho Hoan (Hanoi National University of Education)



Article Info

Publish Date
28 Oct 2026

Abstract

Purpose of the study: This study examines what the content validity index conceals rather than what it reports. Using a two round Delphi validation, it asks how far rater severity varies within one expert panel, how sensitive the consensus conclusion is to the dichotomisation point, and which findings survive across four index specifications. Methodology: The complete Round 2 rating matrix of a two round Delphi validation of a competence framework was re-analysed (20 experts, 169 items, 3,380 ratings). Analyses in base R comprised per rater severity summaries, cut point and threshold sensitivity, a leave one expert out jackknife, four index specifications, pairwise Spearman correlations against a permutation null model, and non-parametric facet comparisons. Main Findings: Expert mean ratings spanned 3.94 to 4.99 and top category use spanned 1.2% to 99.4%, yet the proportion of ratings at or above 4, the only quantity the index uses, spanned just 91.1% to 100%. Omitting any single expert left all 169 items in consensus, whereas restricting relevance to the top category cut qualifying items to 16. Aiken's V and the modified kappa left the conclusion intact. Rank agreement was real but minimal. Novelty/Originality of this study: Validation reports vary the panel and almost never vary the index specification. This study varies both on one dataset, shows that fragility is confined to stricter dichotomisation while indices using the full ordinal scale remain stable, settles by permutation whether weak rank agreement reflects disagreement or attenuation, and derives a diagnostic set requiring no new data. For education researchers and developers of assessment instruments, the practical implication is that a content validity index should be reported together with the rating distribution, per rater summaries and at least one index that uses the full scale before an instrument is used for curriculum, assessment or certification decisions.

Copyrights © 2026






Journal Info

Abbrev

IJoER

Publisher

Subject

Education Languange, Linguistic, Communication & Media Mathematics Social Sciences Other

Description

Indonesian Journal of Education Research (IJoER) is a scientific journal published by Cahaya Ilmu Cendekia Publisher. The IJoER aims to provide a forum for scholarly understanding of the field of education and plays an important role in promoting the process that accumulated knowledge, values, and ...