Purpose of the study: This study examines what the content validity index conceals rather than what it reports. Using a two round Delphi validation, it asks how far rater severity varies within one expert panel, how sensitive the consensus conclusion is to the dichotomisation point, and which findings survive across four index specifications. Methodology: The complete Round 2 rating matrix of a two round Delphi validation of a competence framework was re-analysed (20 experts, 169 items, 3,380 ratings). Analyses in base R comprised per rater severity summaries, cut point and threshold sensitivity, a leave one expert out jackknife, four index specifications, pairwise Spearman correlations against a permutation null model, and non-parametric facet comparisons. Main Findings: Expert mean ratings spanned 3.94 to 4.99 and top category use spanned 1.2% to 99.4%, yet the proportion of ratings at or above 4, the only quantity the index uses, spanned just 91.1% to 100%. Omitting any single expert left all 169 items in consensus, whereas restricting relevance to the top category cut qualifying items to 16. Aiken's V and the modified kappa left the conclusion intact. Rank agreement was real but minimal. Novelty/Originality of this study: Validation reports vary the panel and almost never vary the index specification. This study varies both on one dataset, shows that fragility is confined to stricter dichotomisation while indices using the full ordinal scale remain stable, settles by permutation whether weak rank agreement reflects disagreement or attenuation, and derives a diagnostic set requiring no new data. For education researchers and developers of assessment instruments, the practical implication is that a content validity index should be reported together with the rating distribution, per rater summaries and at least one index that uses the full scale before an instrument is used for curriculum, assessment or certification decisions.
Copyrights © 2026