Machine learning models for cybersecurity taxonomy classification are frequently trained on relational identity features (such as CWE, Tactic, and Technique identifiers) that may encode dataset-specific associations rather than transferable attack semantics, yet this risk is rarely assessed alongside label formation and missing-value handling in a single, integrated design. This study evaluates how feature representation, CAPEC label formation strategy, and imputation method jointly affect the classification of CAPEC attack patterns using XGBoost on the VULDAT dataset (610 CVE samples, 21 candidate features). Eighteen experimental combinations, crossing three feature scenarios, three imputation methods, and two label formation strategies, were evaluated using five-fold stratified group cross-validation, dummy-classifier and label-shuffle sanity checks, gain-based feature importance, out-of-fold error analysis, and the Wilcoxon signed-rank test. Retaining relational identity features yielded 0.9398-0.9418 mean accuracy and 0.8793-0.8816 F1-Macro, whereas keeping only structural relational counts caused accuracy to collapse to 0.6136 (F1-Macro 0.4110). Relational identity features accounted for 65.2% of gain-based importance, indicating that reported performance depends substantially on deterministic taxonomy identifiers rather than solely on relational complexity. Fixed first-listed CAPEC labeling consistently outperformed frequency-based labeling, while the imputation method had a negligible effect. These findings offer a reusable, leakage-aware evaluation protocol for CVE-CWE-CAPEC-ATT&CK classification research.
Copyrights © 2026