<![CDATA[Machine learning-based Intrusion Detection Systems (IDS) often achieve strong performance on known attack distributions but may degrade when encountering unseen attacks. This study evaluates the robustness of two gradient boosting models, XGBoost and LightGBM, under unseen attack distribution shifts using the ToN-IoT dataset. A controlled attack-exclusion strategy was applied, where selected attack types were excluded from training and evaluated only during testing. Unlike conventional unseen-attack evaluations that primarily report performance degradation, this study further investigates attack-specific degradation through feature-level distribution similarity analysis. Feature importance analysis was used to identify key traffic features, while Jensen-Shannon Divergence (JS Divergence) quantified distribution similarity between attack and normal traffic. Model robustness was assessed using Recall, Macro-F1, PR-AUC, False Negatives (FN), and False Negative Rate (FNR) across five random seeds. The results show that performance degradation varied substantially across attack types. Both models maintained near-baseline performance for unseen Scanning attacks, whereas unseen DDoS and especially MITM attacks produced larger increases in FNR and greater performance degradation. Correlation analysis indicated that the proto feature exhibited the strongest relationship between distribution similarity and detection errors, with lower JS Divergence generally associated with higher FNR. These findings suggest that robustness degradation depends not only on attack novelty but also on the similarity between attack and normal traffic distributions, providing additional insight into attack-specific robustness behavior in IDS models.]]>
Copyrights © 2026
