Optimizing Decision Tree and Random Forest with Grid Search and SMOTE for Malware Classification on IoT Network Traffic - Siroj, Muhammad Nurus; Zyen, Akhmad Khanif; Wibowo, Gentur Wahyu Nyipto (Politeknik Negeri Batam, 2025)
The rapid growth of the Internet of Things (IoT) has increased the risk of malware attacks, posing serious threats especially to micro, small, and medium enterprises (MSMEs) that often lack sufficient cybersecurity resources. This study aims to optimize Decision Tree (DT) and Random Forest (RF) classifiers using Grid Search, while addressing the class imbalance problem through the Synthetic Minority Oversampling Technique (SMOTE). The Security Attacks Malware IoT Networks dataset with five classes (Benign, Malware, DDoS, Brute Force, Scanning) was used and divided into training and testing sets with stratified 80:20 split. Experimental results show that DT achieved 67.3% accuracy with a macro F1-score of 42.9%, while RF achieved 70.7% accuracy but a very low macro F1-score of 21.4%, indicating bias toward the majority class despite balancing. Boosting methods provided stronger baselines, with XGBoost reaching 87.0% accuracy and 66.7% F1-score, while LightGBM achieved 85.6% accuracy and 64.4% F1-score. ROC curves and confusion matrices confirmed that boosting methods were more balanced in recognizing minority classes. In terms of efficiency, DT required the shortest training time (8 seconds), while LightGBM provided the best trade-off between accuracy and computational cost (26 seconds). Paired t-tests further confirmed that performance differences between DT and RF were not significant, while boosting methods significantly outperformed RF. Overall, optimizing DT and RF with Grid Search and SMOTE enhances their performance, but boosting methods remain more robust for malware detection in IoT traffic. These findings provide practical insights for MSMEs in balancing accuracy and efficiency when deploying intrusion detection systems.