Malware detection is a major challenge in the world of cybersecurity, especially with the increasing complexity and variety of attacks. Traditional approaches are often unable to identify these new threats, making machine learning (ML) an effective solution. The purpose of this study is to compare the performance of three machine learning algorithms, namely Random Forest, XGBoost, and Support Vector Machine (SVM), in detecting malware in the SOMLAP dataset consisting of Windows executable files. Data processing, the use of the SMOTE technique for class imbalance, and assessment using metrics such as accuracy, precision, recall, and F1 score are all part of the research methodology. This study also applies Z-Score Normalization to reduce the influence of extreme values in the data, which helps the model handle data with different scales. The results show that Random Forest has the best performance with an accuracy of 99.16%, followed by XGBoost and SVM. Random Forest excels in the balance between precision, recall, and accuracy, making it the most effective algorithm for detecting malware. This study suggests further algorithm development using ensemble techniques and other optimizations to improve malware detection accuracy in the future.