This study explores the optimization of machine learning algorithms for fraud detection in electronic payment (e-payment) systems. The rapid growth of e-payment platforms has introduced significant challenges in ensuring the security and integrity of financial transactions. Fraud detection plays a pivotal role in mitigating these risks, and the application of machine learning (ML) has emerged as a powerful tool to identify fraudulent activities. This research examines how Data Quality (DQ), Algorithm Selection (AS), and Optimization Techniques (OT) influence Model Performance (MP) and, subsequently, Fraud Detection Effectiveness (FDE). The study utilizes Partial Least Squares Structural Equation Modeling (PLS-SEM) through SmartPLS 3 to analyze the relationships between these variables. The results demonstrate that high Data Quality significantly enhances Model Performance, while Algorithm Selection and Optimization Techniques also contribute positively, albeit to a lesser extent. The findings reveal that Model Performance plays a crucial mediating role between these factors and the effectiveness of fraud detection. Fraud Detection Effectiveness is found to be significantly impacted by Model Performance, suggesting that improving model accuracy and efficiency is essential for better fraud detection outcomes. Reliability and validity tests show strong internal consistency for all constructs, with Cronbach’s Alpha, Composite Reliability, and Average Variance Extracted (AVE) all reaching satisfactory levels. The study highlights the importance of data preprocessing, the careful selection of machine learning models, and optimization techniques in achieving high-performing fraud detection systems. The results provide valuable insights for the development of more robust and scalable fraud detection mechanisms in e-payment systems, contributing to the broader field of machine learning and cybersecurity. Future research could explore advanced techniques like deep learning and blockchain integration for further enhancement of fraud detection systems.