This research aims to develop an effective fraud detection model in banking transactions using the rule-based model (RBM) approach and the isolation forest (IF) machine learning algorithm. Based on data from the Ministry of Communication and Information Technology, there were more than 405,000 online fraud cases during the 2019–2022 period, indicating the need for a reliable fraud detection system to protect customers. The research method involves collecting banking transaction data for four months through channels such as ATM, internet banking, and mobile banking. The RBM model was used as an initial approach, detecting suspicious transaction patterns based on defined rules. However, it has limitations in detecting transactions that are not defined in the rules. To complement this shortcoming, this research implemented IF, an effective unsupervised learning model for detecting anomalies using the isolation tree (iTree) technique to identify suspicious transactions. The results showed that the IF model could detect anomalous patterns not covered by RBM, thereby improving the accuracy of fraud transaction identification. The precision data of 99% indicates that the model’s predictions of anomalies are indeed anomalies, while a recall value of 1.0 shows that the model successfully identified all anomalies in the dataset. In conclusion, the combination of RBM and IF provides a comprehensive approach to fraud detection in the banking sector. IF’s ability to detect anomalies more dynamically and accurately can reduce fraud losses in the industry.
Copyrights © 2025