The Indonesian Journal of Computer Science
Vol. 15 No. 4 (2026): The Indonesian Journal of Computer Science

TrustPhish-AI: Safety-Gated Source-Aware Evaluation of Three-Class Phishing Detection Under Email-to-SMS Shift

Akam Aziz (Department of Computer Science, University of Raparin, Kurdistan Region, Iraq.)



Article Info

Publish Date
08 Aug 2025

Abstract

Source shift can make phishing detectors appear stronger while concealing unsafe class-specific errors. This study evaluated a three-class workflow (Legitimate, Suspicious, and Phishing) using source-aware governance, controlled augmentation, family-disjoint diagnostics, and pre-specified safety gates. The frozen training set contained 3,732 records from four source families. Evaluation used a locked 584-record validation set, a 587-record internal test, a 788-record SMS compound test, and a 200-record SpamAssassin family-disjoint diagnostic. Classical word/character TF-IDF logistic regression with cell-balanced augmentation achieved internal macro-F1 of 0.9633 and compound macro-F1 of 0.5461. DistilRoBERTa augmentation improved compound macro-F1 by 0.1524 versus its reference (95% CI 0.1143-0.1914), but reduced phishing recall by 0.2374 and failed the severe-error gate. A hierarchical transformer repair achieved internal macro-F1 of 0.9668 and compound macro-F1 of 0.5119; its compound gain was 0.1774 (95% CI 0.1410-0.2152), while phishing recall remained 0.2323 below the reference. Governance review included 492 dual-human consensus records, eight independently adjudicated disagreements, and protected-overlap screening. Candidate replacement sources were excluded under provenance, licensing, channel, privacy, taxonomy, or overlap criteria, so no records were added after the training freeze. The results show that source-aware augmentation can improve targeted and aggregate metrics, but aggregate gains are insufficient when phishing-recall and severe-error safety criteria fail. The study supports a governance and evaluation contribution, not global source resistance or autonomous deployment.

Copyrights © 2026






Journal Info

Abbrev

ijcs

Publisher

Subject

Computer Science & IT Electrical & Electronics Engineering Engineering

Description

The Indonesian Journal of Computer Science (IJCS) is a bimonthly peer-reviewed journal published by AI Society and STMIK Indonesia. IJCS editions will be published at the end of February, April, June, August, October and December. The scope of IJCS includes general computer science, information ...