Komparasi ChatGPT-4o, Claude Sonnet, dan Gemini Pro dalam Deteksi Phishing Berbahasa Indonesia Menggunakan Pendekatan Zero-Shot Prompting

Nur Rizky Aditya (Program Studi Teknologi Informasi, Fakultas Ilmu Komputer, Universitas Mulia)
Nur Hidayat (Program Studi Teknologi Informasi, Fakultas Ilmu Komputer, Universitas Mulia)
Yustian Servanda (Program Studi Teknologi Informasi, Fakultas Ilmu Komputer, Universitas Mulia)

Jurnal Sains dan Teknologi
Vol. 6 No. 2 (2026): Mei - Agustus

Phishing attacks targeting Indonesian internet users continue to grow in volume and sophistication, exploiting local slang, domestic financial institution names, and cultural context, yet the unique linguistic and cultural characteristics of Bahasa Indonesia remain underexplored in the scientific literature on AI-based cybersecurity. This study aims to comparatively evaluate three state-of-the-art Large Language Models (LLMs)—ChatGPT-4o (OpenAI), Claude Sonnet (Anthropic), and Gemini Pro (Google)—for their effectiveness in detecting Indonesian-language phishing threats. Using a zero-shot prompting approach, a curated dataset of 100 samples (46 phishing, 54 legitimate) representing real-world Indonesian phishing modus operandi across banking, e-commerce, government, and social media contexts was evaluated under identical experimental conditions. Inter-annotator reliability was confirmed using Cohen’s Kappa (κ = 0.858), and model performance was measured using Accuracy, Precision, Recall, and F1-score. Results show that Gemini Pro achieved the best overall performance (F1 = 0.967, Accuracy = 97.00%), followed by ChatGPT-4o (F1 = 0.930, Precision = 100%) and Claude Sonnet (F1 = 0.909, Accuracy = 92.00%). All three models demonstrated strong capability as zero-shot phishing detectors, highlighting the potential of generative LLMs as accessible, training-free components for AI-based cybersecurity systems in Indonesian-language environments, and offering practical guidance for selecting LLM tools for phishing detection in Indonesia.