Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi)
Vol 10 No 3 (2026): June 2026

Detecting AI-Generated Text with Fine-Tuned RoBERTa: A Cross-Validated Study with Interpretability Analysis

Yuhefizar Yuhefizar (Politeknik Negeri Padang)
Ronal Watrianthos (Politeknik Negeri Padang)
Dony Marzuki (Politeknik Negeri Padang)



Article Info

Publish Date
21 Jun 2026

Abstract

The rapid proliferation of large language models capable of generating fluent and contextually coherent text has made reliable authorship attribution both a practical and ethical necessity. Outputs from systems such as GPT-4, LLaMA 2, and Claude are increasingly indistinguishable from human-authored prose, yet current detection methodologies remain insufficient to meet this challenge. This study aims to develop and evaluate a fine-tuned RoBERTa-base binary classifier for distinguishing AI-generated text from human-authored text under stratified cross-validation conditions; assess the model's robustness to a known length-based confound through a controlled ablation study; and provide interpretable explanations of the classifier's decisions through dual token-level attribution using Gradient × Input Saliency and Attention Rollout. The model is trained and evaluated via stratified five-fold cross-validation on a balanced corpus of 6,069 samples (3,069 AI-generated and 3,000 human-authored) and benchmarked against four established baselines spanning classical, neural, and zero-shot paradigms. The classifier achieves a mean accuracy of 99.93%, a weighted F1-score of 0.9993, and a mean AUC-ROC of 1.0000 across all five folds. A length-balanced ablation produces an accuracy decline of only 0.11 percentage points, providing empirical evidence that the model has learned genuine semantic and stylistic distinctions rather than superficial length-based cues. Gradient × Input Saliency maps indicate that AI-generated texts are characterised by technically precise, domain-specific vocabulary, whereas human-authored texts are identified through discourse markers and rhetorically structured expression, with both attribution methods converging on the same high-saliency tokens.

Copyrights © 2026






Journal Info

Abbrev

RESTI

Publisher

Subject

Computer Science & IT Engineering

Description

Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi) dimaksudkan sebagai media kajian ilmiah hasil penelitian, pemikiran dan kajian analisis-kritis mengenai penelitian Rekayasa Sistem, Teknik Informatika/Teknologi Informasi, Manajemen Informatika dan Sistem Informasi. Sebagai bagian dari semangat ...