JURNAL INSTEK (Informatika Sains dan Teknologi)
Vol 11 No 2 (2026): OCTOBER

Position and Language as Diagnostic Variables: A Controlled Invariance Test of Sentiment Classifiers on Indonesian-English Code-Switched Reviews

Lie, Kennedy (Unknown)
Santoso, Handri (Unknown)



Article Info

Publish Date
30 Oct 2026

Abstract

Sentiment classifiers are usually judged by accuracy or F1-score, which cannot show whether a model reads the sentiment cue or relies on the surrounding text. Code-switched text, which mixes languages within a sentence, makes this harder to judge. This study proposes a controlled behavioural invariance test in which one sentiment cue is moved across three positions (beginning, middle, end) and written in two languages (Indonesian, English) in 272 reviews, giving 1,632 minimal-pair variants. Logistic Regression, Linear SVC, BERT, mBERT, XLM-RoBERTa and IndoBERT were compared using Prediction Consistency Rate, Prediction Flip Rate and McNemar tests with Holm correction, alongside standard metrics. The two TF-IDF models were largely insensitive to cue position and language but still flipped 5.1% and 1.5% of predictions, and deleting the frequent token “aplikasi” changed their output margins nearly as much as deleting the sentiment cue. BERT and mBERT made more errors on Indonesian than on English cues (adjusted p = 0.014 and 0.031). XLM-RoBERTa was unaffected by the token on the reported split and, over five repeated splits, had the highest mean accuracy and PCR and the lowest PFR. On this dataset, strong standard metrics did not guarantee consistent predictions when the cue was moved or translated.

Copyrights © 2026






Journal Info

Abbrev

instek

Publisher

Subject

Computer Science & IT Control & Systems Engineering Electrical & Electronics Engineering Engineering

Description

The Scope topics include, but are not limited to : Agent System and Multi-Agent Systems Analysis & Design of Information System Artificial Intelligence Big Data and Data Mining Cloud & Grid Computing Computer Vision Cryptography Decision Support System DNA Computing E-Government E-Business ...