Sentiment classifiers are usually judged by accuracy or F1-score, which cannot show whether a model reads the sentiment cue or relies on the surrounding text. Code-switched text, which mixes languages within a sentence, makes this harder to judge. This study proposes a controlled behavioural invariance test in which one sentiment cue is moved across three positions (beginning, middle, end) and written in two languages (Indonesian, English) in 272 reviews, giving 1,632 minimal-pair variants. Logistic Regression, Linear SVC, BERT, mBERT, XLM-RoBERTa and IndoBERT were compared using Prediction Consistency Rate, Prediction Flip Rate and McNemar tests with Holm correction, alongside standard metrics. The two TF-IDF models were largely insensitive to cue position and language but still flipped 5.1% and 1.5% of predictions, and deleting the frequent token “aplikasi” changed their output margins nearly as much as deleting the sentiment cue. BERT and mBERT made more errors on Indonesian than on English cues (adjusted p = 0.014 and 0.031). XLM-RoBERTa was unaffected by the token on the reported split and, over five repeated splits, had the highest mean accuracy and PCR and the lowest PFR. On this dataset, strong standard metrics did not guarantee consistent predictions when the cue was moved or translated.
Copyrights © 2026