<![CDATA[Comments on TikTok can provide valuable data for analyzing public sentiment toward social and cultural phenomena. However, informal language, multilingual expressions, code-mixing, and limited linguistic resources for regional languages present challenges for sentiment classification. Sound Horeg was selected as the research subject because this social and cultural phenomenon generates diverse public responses on social media, including support and opposition, making it a relevant context for examining public perceptions through sentiment analysis. This study aims to develop a hybrid sentiment analysis pipeline for classifying TikTok comments related to Sound Horeg by integrating XLM-RoBERTa, a Javanese sentiment lexicon, and CatBoost. The methodological novelty of this study lies in integrating the multilingual semantic representations of XLM-RoBERTa with a Javanese lexical resource in the context of Sound Horeg sentiment analysis, while evaluating the contribution of each component through an ablation study. The dataset was collected from five Sound Horeg videos and consisted of 7,330 comments, which were processed through data cleaning, preprocessing, Lexicon-Based sentiment labeling, XLM-RoBERTa embedding extraction, and Random Undersampling. Lexicon-based labeling produced 960 labeled comments, consisting of 559 positive and 401 negative comments, while Random Undersampling produced 802 balanced samples for the main model. The 768-dimensional XLM-RoBERTa embeddings were used as input features for CatBoost. The main model achieved an accuracy of 89.44%, an F1-Macro score of 89.44%, and an ROC-AUC of 90.25%. The ablation study showed that the Lexicon-only configuration achieved an accuracy of 94.79%, followed by XLM-R only at 93.23% and XLM-R + Lexicon at 91.15%. These results indicate that the simpler configurations, particularly Lexicon-only and XLM-R only, achieved higher performance than the main pipeline, suggesting that the hybrid pipeline cannot yet be considered the most optimal configuration for operational implementation. These findings indicate that XLM-RoBERTa can provide effective semantic representations without using skor leksikals as the main features, while the contribution of Javanese vocabulary remains limited because it was detected in only 1.6% of Group A comments and 0.8% of Group B comments. This finding implies that the selection of components in a sentiment analysis system should consider the balance between predictive performance, processing complexity, and the linguistic characteristics of the analyzed data.]]>
Copyrights © 2026
