Performance Analysis of BERT and CLIP Models in Multimodal Sentiment Classification of Short Video Content - Very Setiawan; Endang Anggiratih; Najwa Eka Putriningsih; Jonathan Eldo Kusuma (Politeknik Negeri Batam, 2026)
The rapid growth of short video platforms such as YouTube Shorts has increased the need for effective sentiment analysis methods capable of capturing public opinion in multimodal content. This study analyzes and compares the effectiveness of unimodal and multimodal approaches for sentiment classification of Indonesian short videos, focusing on IndoBERT for text-based modeling and CLIP for multimodal integration. The main objective is to investigate whether incorporating visual information alongside textual data can improve sentiment classification performance compared to a text-only approach. The dataset consists of 1,128 Indonesian short videos collected from YouTube Shorts. Audio data are transcribed into text using Automatic Speech Recognition (ASR), while visual information is represented using video thumbnails. Sentiment labels are automatically categorized into three classes (positive, neutral, and negative) using a pre-trained IndoBERT model. In the training phase, the unimodal approach relies solely on textual features extracted by IndoBERT, whereas the multimodal approach integrates textual and visual features using CLIP through feature-level fusion. Model performance is evaluated using accuracy, precision, recall, F1-score, and computational time analysis. The experimental results show that the unimodal text-based model outperforms the multimodal model, achieving higher accuracy (86% vs 82%) and better overall evaluation metrics. IndoBERT also demonstrates better convergence behavior compared to English BERT, with training accuracy increasing from 0.76 to 0.86 and validation accuracy from 0.77 to 0.88, along with lower loss values. In contrast, English BERT achieves lower performance, with training accuracy rising from 0.72 to 0.79 and validation accuracy from 0.73 to 0.80. Furthermore, the unimodal approach requires significantly less computation time (18 minutes compared to 35 minutes). These findings indicate that textual information plays a dominant role in sentiment expression in Indonesian short video content, while visual features increase computational complexity without significant performance gains.