PERBANDINGAN KESESUAIAN PENDAPAT MANUSIA DAN AI (CHATGPT, GEMINI, DEEPSEEK) MENGGUNAKAN PENDEKATAN GROUND TRUTH

Kelvin Kelvin (Universitas Mikroskil)
Sunaryo Winardi (Universitas Mikroskil)
Handoko Handoko (Universitas Mikroskil)
Erwin Setiawan Panjaitan (Universitas Mikroskil)
Rivaldi Lubis (Universitas Mikroskil)

JOURNAL OF SCIENCE AND SOCIAL RESEARCH
Vol. 9 No. 3 (2026): June 2026

Abstract: The rapid advancement of Large Language Models (LLMs) has significantly improved sentiment analysis capabilities. However, the extent to which these models produce sentiment classifications consistent with human judgment remains an important research question. This study aims to evaluate and compare the agreement of ChatGPT, Gemini, and DeepSeek with human-generated ground truth in sentiment analysis. A total of 1,497 product reviews were collected from the Sephora Products and Skincare Reviews dataset. Three independent annotators labeled each review as positive, neutral, or negative to establish the ground truth. The annotation reliability achieved a Fleiss' Kappa coefficient of 0.7053, indicating substantial agreement and confirming the reliability of the ground truth for evaluation purposes. Subsequently, the three LLMs performed sentiment classification using an Expectation–Role–Action (ERA) prompting strategy. Model performance was assessed using Cohen's Kappa to measure agreement with the ground truth and Spearman's rank correlation to evaluate the consistency of sentiment rankings. The results show that DeepSeek achieved the highest performance, with an average Cohen's Kappa of 0.753 and an average Spearman's correlation coefficient of 0.860, followed by Gemini (κ = 0.662; ρ = 0.834), while ChatGPT demonstrated the lowest agreement (κ = 0.223; ρ = 0.367). These findings indicate that the three LLMs exhibit significantly different levels of agreement with human judgment, with DeepSeek producing sentiment classifications that most closely align with the established ground truth.  Keywords: Large Language Models, Sentiment Analysis, Ground Truth, Cohen's Kappa, Spearman's Rank Correlation.   Abstrak: Perkembangan Large Language Models (LLMs) telah meningkatkan kemampuan analisis sentimen berbasis kecerdasan buatan. Namun, tingkat kesesuaian hasil klasifikasi sentimen yang dihasilkan oleh berbagai LLM terhadap penilaian manusia masih memerlukan evaluasi yang komprehensif. Penelitian ini bertujuan membandingkan tingkat kesesuaian hasil analisis sentimen ChatGPT, Gemini, dan DeepSeek terhadap ground truth yang diperoleh melalui anotasi manusia. Penelitian menggunakan 1.497 ulasan produk dari dataset Sephora Products and Skincare Reviews. Sebanyak tiga anotator independen melakukan pelabelan sentimen ke dalam kategori positif, netral, dan negatif untuk membentuk ground truth. Hasil pengujian reliabilitas menunjukkan nilai Fleiss' Kappa sebesar 0,7053, yang mengindikasikan tingkat kesepakatan Substantial Agreement, sehingga ground truth layak digunakan sebagai acuan evaluasi. Selanjutnya, ketiga model LLM melakukan klasifikasi sentimen menggunakan prompt berbasis Expectation–Role–Action (ERA). Tingkat kesesuaian hasil klasifikasi dievaluasi menggunakan Cohen's Kappa, sedangkan konsistensi hubungan dengan ground truth dianalisis menggunakan korelasi Spearman. Hasil penelitian menunjukkan bahwa DeepSeek memberikan performa terbaik dengan rata-rata Cohen's Kappa sebesar 0,753 dan rata-rata korelasi Spearman sebesar 0,860, diikuti oleh Gemini (κ = 0,662; ρ = 0,834), sedangkan ChatGPT memperoleh tingkat kesesuaian terendah (κ = 0,223; ρ = 0,367). Temuan ini menunjukkan bahwa terdapat perbedaan kemampuan interpretasi sentimen antar model LLM, dengan DeepSeek menghasilkan klasifikasi yang paling mendekati penilaian manusia pada dataset yang digunakan. Kata kunci: Large Language Model, Analisis Sentimen, Ground Truth, Cohen's Kappa, Spearman Correlation.