Sistemasi: Jurnal Sistem Informasi
Vol 15, No 6 (2026): Sistemasi: Jurnal Sistem Informasi

Accuracy Evaluation of the Naïve Bayes Classifier for Sentiment Classification of Diploma Authenticity Issues using Orange

Eunike Loise Laapen (Universitas Kristen Satya Wacana)
Indrastanti Ratna Widiasari (Universitas Kristen Satya Wacana)



Article Info

Publish Date
30 Jun 2026

Abstract

The rapid growth of digital text data has increased the demand for effective methods to extract meaningful information, particularly for understanding public opinion. Sentiment analysis is widely used to classify opinions into positive, negative, and neutral categories. However, challenges such as linguistic ambiguity, subjectivity, and class imbalance often degrade classification performance. This study aims to evaluate the performance of the Naïve Bayes algorithm for sentiment classification on the issue of diploma authenticity using a publicly available dataset, while examining the impact of data distribution on model performance. A quantitative experimental approach was employed using an original dataset of 1,014 instances and an oversampled dataset of 1,767 instances. The data were processed through preprocessing, Bag-of-Words feature extraction, and sentiment classification using Orange Data Mining with 10-fold cross-validation. Model performance was evaluated using accuracy and the Area Under the Receiver Operating Characteristic Curve (AUC). The results indicate that the Naïve Bayes model achieved an accuracy of 37.2% and an AUC of 0.704 on the original imbalanced dataset, reflecting relatively poor classification performance. After applying oversampling to balance the class distribution, the model's accuracy increased substantially to 82.1%, while the AUC improved to 0.970. These findings demonstrate that class distribution has a significant impact on the performance of the Naïve Bayes algorithm in sentiment classification and highlight the importance of addressing class imbalance to achieve more reliable classification results.

Copyrights © 2026






Journal Info

Abbrev

stmsi

Publisher

Subject

Computer Science & IT Electrical & Electronics Engineering

Description

Sistemasi adalah nama terbitan jurnal ilmiah dalam bidang ilmu sains komputer program studi Sistem Informasi Universitas Islam Indragiri, Tembilahan Riau. Jurnal Sistemasi Terbit 3x setahun yaitu bulan Januari, Mei dan September,Focus dan Scope Umum dari Sistemasi yaitu Bidang Sistem Informasi, ...