<![CDATA[Abstract. The rapid adoption of Generative Artificial Intelligence (Gen-AI) has generated extensive user reviews, offering valuable insights into user perceptions. This study aims to classify review sentiment using Random Forest and Support Vector Machine (SVM), evaluate the impact of SMOTE on class imbalance, and analyze review themes. This study analyzed 50,000 reviews from five popular Gen-AI applications: ChatGPT, Microsoft Copilot, Google Gemini, Perplexity, and Claude. The models were evaluated using accuracy, weighted precision, weighted recall, weighted F1-score, and confusion matrix. The results showed that SVM without SMOTE achieved the highest overall accuracy of 84% and a weighted F1-score of 0.81. However, its F1-score for the neutral class was only 0.06. Applying SMOTE reduced the overall accuracy of Random Forest from 83% to 81% and SVM from 84% to 76%. Nevertheless, SMOTE increased the neutral-class F1-score from 0.00 to 0.06 for Random Forest and from 0.06 to 0.20 for SVM. These results indicate that SMOTE improved minority-class recognition, particularly for SVM, but introduced a trade-off with overall predictive performance. Ultimately, these findings provide practical insights for improving Gen-AI applications while also serving as a reference for selecting sentiment classification models in imbalanced multiclass sentiment analysis. Purpose: This study aims to classify the sentiment of user reviews on Gen-AI applications using Random Forest and Support Vector Machine (SVM), evaluate the impact of SMOTE on imbalanced multiclass data, and identify key review themes influencing user perceptions. Methods/Study design/approach: A dataset of 50,000 user reviews from five popular Gen-AI applications was analyzed using text preprocessing, TF-IDF feature extraction, rating-based sentiment labeling, and SMOTE applied to the training data. Random Forest and SVM were evaluated using accuracy, weighted precision, weighted recall, weighted F1-score, and confusion matrix. Result/Findings: SVM without SMOTE achieved the best overall performance, with an accuracy of 84% and a weighted F1-score of 0.81. Although SMOTE reduced overall accuracy, it improved the neutral-class F1-score from 0.00 to 0.06 for Random Forest and from 0.06 to 0.20 for SVM, demonstrating better recognition of minority-class samples. Novelty/Originality/Value: This study provides a comparative evaluation of Random Forest and SVM with and without SMOTE for multiclass sentiment analysis of Gen-AI user reviews. It also integrates review theme analysis to identify factors affecting user perceptions, offering practical guidance for selecting classification models and improving Gen-AI applications.]]>
Copyrights © 2026
