Evaluation of Stratified K-Fold Cross Validation for Predicting Bug Severity in Game Review Classification - Mayangsari, Mustika Kurnia; Syarif, Iwan; Barakbah, Aliridho (Universitas Muhammadiyah Malang, 2023)
Steam review data provides a lot of information for the game development team, either positive or negative reviews. It is essential as negative and positive reviews provide crucial information, and 7% of positive reviews contains bug reports. These bug reports were captured after the game was released, and many reports of common problems still exist. If players found an issue in the game, they could report it directly through the review feature provided by the online game platform. However, it took a long time for the development team to manually analyze and classify the reviews. This study proposed a new approach to automatically classify the reviews on Steam based on the bug severity level. Therefore, to solve this problem, we recommend a solution based on the research background indicated above. For this experiment, we analyzed reviews on two popular game titles namely, FIFA 23 and Apex Legends. We implemented three different classifiers, namely KNN, Decision Tree, and Naïve Bayes, which would be used to train a dataset to classify the bug severity level. Due to the imbalanced dataset, we performed cross-validation to reduce bias in the dataset. Performance in this model would be evaluated using accuracy rate, precision, recall, and F1 score. As a result, the experiment showed that game reviews of different game titles achieved different accuracy scores. The game review classification for FIFA 23 performed better than the game review classification for Apex Legends. The mean accuracy score of FIFA 23 was 72% with Decision Tree and Apex Legend was 64% with KNN.