Comparative Study of Random Forest and Ordinal Regression in Concept Map Quality Assessment: The Role of TF-IDF, BERT, and SMOTE-based Balancing - Rismayanti, Nurul; Prasetya, Didik Dwi; Widiyaningtyas, Triyanna; Hirashima, Tsukasa (Prodi Teknik Informatika FIK Universitas Muslim Indonesia, 2025)
Automatic assessment of concept map quality is an important challenge in the field of education, particularly in evaluating students' conceptual understanding objectively and efficiently. This study aims to compare the performance of two machine learning algorithms, namely Random Forest and Ordinal Regression, in classifying the quality of concept maps. The evaluation was conducted on three approaches to text feature representation: Term Frequency-Inverse Document Frequency (TF-IDF), Bidirectional Encoder Representations from Transformers (BERT), and a combination of both (TF-IDF + BERT). Additionally, this study compares the performance of the models under two dataset conditions: original data and data balanced using the Synthetic Minority Over-sampling Technique (SMOTE), to address the class imbalance that often occurs in educational data. The data used consists of a collection of propositions from students' concept maps that have been labeled with ordinal scores based on quality. Text representation is extracted using the TF-IDF and BERT approaches, and then used as input to build the classification model. Performance evaluation was conducted using the metrics of Accuracy, Precision, Recall, F1-score, Cohen’s Kappa, and MAE. The results show that the Ordinal Regression model with TF-IDF representation combined with SMOTE achieved the best performance, with an accuracy of 0.8777, an F1-score of 0.8773, and a Cohen’s Kappa of 0.7701. These results indicate that classical feature representations like TF-IDF remain effective in limited data scenarios, and that the SMOTE technique successfully improved the model's performance by reducing bias towards the majority class. This research contributes to the development of an automatic concept map assessment system and suggests optimal classification strategies for educational datasets with ordinal and imbalanced characteristics