AI-Assisted Labeling for Indonesian Hadith Classification using Four Thematic Categories: Evaluating TF-IDF+SVM and IndoBERT - Domi Sepri; Ahmad Fauzi; Firman Firman; Muhammad Nabil (Forum Kerjasama Pendidikan Tinggi (FKPT), 2026)
Hadith is the second primary source of Islamic law after the Qur'an, and its large volume makes manual thematic classification time-consuming and inefficient. This study proposes an AI-assisted labeling approach to construct an Indonesian thematic hadith dataset and evaluates the performance of two text classification methods, namely TF-IDF + Support Vector Machine (SVM) and IndoBERT. The dataset consists of 6,600 Indonesian-translated Sahih Bukhari hadiths collected from the Hadith API and categorized into four thematic classes: aqidah, ibadah, akhlak, and muamalah. The annotation process employed Gemini 2.5 Flash with a structured prompt and JSON-based output format, followed by validation performed by a hadith researcher validation on a randomly selected 5% sample, achieving an overall agreement of 73.3%. The annotated data were divided into training and testing sets using an 80:20 stratified split. Model performance was evaluated using Accuracy, Macro F1-score, and Weighted F1-score. Experimental results show that TF-IDF + SVM achieved an Accuracy of 73.1%, a Macro F1-score of 70.1%, and a Weighted F1-score of 73.0%, while IndoBERT achieved an Accuracy of 72.3%, a Macro F1-score of 69.7%, and a Weighted F1-score of 72.2%. The results indicate that the conventional TF-IDF + SVM approach slightly outperformed the Transformer-based IndoBERT model on the proposed dataset. The main contributions of this study are the construction of an Indonesian thematic hadith dataset through AI-assisted labeling and a comparative evaluation of conventional and Transformer-based methods for Indonesian hadith classification.