Analisis Pengaruh Karakteristik Masukan Teks terhadap Kinerja MiniLMv2-L6-H384 dan BERT-Base-Uncased pada Quora Question Pairs - Ken Ratri Wardani; Inge Martina; Jimmy Fong Xin Wern (Yayasan Petra Harapan Bangsa, 2025)
Knowledge distillation is a technique for simplifying large language models into more concise models while maintaining accuracy. Bidirectional encoder representations from transformers (BERT) offer strong performance but require significant computational resources, whereas mini language models (MiniLM) are five times smaller. This study aims to compare the performance of these two models on a Quora question-pair dataset, focusing on the effects of sequence length and token rarity on classification accuracy. Both models were trained using identical training parameters. Test results show that BERT achieves 91.22% accuracy and 88.17% F1-score, slightly outperforming MiniLM, which achieves 90.12% accuracy and 86.73% F1-score. However, MiniLM provides 5.3 times faster inference speed. These findings provide empirical guidance for model optimisation in environments with limited computational resources or real-time response requirements, where MiniLM's efficiency is acceptable with a slight decrease in accuracy. Future research is recommended to explore hybrid systems that delegate complex tasks to large models and general tasks to smaller models.