The adoption of Large Language Models (LLM) for software issue resolution causes token consumption inefficiency due to noisy and redundant context. This research proposes a hybrid system placing a BERT classification model as a context relevance pre-filter upstream of the Retrieval-Augmented Generation (RAG) pipeline, with hyperparameters optimized using Bayesian Optimization. A Design Science Research approach is applied through an ablation study of five scenarios (S0-S4) on GitHub Issues and SWE-bench datasets. The classification model achieves an F1-macro of 0.887, and BERT-based filtering (S2) reduces token consumption by 8.35% versus baseline, statistically significant (Wilcoxon, p<0.05). Bayesian Optimization yields a validation F1-macro of 0.889, higher than default hyperparameters. The main finding: token efficiency comes from the BERT pre-filter component, not from RAG which adds context.
Copyrights © 2026