This study investigates the relationship between students’ mathematics self-efficacy and mathematical critical thinking (MCT) performance in the context of vocational high schools. Drawing on Bandura’s theory of self-efficacy, the research explores how students’ belief in their mathematical abilities predicts their capacity to analyze, evaluate, and reason through mathematical problems, particularly within the topic of matrices. 137 eleventh-grade students from four state vocational schools in Indonesia participated in this study. Data were collected using validated self-efficacy and MCT instruments and analyzed through correlational and regression techniques. Results indicate a strong, positive, and statistically significant correlation between mathematics self-efficacy and MCT (r = 0.743, p < 0.001), with self-efficacy explaining 55.2% of the variance in MCT scores. This relationship remained robust after controlling for gender and vocational program. Regression analysis confirmed self-efficacy significantly predicted students’ MCT performance (β = 0.743, p < 0.001). Additionally, components of self-efficacy—magnitude and strength—were identified as key predictors, while magnitude showed the highest predictive power. Analysis across MCT components revealed that students’ self-efficacy beliefs most influenced inference and analytical reasoning. These findings underscore the critical role of self-efficacy in shaping higher-order mathematical thinking and suggest that enhancing students’ confidence in mathematics may significantly improve their reasoning and problem-solving abilities. The study provides practical implications for instructional design in vocational education settings, emphasizing the integration of motivational strategies to support students’ cognitive performance.