Evaluating AI Tutor Interaction with Ambiguous Junior High Mathematics Questions Using Black-Box - Islam, Khoirul; Saidah, Nisfu Laili; Yahya, Saifudin; Syaikhuddin, Muhammad Miftakhul (Prodi Bisnis Digital Universitas Pahlawan Tuanku Tambusai, 2026)
The growth of generative artificial intelligence as a mathematics education tutoring tool presents new possibilities of assisting student learning. Nevertheless, successful learning interactions involve AI tutors to address ambiguous questions posed by students in a proper manner, particularly in junior high school mathematics, where questions are frequently incomplete, have ambiguous concepts, or lack reference points to the surrounding context. This paper compares the quality of interaction of two popular AI tutors, ChatGPT and Gemini, in answering ambiguous questions in mathematics at the junior high school level. An approach that was used was black-box testing where only observable input-output behavior was tested without accessing internal model mechanisms. A sample of 50 ambiguous mathematics situations was randomly designed around five types of ambiguity, namely incomplete information, conceptual ambiguity, output format ambiguity, missing context, and contradictory information. Both AI tutors were tested on each scenario once giving a total of 100 dialog interactions. Two independent raters evaluated all interactions based on a Human–AI Interaction rubric which includes ambiguity detection, relevance of clarification, transparency of assumptions, quality of interaction, and quality of solution. The findings show that both systems can identify ambiguity, but Gemini shows higher rates of clarification and more pedagogically suitable interaction patterns than ChatGPT, especially in situations related to the lack of information and contextual ambiguity. The results demonstrate the significance of clarification behavior and interaction design in AI-based tutoring systems and offer a viable way of how AI tutors can be responsibly used in mathematics education at junior high schools.