Exploring the strengths and limitations of AIs, using the socratic method, in solving physics problems - André da Silva Ramos de Faria; Márcio Velloso da Silveira; Antonio Carlos Fontes dos Santos (Journal of Environment and Sustainability Education, 2026)
The research is designed as a qualitative-analytical case study, employing the Socratic method as an analytical tool to probe reasoning consistency, contradiction recognition, and conceptual stability. Three AI systems: ChatGPT, Gemini, and Copilot, were examined while analyzing a non-standard equilibrium problem involving a pen attached to a vertical wall. The findings indicate that, although the AIs correctly articulate isolated physics concepts, they struggle to integrate these concepts coherently in unfamiliar contexts, exhibiting contradictions, diagrammatic inconsistencies, and an inability to revise reasoning when challenged. These results demonstrate that the principal limitation of current AI systems lies not merely in language generation, but in conceptual reasoning and meaning construction in physics. The study highlights important implications for physics education, emphasizing the need for critical mediation by educators and caution against treating AI-generated explanations as epistemically authoritative.