TutorMoments Framework Evaluates AI Tutoring Decision-Making

TutorMoments is a new evaluation framework that uses real-world math tutoring transcripts to test whether AI models can effectively decide when to scaffold a student's learning and when to push for deeper reasoning. By pausing recorded tutoring sessions at critical decision points, the framework tasks language models with continuing the interaction alongside a simulated student. The project aims to address the tendency of AI models to over-help, which can inadvertently hinder the productive struggle necessary for learning.

Preliminary findings indicate that while models often default to being overly helpful, performance improves when prompts explicitly define the trade-off between scaffolding and rigor. However, models still exhibit significant variability in their decision-making compared to human tutors. The researchers have released a dataset of de-identified transcripts, code for the replay pipeline, and model replays to support further research. They emphasize that these results measure tutor behavior rather than student learning outcomes and note that the current dataset is limited to U.S.-based elementary and middle-school math.