AI tutors struggle to know when to help versus push students, study finds

TutorMoments emerged from a fundamental question about what makes a good tutor: the ability to read a student's moment and decide whether scaffolding (simplifying the problem) or a push for rigor (encouraging deeper thinking) is the right move. Most existing benchmarks for AI tutors don't capture this tension. They reward one behavior consistently—say, never giving away the answer—without asking whether that behavior fit the student's actual understanding. But tutoring is not a single fixed behavior; it's a judgment call.

The evaluation framework is built on 462 real tutoring transcripts from a U.S. high-dosage tutoring program serving students in grades 2–7, most of whom attend Title I schools. A pool of 27 experienced U.S.-based math teachers read these transcripts and flagged over 1,500 key decision points—moments where a tutor had to weigh scaffolding against pushing for rigor. To evaluate an AI tutor, TutorMoments pauses a transcript at one of these moments and hands the session to a language model, which takes over for five simulated turns with another language model playing the student. An LLM-based scoring pipeline then rates whether the model (1) scaffolded when the student needed support, (2) pushed for rigor when the student was ready, and (3) avoided over-scaffolding. The ground truth comes from teacher annotations: if the majority of annotators judged a moment as calling for rigor, that is the expected move.

Testing seven models with two prompts revealed a sharp divide. Under a plain prompt that only tells the model to "tutor well" using its knowledge of good tutoring, models scored between 0.39 and 0.68 on appropriate scaffolding, 0.04 and 0.18 on appropriate rigor, and 0.37 to 0.54 on avoiding over-scaffolding. When given an evaluation-aware prompt that explicitly spelled out the three options (scaffold, push for rigor, or avoid over-scaffolding), every model's scores rose—but the gains were uneven. The clearest pattern: prompting matters enormously. Models' default helpful-assistant behavior is not enough on its own. However, even the best-scoring models had substantial room for improvement, and they differed widely in how reliably they made the right call.

Breaking down the actual moves revealed another limitation: while the evaluation-aware prompt encouraged models to push for rigor, they relied on fewer strategies than human tutors did, often defaulting to asking students to explain their answers. Human tutors, by contrast, employ a much wider range of moves and are far more likely to step back and let students work independently. The researchers note several important caveats: human tutors scored below the models' best evaluation-aware results, but that's partly because the dataset is built from annotators looking for moments tutoring could have improved, not ideal practice. The numbers also measure tutor behavior at a decision point, not whether a real student actually learned. Finally, the dataset is narrow—U.S.-based, limited to elementary and middle-school math, and annotated by a single pool of educators—so findings may not generalize to other subjects, grade levels, or educational settings. The team is releasing the data, code, and model replays for reproducibility and has committed to building toward a larger, multimodal dataset and stronger scoring pipeline based on feedback.