In mathematics tutoring, using appropriate instructional discursive strategies, called "talk moves'', is critical to support student learning. Training tutors in the appropriate use of talk moves is a key component of tutor development programs. However, tutor development at scale is a challenge. Recent research has shown that automatic talk moves classification of tutorial discourse can facilitate large-scale delivery of personalized talk moves feedback. In this paper, we build on this work and share our current progress using large language models to classify talk moves in transcripts of tutoring sessions. We report classification results from fine-tuned models, prompt optimization, and supervised embedding vectors classification. The fine-tuned strategy performed best, yielding better performance (.87 macro and .93 weighted f1 score in predicting expert labels) than the current state-of-the-art RoBERTa model. We discuss trade-offs across methods and models.
We tested a multi-party, multi-level AI-enhanced coach-tutor model, where AI feedback on tutoring discourse provided to instructional coaches (level 1) leads to improved tutor discourse practices (level 2), resulting in student achievement growth (level 3). We tested this model in the context of Saga tutoring, a large high-dosage tutoring (HDT) provider (37 coaches, 172 tutors, 2,966 students). Using latent growth curve modeling and multi-level moderated mediation techniques, we validated key mechanisms of our theory of change and found that growth in tutors discourse moves that promoted rigorous thinking and building on student ideas was associated with student achievement growth and mediated baseline-to-growth pathways. Further, the extent of engagement with the AI feedback amplified key pathways in the theory of change. Overall, these findings provide support for the effectiveness of AI-enhanced coaching tools that emphasize discourse as a lever for improving student outcomes at scale.