This paper explores the potential of Large Language Models (LLMs), specifically GPT-4, to enhance the precision and effectiveness of Automated Assessment Systems (AAS) for open-ended mathematics problems. While LLMs have demonstrated transformative capabilities across various disciplines, their application in AAS, particularly for mathematical logic and open-ended problem-solving, still needs to be explored. Our research addresses this gap by developing and critically evaluating a GPT-4-based AAS. We analyzed 4,180 responses to open-ended mathematics questions from 380 6th-grade primary school students. Three human experts and the GPT-4 model independently assessed these responses using a pre-established rubric. Our findings reveal high consistency between human and GPT-4 assessments in most instances, highlighting the potential of integrating GPT-4 into AAS. We categorized scoring discrepancies from GPT-4 and human raters by error type and identified specific mathematical content areas where automated assessment faced limitations. We evaluated two strategies to enhance GPT-4’s assessment capabilities: (1) using elaborate prompts and (2) implementing advanced prompt engineering techniques such as Chain-of-thought, Self-consistency, and Tree-of-thought. While comprehensive prompts significantly improved assessment quality, applying advanced prompt engineering techniques directly produced suboptimal results, indicating a need for further refinement. This study contributes to the emerging body of research evaluating GPT-4 in the context of AAS for open-ended mathematics problems, shedding light on both the strengths and limitations of this approach. Our findings provide valuable insights and a foundation for future research to refine the integration of LLMs in AAS, particularly in mathematics education.

This preliminary study explores how GPT-4 Vision (GPT-4V) technology can be integrated into teacher analytics through observational assessment, aiming to improve reflective teaching practice. Our study develops a Video-based Automatic Assessment System (VidAAS) powered by GPT-4V. This approach uses Generative Artificial Intelligence (GenAI) to provide detailed insights into classroom dynamics. Our study encompasses various methods with multiple steps: a comprehensive literature review, prototype development of the VidAAS, and usability testing with in-service teachers. The study findings reveal that VidAAS demonstrates high accuracy in evaluating skills in the behavioral (psychomotor) domain and offers comprehensive explanations for each assessment. While showing promise in these areas, the system also indicates potential for further enhancement in processing speed and refinement in assessing cognitive and affective domains. We discuss how VidAAS supports teachers’ reflection-in-action and reflection-on-action, emphasizing the need to balance AI-driven insights and human judgment. Our study findings also guide future research avenues for VidAAS design, implementation, and integration in teacher analytics, underscoring GPT-4V’s potential for real-time, scalable feedback and a deeper classroom understanding.