While many studies have addressed the benefits of technology-assisted L2 writing, limited research has delved into how generative artificial intelligence (GAI) supports students in completing their writing tasks in Mandarin Chinese. In this study, 26 university-level Mandarin Chinese foreign language students completed two writing tasks on two separate days. The writing prompts were from the AP Chinese Language and Culture Exam free response questions. Students were permitted to use ChatGPT for assistance with only one of the given prompts each day. After data analysis, writing scores were compared with and without the use of ChatGPT to discern which aspects of ChatGPT students relied on the most. To maintain the validity and reliability of the scores, students' writing samples were independently evaluated by two professional raters, yielding a high degree of inter-rater reliability. The results indicate that utilizing ChatGPT for assistance consistently resulted in higher scores. Additionally, students primarily sought assistance from ChatGPT for vocabulary-related and translation-related questions. Quantitative and qualitative data were used to analyze students’ perceptions of using GAI to support the students in completing the writing tasks. The current study seeks to bridge existing knowledge gaps and offer empirical data on how GAI tools, like ChatGPT, can develop students’ L2 writing within the Mandarin Chinese language learning context.
25 November 2024 in Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi
The integration of generative AI in K12 education and assessment development holds the potential to revolutionize instructional practices, assessment development, and content alignment. This article presents analytical insights and findings from early adoption studies utilizing AI-powered tools developed by Finetune—Generate and Catalog. Generate enhances the efficiency of assessment item development through customized natural language generation, producing high-quality, psychometrically valid items. Catalog intelligently tags and aligns educational content to various standards and frameworks, improving precision and reducing subjectivity. Through three comprehensive case studies, we explore the practical applications, benefits, and lessons learned from employing these AI systems in real-world educational settings. The purpose of this series of studies was to investigate the ways generative AI is currently being used in practical applications in test development to improve processes and products. The studies demonstrate significant reductions in time and costs, enhanced accuracy, and consistency in content alignment, and improved quality of educational and assessment materials. The findings underscore the substantial benefits and critical importance of customized AI systems, rigorous training for both AI models and users, and adopting appropriate evaluation metrics. With the use of off-the-shelf generative AI models expanding rapidly, it is vital that the effectiveness of AI systems that are highly customized through collaborations with measurement experts be presented, in order to maximize benefits and uphold the fundamental principles and best practices of test development.
The integration of large language models (LLMs) into automated essay scoring (AES) represents a significant shift in how essay scoring is approached. While traditional AES systems have long depende...
12 March 2026 in Educational Measurement: Issues and Practice
Advances in automated scoring are closely aligned with advances in machine-learning and natural-language-processing techniques. With recent progress in large language models (LLMs), the use of ChatGPT, Gemini, Claude, and other generative-AI chatbots for automated scoring has been explored. Given their strong reasoning capabilities, LLMs can also produce rationales to support the scores they assign. Thus, evaluating the rationales provided by both human and LLM raters can help improve the understanding of the reasoning that each type of rater applies when assigning a score. This study investigates the rationales of human and LLM raters to identify potential causes of scoring inconsistency. Using essays from a large-scale test, the scoring accuracy of GPT-4o, Gemini, and other LLMs is examined based on quadratic weighted kappa and normalized mutual information. Cosine similarity is used to evaluate the similarity of the rationales provided. In addition, clustering patterns in rationales are explored using principal component analysis based on the embeddings of the rationales. The findings of this study provide insights into the accuracy and ``thinking'' of LLMs in automated scoring, helping to improve the understanding of the rationales behind both human scoring and LLM-based automated scoring.