In large-scale programming courses, providing learners with immediate and effective feedback is a significant challenge. This study explores the potential of Large Language Models (LLMs) to generate feedback on code assignments and to address the gaps in Automated Test-based Feedback (ATF) tools commonly employed in programming courses. We applied dedicated metrics in a Massive Open Online Course (MOOC) on programming to assess the correctness of feedback generated by two models, GPT-3.5-turbo and GPT-4, using a reliable ATF as a benchmark. The findings point to effective error detection, yet the feedback is often inaccurate, with GPT-4 outperforming GPT-3.5-turbo. We used insights gained from the prompt practices to develop Gipy, an application for submitting course assignments and obtaining LLM-generated feedback. Learners participating in a field experiment perceived the feedback provided by Gipy as moderately valuable, while at the same time recognizing its potential to complement ATF. Given the learners' critique and their awareness of the limitations of LLM-generated feedback, the studied implementation may be able to take advantage of the best of both ATF and LLMs as feedback resources. Further research is needed to assess the impact of LLM-generated feedback on learning outcomes and explore the capabilities of more advanced models.
27 September 2021 in Journal of Computer Assisted Learning
This study examines the relationship between online learning behavior and learning outcomes with the aim of identifying early predictors of learners' persistence and success. Research focused on the first learning units of online language courses in a developing country in order to provide teachers and administrators with a simple model for identifying at-risk students. Using data from 716 students enrolled in 24 English courses at a Peruvian university, learning analytics approach was applied, framed by the self-determination theory (SDT). Results suggest that unit completion rates and time spent on learning, which are both related to sense of autonomy, strongly predict persistence at the course mid-point. Moreover, the same variables can predict student persistence as early as unit three, providing even earlier indications for dropping out. Quiz score and midterm grade, which are related to the SDT competence construct, moderately predict achievement, defined as the final exam grade. Relatedness factors (emails and Facebook activity) were not found to be early predictors.