We propose a video-based transfer learning approach for predicting problem outcomes of students working with an intelligent tutoring system (ITS) by analyzing their faces and gestures. The ability to predict such outcomes enables tutoring systems to adjust interventions and ultimately yield improved student learning. We collected and released a labeled dataset of 2,749 problem-solving interaction samples of 54 students working with an intelligent online math tutor. Our transfer-learning challenge was then to design a representation in the source domain of images obtained from the Internet for facial expression analysis, and transfer this learned representation for human behavior prediction in the domain of webcam videos of students in a classroom environment. We developed a novel facial affect representation and a user-personalized training scheme that unlocks the potential of this representation. We designed several variants of a recurrent neural network that models the temporal structure of video sequences. Our final model, named ATL-BP for Affect Transfer Learning for Behavior Prediction, achieves a relative increase in the mean F-score of 50% over the state-of-the-art method on this new dataset. We also propose an additional set of annotations to predict students’ engagement while solving a specific problem, and present models that can predict such engagement.
Equipping an Intelligent Tutoring System (ITS) with the ability to interpret affective signals from students could potentially improve the learning experience of students by enabling the tutor to monitor the students' progress and provide timely interventions as well as present appropriate affective reactions via a virtual tutor. Most ITSs equipped with affect modeling capabilities attempt to predict the emotional state of users. However, the focus in this work is instead on trying to directly predict the learning outcomes of students from a stream of video capturing the students faces as they work on a set of math problems. Using facial features extracted from a video stream, we train classifiers to directly predict the success or failure of a student's attempt to answer a question while the student has just begun to work on the problem. In this work, we first introduce a novel dataset of student interactions with MathSpring, a popular ITS. We provide an exploratory analysis of the different problem outcome classes using typical facial action unit activations. We develop baseline models to predict the problem outcome labels of students solving math problems and discuss how early problem outcome labels can be forecasted and utilized to provide possible interventions.
Artificial intelligence (AI) has made large changes in major industries and disrupted or reorganized many disciplines. As a result, traditional educational practices need to be reexamined to enable learners to develop new skills, to manage and utilize new technologies and to increase productivity in a rapidly changing world. A more flexible educational system is required to enable life-long and life-wide upskilling and reskilling. This article provides four grand challenges for AI and education to optimize digital learning, online resources and virtual classrooms. It suggests several problems to address, visions to spur the field forward and strategies that will make teaching and learning more effective. The article also considers ethical use of technology, decreased jobs in some sectors and the possibility that AI will exacerbate an existing deficit in diversity and equity among students.