Most research on learning technology uses clickstreams and questionnaires as their primary source of quantitative data. This study presents the outcomes of a systematic literature review of empirical evidence on the capabilities of multimodal data (MMD) for human learning. This paper provides an overview of what and how MMD have been used to inform learning and in what contexts. A search resulted in 42 papers that were included in the analysis. The results of the review depict the capabilities of MMD for learning and the ongoing advances and implications that emerge from the employment of MMD to capture and improve learning. In particular, we identified the six main objectives (ie, behavioral trajectories, learning outcome, learning‐task performance, teacher support, engagement and student feedback) that the MMLA research has been focusing on. We also summarize the implications derived from the reviewed articles and frame them within six thematic areas. Finally, this review stresses that future research should consider developing a framework that would enable MMD capacities to be aligned with the research and learning design (LD). These MMD capacities could also be utilized on furthering theory and practice. Our findings set a baseline to support the adoption and democratization of MMD within future learning technology research and development. Practitioner Notes: What is already known about this topic Capturing and measuring learners’ engagement and behavior using MMD has been explored in recent years and exhibits great potential. There are documented challenges and opportunities associated with capturing, processing, analyzing and interpreting MMD to support human learning. MMD can provide insights into predicting learning engagement and performance as well as into supporting the process. What this paper adds Provides a systematic literature review (SLR) of empirical evidence on MMD for human learning. Summarizes the insights MMD can give us about the learning outcomes and process. Identifies challenges and opportunities of MMD to support human learning. Implications for practice and/or policy Learning analytics researchers will be able to use the SLR as a guide for future research. Learning analytics practitioners will be able to use the SLR as a summary of the current state of the field.

The interaction with the various learners in a Massive Open Online Course (MOOC) is often complex. Contemporary MOOC learning analytics relate with click-streams, keystrokes and other user-input variables. Such variables however, do not always capture users’ learning and behavior (e.g., passive video watching). In this paper, we present a study with 40 students who watched a MOOC lecture while their eye-movements were being recorded. We then proposed a method to define stimuli-based gaze variables that can be used for any kind of stimulus. The proposed stimuli-based gaze variables indicate students’ content-coverage (in space and time) and reading processes (area of interest based variables) and attention (i.e., with-me-ness), at the perceptual (following teacher’s deictic acts) and conceptual levels (following teacher discourse). In our experiment, we identified a significant mediation effect of the content coverage, reading patterns and the two levels of with-me-ness on the relation between students’ motivation and their learning performance. Such variables enable common measurements for the different kind of stimuli present in distinct MOOCs. Our long-term goal is to create student profiles based on their performance and learning strategy using stimuli-based gaze variables and to provide students gaze-aware feedback to improve overall learning process. One key ingredient in the process of achieving a high level of adaptation in providing gaze-aware feedback to the students is to use Artificial Intelligence (AI) algorithms for prediction of student performance from their behaviour. In this contribution, we also present a method combining state-of-the-art AI technique with the eye-tracking data to predict student performance. The results show that the student performance can be predicted with an error of less than 5%.

Students' on‐task engagement during adaptive learning activities has a significant effect on their performance, and at the same time, how these activities influence students' behavior is reflected in their effort exertion. Capturing and explaining effortful (or effortless) behavior and aligning it with learning performance within contemporary adaptive learning environments, holds the promise to timely provide proactive and actionable feedback to students. Using sophisticated machine learning (ML) algorithms and rich learner data, facilitates inference‐making about several behavioral aspects (including effortful behavior) and about predicting learning performance, in any learning context. Researchers have been using ML methods in a “black‐box” approach, ie, as a tool where the input data is the learner data and the output is a given class from the chosen construct. This work proposes a methodological shift from the “black‐box” approach to a “grey‐box” approach that bridges the hypothesis/literature‐driven (feature extraction) “white‐box” approach with the computation/data‐driven (feature fusion) “black‐box” approach. This will allow us to utilize data features that are educationally and contextually meaningful. This paper aims to extend current methodological paradigms, and puts into practice the proposed approach in an adaptive self‐assessment case study taking advantage of new, cutting‐edge, interdisciplinary work on building pipelines for educational data, using innovative tools and techniques. Practitioner Notes: What is already known about this topic Capturing and measuring learners' engagement and behavior using physiological data has been explored during the last years and exhibits great potential. Effortless behavioral patterns commonly exhibited by learners, such as “cheating,” “guessing” or “gaming the system” counterfeit the learning outcome. Multimodal data can accurately predict learning engagement, performance and processes. What this paper adds Generalizes a methodology for building machine learning pipelines for multimodal educational data, using a modularized approach, namely the “grey‐box” approach. Showcases that fusion of eye‐tracking, facial expressions and arousal data provide the best prediction of effort and performance in adaptive learning settings. Highlights the importance of fusing data from different channels to obtain the most suited combinations from the different multimodal data streams, to predict and explain effort and performance in terms of pervasiveness, mobility and ubiquity. Implications for practice and/or policy Learning analytics researchers shall be able to use an innovative methodological approach, namely the “grey‐box,” to build machine learning pipelines from multimodal data, taking advantage of artificial intelligence capabilities in any educational context. Learning design professionals shall have the opportunity to fuse specific features of the multimodal data to drive the interpretation of learning outcomes in terms of physiological learner states. The constraints from the educational contexts (eg, ubiquity, low‐cost) shall be catered using the modularized gray‐box approach, which can also be used with standalone data sources.

Adaptive learning technologies are closely related to learners' self‐regulatory processes in individual and collaborative learning. This study presents the outcomes of a systematic literature review of empirical evidence on adaptive learning environments to foster self‐regulation and shared regulation of learning in collaborative settings. We provide an overview of what and how adaptive technologies have been used to understand and promote self‐regulated learning in collaborative contexts. A search resulted in 38 papers being analysed. Specifically, we identified the seven main objectives (feedback and scaffolding, self‐regulatory skills and strategies, learning trajectories, collaborative learning processes, adaptation and regulation, self‐assessment, and help‐seeking behaviour) that the adaptive technology research has been focusing on. We also summarize the implications derived from the reviewed papers and frame them within seven thematic areas. Finally, this review stresses that future research should consider developing a converging theoretical framework that would enable concrete monitoring and support for self‐regulation and socially shared regulation of learning. Our findings set a baseline to support the adoption and proliferation of adaptive learning technology within self‐regulated learning research and development. Practitioner notes: What is already known about this topic By providing personalized and learner‐centric adaptive learning environments (ADLEs), adaptive learning technology can support and foster self‐regulated learning (SRL) practices. It is possible to create a more student‐centred and effective learning environment by combining adaptive learning and collaborative learning. Socially shared regulatory activities can involve planning, monitoring, controlling and reflecting on a group's learning processes. What this paper adds Provides a systematic literature review of empirical evidence on ADLEs, SRL and socially shared regulation of learning (SSRL) in collaborative contexts. Summarizes the insights on (S)SRL through ADLEs in collaborative learning. Identifies challenges and opportunities for ADLEs to support (S)SRL in collaborative learning. Implications for practice and/or policy Learning analytics and educational technology researchers will be able to use the systematic review as a guide for future research. Learning analytics and educational technology practitioners will be able to use the systematic review as a summary of the field's current state.

Background: Problem‐solving is a multidimensional and dynamic process that requires and interlinks cognitive, metacognitive, and affective dimensions of learning. However, current approaches practiced in computing education research (CER) are not sufficient to capture information beyond the basic programming process data (i.e., IDE‐log data). Therefore, how cognition and affect intertwine and unfold over time in programming problem‐solving activities are rarely investigated.Objectives: In this study, we examined how the theory‐informed measures from multimodal data that we have selected as proxies for cognitive and affective dimensions of learning, are associated with student performance, and in comparison, to prior‐knowledge.Methods: A high‐frequency temporal data was collected with a camera, an electroencephalogram, and an eye‐tracker from 40 computer science students (bachelor and master studies) in the context of a code‐debugging activity. To study the cognitive processes associated with learning we focused on cognitive load theory (CLT) and the human information processing model. In addition, we complemented CLT with the model of affective dynamics in learning to avoid the machine reductionism perspective.Results: Our findings demonstrated that attention, convergent thinking, and frustration were positively correlated with students' successful code‐debugging (i.e., performance), and frequently manifested by high performing participants. Cognitive load, memory load, and boredom were negatively correlated with students' performance, and typically manifested by low performing participants.Implications: Extending the context of analysis in reference to student cognitive processes and affective states, affords educators not just to identify lower performers, but also to understand the potential reasons behind their performance, making our method an important contribution in the confluence of CER and the learning technology communities. In addition, the insights extracted from our analyses allow us to discuss potential avenues for improving learning design and the design of interactive learning systems to support the development of students' problem‐solving skills.

Investigating and explaining the patterns of learners' engagement in adaptive learning conditions is a core issue towards improving the quality of personalized learning services. This study collects learner data from multiple sources during an adaptive learning activity, and employs a fuzzy set qualitative comparative analysis (fsQCA) approach to shed light to learners' engagement patterns, with respect to their learning performance. Specifically, this study measures and codes learners' engagement by fusing and compiling clickstreams (e.g., response time), physiological data (e.g., eye-tracking, EEG, electrodermal activity) and survey data (e.g., goal-orientation) to determine what configurations of those data explain when learners can attain high or medium/low learning performance. For the evaluation of the approach, an empirical study with 32 undergraduates was conducted. The analysis revealed six configurations that explain learners' high performance and three that explain learners' medium/low performance, driven by engagement measures coming from the multimodal data. Since fsQCA explains the outcome of interest itself, rather than its variance, these findings advance our understanding on the combined effect of the multiple indicators of engagement on learners' performance. Limitations and potential implications of the findings are also discussed.