Collaborative problem solving has numerous benefits for learners, such as improving higher-level reasoning and developing critical thinking. While learners engage in collaborative activities, they often experience impasse, a potentially brief encounter with differing opinions or insufficient ideas to progress. Impasses provide valuable opportunities for learners to critically discuss the problem and re-evaluate their existing knowledge. Yet, despite the increasing research efforts on developing multimodal modeling techniques to analyze collaborative problem solving, there is limited research on detecting impasse in collaboration. This paper investigates multimodal detection of impasse by analyzing 46 middle school learners’ collaborative dialogue—including speech and facial behaviors—during a coding task. We found that the semantics and speaker information in the linguistic modality, the pitch variation in the audio modality, and the facial muscle movements in the video modality are the most significant unimodal indicators of impasse. We also trained several multimodal models and found that combining indicators from these three modalities provided the best impasse detection performance. To the best of our knowledge, this work is the first to explore multimodal modeling of impasse during the collaborative problem solving process. This line of research contributes to the development of real-time adaptive support for collaboration.
27 February 2024 in British Journal of Educational Technology
Intelligent systems to support collaborative learning rely on real-time behavioral data, including language, audio, and video. However, noisy data, such as word errors in speech recognition, audio static or background noise, and facial mistracking in video, often limit the utility of multimodal data. It is an open question of how we can build reliable multimodal models in the face of substantial data noise. In this paper, we investigate the impact of data noise on the recognition of confusion and conflict moments during collaborative programming sessions by 25 dyads of elementary school learners. We measure language errors with word error rate (WER), audio noise with speech-to-noise ratio (SNR), and video errors with frame-by-frame facial tracking accuracy. The results showed that the model’s accuracy for detecting confusion and conflict in the language modality decreased drastically from 0.84 to 0.73 when the WER exceeded 20%. Similarly, in the audio modality, the model’s accuracy decreased sharply from 0.79 to 0.61 when the SNR dropped below 5 dB. Conversely, the model’s accuracy remained relatively constant in the video modality at a comparable level (> 0.70) so long as at least one learner’s face was successfully tracked. Moreover, we trained several multimodal models and found that integrating multimodal data could effectively offset the negative effect of noise in unimodal data, ultimately leading to improved accuracy in recognizing confusion and conflict. These findings have practical implications for the future deployment of intelligent systems that support collaborative learning in actual classroom settings.
7 March 2024 in International Journal of Child-Computer Interaction
The COVID-19 pandemic shifted many U.S. schools from in-person to remote instruction. While collaborative CS activities had become increasingly common in classrooms prior to the pandemic, the sudden shift to remote learning presented challenges for both teachers and students in implementing and supporting collaborative learning. Though some research on remote collaborative CS learning has been conducted with adult learners, less has been done with younger learners such as elementary school students. This experience report describes lessons learned from a remote after-school camp with 24 elementary school students who participated in a series of individual and paired learning activities over three weeks. We describe the design of the learning activities, participant recruitment, group formation, and data collection process. We also provide practical implications for implementation such as how to guide facilitators, pair students, and calibrate task difficulty to foster collaboration. This experience report contributes to the understanding of remote CS learning practices, particularly for elementary school students, and we hope it will provoke methodological advancement in this important area.
About Me
I am generally interested in artificial intelligence in precision health.
I am generally interested in artificial intelligence in precision health.