Verbal and non-verbal activities convey insightful information about people’s affect, empathy, and engagement during social interactions. In this paper, we investigate the usage of inertial sensors to recognize verbal (e.g., speaking), non-verbal (e.g., head nodding, shaking) and other activities (e.g., eating, no movement). We implement an end-to-end deep neural network to distinguish among these activities. We then explore the generalizability of the approach in three scenarios: (1) using new data to detect a known activity from a known user, (2) detecting a novel activity of a known user and (3) detecting the activity of an unknown user. Results show that using accelerometer and gyroscope sensors, the model achieves a balanced accuracy of 55% when tested on data from a new user, 41% on a new activity of an existing user, and 80% on new data of a known activity from an existing user. The results are between 7-47 percentage points higher than baseline classifiers.

The latest developments in wearable sensors have resulted in a wide range of devices available to consumers, allowing users to monitor and improve their physical activity, sleep patterns, cognitive load, and stress levels. However, the lack of out-of-the-lab labelled data hinders the development of advanced machine learning models for predicting affective states. Furthermore, to the best of our knowledge, there are no publicly available datasets in the area of Human Memory Augmentation. This paper presents a dataset we collected during a 13-week study in a university setting. The dataset, named LAUREATE, contains the physiological data of 42 students during 26 classes (including exams), daily self-reports asking the students about their lifestyle habits (e.g. studying hours, physical activity, and sleep quality) and their performance across multiple examinations. In addition to the raw data, we provide expert features from the physiological data, and baseline machine learning models for estimating self-reported affect, models for recognising classes vs breaks, and models for user identification. Besides the use cases presented in this paper, among which Human Memory Augmentation, the dataset represents a rich resource for the UbiComp community in various domains, including affect recognition, behaviour modelling, user privacy, and activity and context recognition.

We propose a study design for validating explainable AI (XAI) tools in longitudinal contexts, focusing on ubiquitous and wearable computing. We aim to address unique challenges such as evolving user behavior and temporal dependencies inherent in longitudinal data. The study design includes both general and longitudinal explanation validation, assessing five XAI constructs: trust, satisfaction, actionability, understanding, and reliance through end-user studies. We demonstrate the proposed study design through a working example utilizing two XAI techniques -SHAP-based feature importance and Bayesian counterfactual explanations- on the LAUREATE dataset, which includes machine learning models for estimating weekly affective states of students based on physiological and behavioral data. To the best of our knowledge, this is the first study design that aims to investigate the longitudinal aspects of XAI, by proposing to evaluate the impact of the longitudinal order in which instances and their corresponding explanations are presented to end-users.

Affective states and cognition share a close relationship, influencing each other in several ways. In particular, our affect, i.e. our feelings and mood, impacts our perception of the world, our memory, and our decisions. This relationship also influences the cognitive process of learning. Working towards personalised student interventions for improving learning and mental well-being, we conducted a longitudinal study during a university semester. The resulting dataset, called LAUREATE (the Longitudinal multimodAl stUdent expeRience datasEt for AffecT and mEmory research), consists of daily self-reports from students, audiovisual material from lectures, physiological signals from students and lecturers, students’ grades, and more. In this work, we present an initial analysis of lecturers’ and students’ experiences during lectures in terms of self-reported affect, especially positive activation (PA), negative activation (NA), valence (VA), and engagement.