The ubiquitous availability of smartphones and smartwatches with integrated inertial measurement units (IMUs) enables straightforward capturing of human activities through collecting movement data. For specific applications of sensor-based human activity recognition (HAR), however, logistical challenges and burgeoning costs render especially the ground-truth annotation of such data a difficult endeavor, resulting in limited scale and diversity of datasets available for deriving effective HAR systems and less than ideal recognition capabilities. Transfer learning, i.e., leveraging publicly available labeled datasets to first learn useful representations that can then be fine-tuned using limited amounts of labeled data from a target domain, can alleviate some of the performance issues of contemporary HAR systems. Yet they can fail when the differences between source and target conditions are too large and/or only few samples from a target application domain are available—each of which are typical challenges in real-world human activity recognition scenarios. In this article, we present an approach for economic use of publicly available labeled HAR datasets for effective transfer learning. We introduce a novel transfer learning framework—Cross-Domain HAR—which follows the teacher-student self-training paradigm to more effectively recognize activities with very limited label information. It bridges conceptual gaps between source and target domains, including sensor locations and type of activities. Cross-Domain HAR enables substantial performance improvements over the state-of-the-art in sensor-based HAR scenarios. Through our extensive experimental evaluation on a range of benchmark datasets we specifically demonstrate the effectiveness of our approach for practically relevant few-shot activity recognition scenarios. We also present a detailed analysis into how the individual components of our framework affect downstream performance and provide practical suggestions for using the framework in real-world applications.
9 June 2025 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
In the many years since the inception of wearable sensor-based Human Activity Recognition (HAR), a wide variety of methods have been introduced and evaluated for their ability to recognize activities. Substantial gains have been made since the days of hand-crafting heuristics as features, yet, progress has seemingly stalled on many popular benchmarks, with performance falling short of what may be considered 'sufficient'-despite the increase in computational power and scale of sensor data, as well as rising complexity in techniques being employed. The HAR community approaches a new paradigm shift, this time incorporating world knowledge from foundational models. In this paper, we take stock of sensor-based HAR - surveying it from its beginnings to the current state of the field, and charting its future. This is accompanied by a hands-on tutorial, through which we guide practitioners in developing HAR systems for real-world application scenarios. We provide a compendium for novices and experts alike, of methods that aim at finally solving the activity recognition problem.
3 March 2025 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Human activity recognition (HAR) using ambient sensors in smart homes has numerous applications for human healthcare and wellness. However, building general-purpose HAR models that can be deployed to new smart home environments requires a significant amount of annotated sensor data and training overhead. Most smart homes vary significantly in their layouts, i.e., floor plans and the specifics of sensors embedded, resulting in low generalizability of HAR models trained for specific homes. We address this limitation by introducing a novel, layout-agnostic modeling approach for HAR systems in smart homes that utilizes the transferrable representational capacity of natural language descriptions of raw sensor data. To this end, we generate Textual Descriptions Of Sensor Triggers (TDOST) that encapsulate the surrounding trigger conditions and provide cues for underlying activities to the activity recognition models. Leveraging textual embeddings, rather than raw sensor data, we create activity recognition systems that predict standard activities across homes without (re-)training or adaptation to target homes. Through an extensive evaluation, we demonstrate the effectiveness of TDOST-based models in unseen smart homes through experiments on benchmark Orange4Home and CASAS datasets. Furthermore, we conduct a detailed analysis of how the individual components of our approach affect downstream activity recognition performance.
Snacking, unlike meals, can happen in short, sporadic bursts, often coupled with distractions, thereby rendering the detection of such episodes a challenging problem. Despite the importance of understanding snacking behavior to monitor the dietary habits of individuals, few eating detection systems report their efficacy for detecting snacking episodes. Furthermore, due to the nature of the study designs, most eating detection systems are suited for detecting long and continuous eating events such as meals. Through a semi-naturalistic user study in the lab with 18 participants, we present our system for detecting short and sporadic chewing episodes, which are present in snacking, passively using Nokia eSense earbuds. We find that IMU data from earbuds have limited generalizability (n=18) for detecting short chewing episodes for unseen participants. Our study unpacks the significance of personalization (n=14) in short and sporadic chewing detection and quantifies the data necessary for developing such systems using a personalized approach.
The ubiquitous availability of wearable sensing platforms renders recording activity data a straightforward endeavor. For many scenarios, however, obtaining accurate annotations for these sensor data can be infeasible due to practical constraints on logistics and costs, as well as concerns regarding privacy and ethics. Thus, we turn to unsupervised clustering which explores techniques which are capable of learning to group data in the absence of ground truth annotations. Our work focuses on learning to cluster movement data by exploiting the consistent patterns which we expect them to contain. This could allow vast amounts of unlabelled sensor data, which can be collected at scale, to be leveraged to bootstrap powerful classifiers. We adapt the Semantic Clustering by Adopting Nearest Neighbors (SCAN) framework to human activity recognition, which allows for the grouping of unlabelled windows of multi-sensor recordings based on the similarity of underlying movements. Using SCAN, we outperform the existing state-of-the-art unsupervised clustering technique for human activity recognition by clear margins on four diverse benchmark datasets.
The prospect of learning effective representations from unlabeled data alone has led to a boost in developing self-supervised learning (SSL) methods for sensor-based Human Activity Recognition (HAR). Typically, (large-scale) unlabeled data are used for pre-training, with the learned weights being used as feature extractors for recognizing activities. While prior works have focused on the impact of increased data scale on performance, instead, we aim to discover the pre-training data efficiency of self-supervised methods. We empirically determine the minimal quantities of unlabeled data required for obtaining comparable performance to using all available data. We investigate three established SSL methods for HAR on three target datasets. Out of these three methods, we discover that Contrastive Predictive Coding (CPC) is the most efficient in terms of pre-training data requirements: just 15 minutes of sensor data across participants is sufficient to obtain competitive activity recognition performance. Further, around 5 minutes of source data is enough when there are sufficient amounts of target application data available. These findings can serve as starting point for more efficient data collection practices.