We present a novel longitudinal multimodal corpus of physiological and behavioral data collected from direct clinical providers in a hospital workplace. We designed the study to investigate the use of off-the-shelf wearable and environmental sensors to understand individual-specific constructs such as job performance, interpersonal interaction, and well-being of hospital workers over time in their natural day-to-day job settings. We collected behavioral and physiological data fromn = 212 participants through Internet-of-Things Bluetooth data hubs, wearable sensors (including a wristband, a biometrics-tracking garment, a smartphone, and an audio-feature recorder), together with a battery of surveys to assess personality traits, behavioral states, job performance, and well-being over time. Besides the default use of the data set, we envision several novel research opportunities and potential applications, including multi-modal and multi-task behavioral modeling, authentication through biometrics, and privacy-aware and privacy-preserving machine learning.
4 July 2022 in IEEE Transactions on Affective Computing
Given the widespread adverse outcomes of stress-exacerbated by the current pandemic-wearable sensing provides unique opportunities for automated stress tracking to inform well-being interventions. However, its success in the wild and at scale depends on the robustness and validity of automated stress inference, which is limited in current systems. In this work, we enumerate the properties of robustness and validity necessary for achieving viable automated stress inference using wearable sensors, and we underscore present challenges to constructing and evaluating these systems. Using these criteria as guiding principles, we present automated stress inference results from a large (N=606) in situ longitudinal wearable and contextual sensing study of information workers. Using a multimodal approach encompassing a wearable sensor, relative location tracking, smartphone usage, and environmental sensing, we trained regression models to predict daily self-reported perceived stress in a participant-independent fashion. Our models significantly outperformed baseline variants with shuffled stress scores and were consistent with small-to-moderate effects. Our findings highlight the performance disparity between robust and valid approaches to automated perceived stress inference and current approaches and suggest that further performance gains might require additional sensing modalities and enhanced contextual awareness than existing approaches.
20 August 2019 in Journal of Medical Internet Research
Background: Recent advances in mobile technologies for sensing human biosignals are empowering researchers to collect real-world data outside of the laboratory, in natural settings where participants can perform their daily activities with minimal disruption. These new sensing opportunities usher a host of challenges and constraints for both researchers and participants. Objective: This viewpoint paper aims to provide a comprehensive guide to aid research teams in the selection and management of sensors before beginning and while conducting human behavior studies in the wild. The guide aims to help researchers achieve satisfactory participant compliance and minimize the number of unexpected procedural outcomes. Methods: This paper presents a collection of challenges, consideration criteria, and potential solutions for enabling researchers to select and manage appropriate sensors for their research studies. It explains a general data collection framework suitable for use with modern consumer sensors, enabling researchers to address many of the described challenges. In addition, it provides a description of the criteria affecting sensor selection, management, and integration that researchers should consider before beginning human behavior studies involving sensors. On the basis of a survey conducted in mid-2018, this paper further illustrates an organized snapshot of consumer-grade human sensing technologies that can be used for human behavior research in natural settings. Results: The research team applied the collection of methods and criteria to a case study aimed at predicting the well-being of nurses and other staff in a hospital. Average daily compliance for sensor usage measured by the presence of data exceeding half the total possible hours each day was about 65%, yielding over 355,000 hours of usable sensor data across 212 participants. A total of 6 notable unexpected events occurred during the data collection period, all of which had minimal impact on the research project. Conclusions: The satisfactory compliance rates and minimal impact of unexpected events during the case study suggest that the challenges, criteria, methods, and mitigation strategies presented as a guide for researchers are helpful for sensor selection and management in longitudinal human behavior studies in the wild.
Engagement is critical to satisfaction and performance in a number of domains but is challenging to measure and sustain. Thus, there is considerable interest in developing affective computing technologies to automatically measure and enhance engagement, especially in the wild and at scale. This article provides an accessible introduction to affective computing research on engagement detection and enhancement using educational applications as an application domain. We begin with defining engagement as a multicomponential construct (i.e., a conceptual entity) situated within a context and bounded by time and review how the past six years of research has conceptualized it. Next, we examine traditional and affective computing methods for measuring engagement and discuss their relative strengths and limitations. Then, we move to a review of proactive and reactive approaches to enhancing engagement toward improving the learning experience and outcomes. We underscore key concerns in engagement measurement and enhancement, especially in digitally enhanced learning contexts, and conclude with several open questions and promising opportunities for future work.
The TILES-2019 data set consists of behavioral and physiological data gathered from 57 medical residents (i.e., trainees) working in an intensive care unit (ICU) in the United States. The data set allows for the exploration of longitudinal changes in well-being, teamwork, and job performance in a demanding environment, as residents worked in the ICU for three weeks. Residents wore a Fitbit, a Bluetooth-based proximity sensor, and an audio-feature recorder. They completed daily surveys and interviews at the beginning and end of their rotation. In addition, we collected data from environmental sensors (i.e., Internet-of-Things Bluetooth data hubs) and obtained hospital records (e.g., patient census) and residents’ job evaluations. This data set may be may be of interest to researchers interested in workplace stress, group dynamics, social support, the physical and psychological effects of witnessing patient deaths, predicting survey data from sensors, and privacy-aware and privacy-preserving machine learning. Notably, a small subset of the data was collected during the first wave of the COVID-19 pandemic.