Remote photo-plethysmography (rPPG) has emerged as a non-intrusive and promising physiological sensing capability in human–computer interface (HCI) research, gradually extending its applications in health-monitoring and clinical care contexts. With advanced machine learning models, recent datasets collected in real-world conditions have gradually enhanced the performance of rPPG methods in recovering heart-rate and heart-rate-variability metrics. However, the signal quality of reference ground-truth PPG data in existing datasets is by and large neglected, while poor-quality references negatively influence models. Here, this work introduces a new imaging blood volume pulse (iBVP) dataset of synchronized RGB and thermal infrared videos with ground-truth PPG signals from ear with their high-resolution-signal-quality labels, for the first time. Participants perform rhythmic breathing, head-movement, and stress-inducing tasks, which help reflect real-world variations in psycho-physiological states. This work conducts dense (per sample) signal-quality assessment to discard noisy segments of ground-truth and corresponding video frames. We further present a novel end-to-end machine learning framework, iBVPNet, that features an efficient and effective spatio-temporal feature aggregation for the reliable estimation of BVP signals. Finally, this work examines the feasibility of extracting BVP signals from thermal video frames, which is under-explored. The iBVP dataset and source codes are publicly available for research use.

The proliferation of physiological sensors opens new opportunities to explore interactions, conduct experiments and evaluate the user experience with continuous monitoring of bodily functions. Commercial devices, however, can be costly or limit access to raw waveform data, while low-cost sensors are efforts-intensive to setup. To address these challenges, we introduce PhysioKit, an open-source, low-cost physiological computing toolkit. PhysioKit provides a one-stop pipeline consisting of (i) a sensing and data acquisition layer that can be configured in a modular manner per research needs, and (ii) a software application layer that enables data acquisition, real-time visualization and machine learning (ML)-enabled signal quality assessment. This also supports basic visual biofeedback configurations and synchronized acquisition for co-located or remote multi-user settings. In a validation study with 16 participants, PhysioKit shows strong agreement with research-grade sensors on measuring heart rate and heart rate variability metrics data. Furthermore, we report usability survey results from 10 small-project teams (44 individual members in total) who used PhysioKit for 4–6 weeks, providing insights into its use cases and research benefits. Lastly, we discuss the extensibility and potential impact of the toolkit on the research community.

Remote Photoplethysmography (rPPG) has emerged as a non-intrusive and promising physiological sensing capability in HCI research, gradually extending its applications in health-monitoring and clinical care contexts. With advanced machine learning models, recent datasets collected in real-world conditions have gradually enhanced the performance of rPPG methods in recovering heart-rate and heart-rate variability metrics. However, the signal quality of reference ground-truth PPG data in existing datasets is by and large neglected, while poor quality references negatively influence models. Here, this work introduces a new imaging blood volume pulse (iBVP) dataset of synchronized RGB and thermal infrared videos with PPG ground-truth signals from the ear and its high resolution signal quality labels, for the first time. Participants perform rhythmic breathing, head-movement, and stress-inducing tasks, which help reflect real-world variations in psycho-physiological states. This work conducts dense (per sample) signal quality assessment to discard noisy segments of ground-truth and corresponding video frames. We further present a novel end-to-end machine learning framework, iBVPNet that features an efficient and effective spatio-temporal feature aggregation for reliable estimation of BVP signals. Finally, this work examines the feasibility of extracting BVP signals from thermal video frames, which is underexplored. The iBVP dataset and source codes are publicly available for research use.

The proliferation of physiological sensors opens new opportunities to explore interactions, conduct experiments and evaluate the user experience with continuous monitoring of bodily functions. Commercial devices, however, can be costly or limit access to raw waveform data, while low-cost sensors are efforts-intensive to setup. To address these challenges, we introduce PhysioKit, an open-source, low-cost physiological computing toolkit. PhysioKit consists of (i) a sensor and hardware layer that can be configured in a modular manner per research needs, (ii) a software application layer that enables data acquisition, real-time visualization and machine learning (ML)-enabled signal quality assessment. This also supports basic visual biofeedback configurations and synchronized acquisition for co-located or remote multi-user settings. In a validation study for photo-plethysmography (PPG) sensor with 15 participants, PhysioKit shows strong agreement with research-grade PPG sensors. Furthermore, we report usability survey results from 10 micro-project teams (44 individual members in total) who use PhysioKit for 4-6 weeks, providing insights into its use cases and research benefits. We conclude by discussing extensibility and potential impact of the toolkit on the research community.

Segmentation of thermal facial images is a challenging task. This is because facial features often lack salience due to high-dynamic thermal range scenes and occlusion issues. Limited availability of datasets from unconstrained settings further limits the use of the state-of-the-art segmentation networks, loss functions and learning strategies which have been built and validated for RGB images. To address the challenge, we propose Self-Adversarial Multi-scale Contrastive Learning (SAM-CL) framework as a new training strategy for thermal image segmentation. SAM-CL framework consists of a SAM-CL loss function and a thermal image augmentation (TiAug) module as a domain-specific augmentation technique. We use the Thermal-Face-Database to demonstrate effectiveness of our approach. Experiments conducted on the existing segmentation networks (UNET, Attention-UNET, DeepLabV3 and HRNetv2) evidence the consistent performance gains from the SAM-CL framework. Furthermore, we present a qualitative analysis with UBComfort and DeepBreath datasets to discuss how our proposed methods perform in handling unconstrained situations.

Examining remote virtual audiences is a vital part of understanding social experiences in modern human–computer interaction contexts. Doing so raises intriguing questions about how these mediated connections relate to emotional and physiological states. The vagus nerve is central to socioemotional processing and physiological well-being, with its activation represented as vagally mediated heart rate variability (vmHRV). We examine how participants’ social connectedness to virtual partners relates to their experience of socioemotional competences and psychophysiological states while observing streamed gameplay. In this experimental study with 48 participants, we compared self-reported empathy, empathic concern, and continuously measured vmHRV (from a PPG sensor) during different types of gameplay. The results revealed that viewers who felt greater social connectedness to remote partners also felt more empathic concern (quantitative detail) and had significantly heightened vmHRV (quant detail) across all conditions compared to those who felt lower connectedness. These findings reveal that stronger feelings of connectedness to remote partners are associated with enhanced socioemotional competences and physiological well-being. This research highlights the intertwined nature of social connectedness, empathy, and physiological health, providing valuable insights for designing virtual platforms that foster deeper interpersonal connections and promote well-being.