We introduce a novel approach for robust estimation of physiological parameters such as interbeat interval (IBI) and heart rate variability (HRV) from cardiac signals captured with wearable sensors in the presence of motion artifacts. Motion artifact due to physical exercise is known as a major source of noise that contributes to a significant decline in the performance of IBI and HRV estimation techniques for cardiac monitoring in free-living environments. Therefore, developing robust estimation algorithms is essential for utilization of wearable sensors in daily life situations. The proposed approach includes two algorithmic components. First, we propose a combinatorial technique to select characteristic points that define heartbeats in noisy signals in time domain. The heartbeat detection problem is defined as a shortest path search problem on a direct acyclic graph that leverages morphological features of the cardiac signals by taking advantage of the time-continuity of heartbeats - each heartbeat ends with the starting point of the next heartbeat. The graph is constructed with vertices and edges representing candidate morphological features and IBIs, respectively. Second, we propose a fusion technique to combine physiological parameters estimated from different morphological features using the shortest path algorithm to obtain more accurate IBI/HRV estimations. We evaluate our techniques on motion-corrupted photoplethysmogram and electrocardiogram signals. Our results indicate that the estimated IBIs are highly correlated with the ground truth (r = 0.89) and detected HRV parameters indicate high correlation with the true HRV parameters. Furthermore, our findings demonstrate that the developed fusion technique, which utilizes different morphological features, achieves a correlation coefficient that is at least 3% higher than that obtained using single physiological characteristic.
Robots interacting with humans in assistive contexts have to be sensitive to human cognitive states to be able to provide help when it is needed and not overburden the human when the human is busy. Yet, it is currently still unclear which sensing modality might allow robots to derive the best evidence of human workload. In this work, we analyzed and modeled data from a multi-modal simulated driving study specifically designed to evaluate different levels of cognitive workload induced by various secondary tasks such as dialogue interactions and braking events in addition to the primary driving task. Specifically, we performed statistical analyses of various physiological signals including eye gaze, electroencephalography, and arterial blood pressure from the healthy volunteers and utilized several machine learning methodologies including k-nearest neighbor, naive Bayes, random forest, support-vector machines, and neural network-based models to infer human cognitive workload levels. Our analyses provide evidence for eye gaze being the best physiological indicator of human cognitive workload, even when multiple signals are combined. Specifically, the highest accuracy (in %) of binary workload classification based on eye gaze signals is 80.45 ∓ 3.15 achieved by using support-vector machines, while the highest accuracy combining eye gaze and electroencephalography is only 77.08 ∓ 3.22 achieved by a neural network-based model. Our findings are important for future efforts of real-time workload estimation in the multimodal human-robot interactive systems given that eye gaze is easy to collect and process and less susceptible to noise artifacts compared to other physiological signal modalities.
This paper proposes a novel method to estimate heart rate variability (HRV) and interbeat interval (IBI) in the presence of motion artifacts. A robust heartbeat selection method is introduced as a shortest path exploration problem to detect HRV/IBI from a noisy signal. The time-continuity of heartbeats is used to model the shortest path. The end point of one heartbeat is the starting point of the next one. A graph is constructed whose vertices and edges are assumed to be the beginning of each possible heartbeats and the IBIs, respectively. Weights are assigned to the edges by using average heart rate (HR) values from state-of-the-art algorithms. The results indicate that IBI percentage error is around 3% for distorted ECG signal even with 0dB SNR. In addition, HRV parameters of clean versus noisy signals show high correlation.
In recent years, there has been an increasing interest in human-machine teaming for search and rescue operations, deep space missions, and agricultural tasks, among others. To be effective teammates, artificial agents should be able to detect and be responsive to systemic human cognitive states such as workload, sense of urgency, mind wandering, interference, and distraction. Here, we introduce an experimental paradigm and a comprehensive multimodal dataset that provides the necessary data for analyzing the relationships among multiple systematic human cognitive states, enables the development of robust prediction models of these states, and details the framework for developing new experiments. The introduced experimental setup allows for the synchronized real-time recording of multiple data streams from various sensing devices including fNIRS, EEG, pupillometry, respiration, electrodermal activity, and plethysmography, and is applicable in many other interactive task settings where human performance needs to be monitored. The dataset was acquired from 80 subjects performing a driving task and several secondary tasks including car braking events, dialogue communications, and tactile stimulations.
Robust estimation of systemic human cognitive states is critical for many applications, from simply detecting inefficiencies in human task performance to adapting the behaviors of artificial agents to improve team performance in mixed-initiative human-machine teams. Here we use comprehensive analyses of a multi-modal dataset from a multi-tasking driving experiment to provide systematic evidence that percentage change in pupil size (PCPS) in human eye gaze is the most reliable biomarker for assessing three human cognitive states relevant to task performance: workload, sense of urgency, and mind wandering. Specifically, we performed comprehensive statistical tests that establish the superior performance of PCPS compared other physiological signals like electroencephalogram (EEG), functional near-infrared spectroscopy (fNIRS), respiration, arterial blood pressure (ABP), and skin conductance. Based on the eye gaze data, we also characterized the relation between workload and sense of urgency, observing that consecutive occurrences of higher sense of urgency tended to increase overall workload. We also trained five state-of-the-art machine learning models on the data to determine their potential for predicting the three systemic cognitive states across subjects, showing that four of them had similar highest accuracy in cognitive state classification based on PCPS (with one, random forest, showing inferior performance). In combination with the statistical analyses, the comparative modeling results demonstrate that PCPS, in contrast to prior findings in the literature, represents the optimal method for estimating the three performance-relevant cognitive states.