Considerable progress has been made in improving the estimation accuracy of cognitive workload using various sensor technologies. However, the overall performance of different algorithms and methods remain suboptimal in real-world applications. Some studies in the literature demonstrate that a single modality is sufficient to estimate cognitive workload. These studies are limited to controlled settings, a scenario that is significantly different from the real world where data gets corrupted, interrupted, and delayed. In such situations, the use of multiple modalities is needed. Multimodal fusion approaches have been successful in other domains, such as wireless-sensor networks, in addressing single-sensor weaknesses and improving information quality/accuracy. These approaches are inherently more reliable when a data source is lost. In the cognitive workload literature, sensors, such as electroencephalography (EEG), electrocardiography (ECG), and eye tracking, have shown success in estimating the aspects of cognitive workload. Multimodal approaches that combine data from several sensors together can be more robust for real-time measurement of cognitive workload. In this article, we review the published studies related to multimodal data fusion to estimate the cognitive workload and synthesize their main findings. We identify the opportunities for designing better multimodal fusion systems for cognitive workload modeling.

Pain is a highly unpleasant sensory and emotional experience, and no objective diagnosis test exists to assess it. In clinical practice there are two main methods for the estimation of pain, a patient’s self-report and clinical judgement. However, these methods are highly subjective and the need of biomarkers to measure pain is important to improve pain management, reduce risk factors, and contribute to a more objective, valid, and reliable diagnosis. Therefore, in this study we propose the use of functional near-infrared spectroscopy (fNIRS) and machine learning for the identification of a possible biomarker of pain. We collected pain information from 18 volunteers using the thermal test of the quantitative sensory testing (QST) protocol, according to temperature level (cold and hot) and pain intensity (low and high). Feature extraction was completed in three different domains (time, frequency, and wavelet), and a total of 69 features were obtained. Feature selection was carried out according to three criteria, information gain (IG), joint mutual information (JMI), and Chi-squared (χ²). The significance of each feature ranking was evaluated using three learning models separately, linear discriminant analysis (LDA), the K-nearest neighbour (K-NN) and support vector machines (SVM) using the linear and Gaussian and polynomial kernels. The results showed that the Gaussian SVM presented the highest accuracy (94.17%) using only 25 features to identify the four types of pain in our database. In addition, we propose the use of the top 13 features according to the JMI criteria, which exhibited an accuracy of 89.44%, as promising biomarker of pain. This study contributes to the idea of developing an objective assessment of pain and proposes a potential biomarker of human pain using fNIRS.

This paper compares the three levels of data fusion with the goal of determining the optimal level of data fusion for multi-sensor human activity data. Using the data processing pipeline, gyroscope and accelerometer data was fused at the sensor-level, feature-level and decision-level. For each level of data fusion four different techniques were used with varying levels of success. This analysis was performed on four human activity publicly-available datasets along with four well-known machine learning classifiers to validate the results. The decision-level fusion (Acc = 0.7443±0.0850) out performed the other two levels of fusion in regards to accuracy, sensor level (Acc = 0.5934±0.1110) and feature level (Acc = 0.6742± 0.0053), but, the processing time and computational power required for training and classification were far greater than practical for a HAR system. However, Kalman filter appear to be the more efficient method, since it exhibited both good accuracy (Acc = 0.7536 ± 0.1566) and short processing time (time = 61.71ms ± 63.85); properties that play a large role in real-time applications using wearable devices. The results of this study also serve as baseline information in the HAR literature to compare future methods of data fusion.

Pain is a complex and personal experience that presents diverse measurement challenges. Different sensing technologies can be used as a surrogate measure of pain to overcome these challenges. The objective of this review is to summarise and synthesise the published literature to: (a) identify relevant non-invasive physiological sensing technologies that can be used for the assessment of human pain, (b) describe the analytical tools used in artificial intelligence (AI) to decode pain data collected from sensing technologies, and (c) describe the main implications in the application of these technologies. A literature search was conducted in July 2022 to query PubMed, Web of Sciences, and Scopus. Papers published between January 2013 and July 2022 are considered. Forty-eight studies are included in this literature review. Two main sensing technologies (neurological and physiological) are identified in the literature. The sensing technologies and their modality (unimodal or multimodal) are presented. The literature provided numerous examples of how different analytical tools in AI have been applied to decode pain. This review identifies different non-invasive sensing technologies, their analytical tools, and the implications for their use. There are significant opportunities to leverage multimodal sensing and deep learning to improve accuracy of pain monitoring systems. This review also identifies the need for analyses and datasets that explore the inclusion of neural and physiological information together. Finally, challenges and opportunities for designing better systems for pain assessment are also presented.

Pain assessment is a challenging task encountered by clinicians. In clinical settings, patients’ self-report is considered the gold standard in pain assessment. However, patients who are unable to self-report pain are at a higher risk of undiagnosed pain. In the present study, we explore the use of multiple sensing technologies to monitor physiological changes that can be used as a proxy for objective measurement of acute pain. Electrodermal activity (EDA), photoplethysmography (PPG), and respiration (RESP) signals were collected from 22 participants under two pain intensities (low and high) and on two different anatomical locations (forearm and hand). Three machine learning models were implemented, including support vector machines (SVM), decision trees (DT), and linear discriminant analysis (LDA) for the identification of pain. Various pain scenarios were investigated, identification of pain (no pain, pain), multiclass (no pain, low pain, high pain), and identification of pain location (forearm, hand). Reference classification results from individual sensors and from all sensors together were obtained. After feature selection, results showed that EDA was the most informative sensor in the three pain conditions, 93.2±8% in identification of pain, 68.9±10% in the multiclass problem, and 56.0±8% for the identification of pain location. These results identify EDA as the superior sensor in our experimental conditions. Future work is required to validate the obtained features to improve its feasibility in more realistic scenarios. Finally, this study proposes EDA as a candidate to design a tool that can assist clinicians in the assessment of acute pain of nonverbal patients.

Acupuncture is a practice of treatment based on influencing specific points on the body by inserting needles. According to traditional Chinese medicine, the aim of acupuncture treatment for pain management is to use specific acupoints to relieve excess, activate qi (or vital energy), and improve blood circulation. In this context, the Hegu point is one of the most widely-used acupoints for this purpose, and it has been linked to having an analgesic effect. However, there exists considerable debate as to its scientific validity. In this pilot study, we aim to identify the functional connectivity related to the three main types of acupuncture manipulations and also identify an analgesic effect based on the hemodynamic response as measured by functional near-infrared spectroscopy (fNIRS). The cortical response of eleven healthy subjects was obtained using fNIRS during an acupuncture procedure. A multiscale analysis based on wavelet transform coherence was employed to assess the functional connectivity of corresponding channel pairs within the left and right somatosensory region. The wavelet analysis was focused on the very-low frequency oscillations (VLFO, 0.01–0.08 Hz) and the low frequency oscillations (LFO, 0.08–0.15 Hz). A mixed model analysis of variance was used to appraise statistical differences in the wavelet domain for the different acupuncture stimuli. The hemodynamic response after the acupuncture manipulations exhibited strong activations and distinctive cortical networks in each stimulus. The results of the statistical analysis showed significant differences ( $p<0.05$ ) between the tasks in both frequency bands. These results suggest the existence of different stimuli-specific cortical networks in both frequency bands and the anaesthetic effect of the Hegu point as measured by fNIRS.