Assistive robots play an important role in improving the quality of life of patients at home. Among all the monitoring tasks, gait disorders are prevalent in elderly and people with neurological conditions and this increases the risk of fall. Therefore, the development of mobile systems for gait monitoring at home in normal living conditions is important. Here, we present a mobile system that is able to track humans and analyze their gait in canonical coordinates based on a single RGB-D camera. First, view-invariant three-dimensional (3-D) lower limb pose estimation is achieved by fusing information from depth images along with 2-D joints derived in RGB images. Next, both the 6-D camera pose and the 3-D lower limb skeleton are real-time tracked in a canonical coordinate system based on simultaneously localization and mapping (SLAM). A mask-based strategy is exploited to improve the re-localization of the SLAM in dynamic environments. Abnormal gait is detected by using the support vector machine and the bidirectional long-short term memory network with respect to a set of extracted gait features. To evaluate the robustness of the system, we collected multi-cameras, ground truth data from 16 healthy volunteers performing 6 gait patterns that mimic common gait abnormalities. The experiment results demonstrate that our proposed system can achieve good lower limb pose estimation and superior recognition accuracy compared to previous abnormal gait detection methods.

For abnormal gait recognition, pattern-specific features indicating abnormalities are interleaved with the subject-specific differences representing biometric traits. Deep representations are, therefore, prone to overfitting, and the models derived cannot generalize well to new subjects. Furthermore, there is limited availability of abnormal gait data obtained from precise Motion Capture (Mocap) systems because of regulatory issues and slow adaptation of new technologies in health care. On the other hand, data captured from markerless vision sensors or wearable sensors can be obtained in home environments, but noises from such devices may prevent the effective extraction of relevant features. To address these challenges, we propose a cascade of deep architectures that can encode cross-modal and cross-subject transfer for abnormal gait recognition. Cross-modal transfer maps noisy data obtained from RGBD and wearable sensors to accurate 4-D representations of the lower limb and joints obtained from the Mocap system. Subsequently, cross-subject transfer allows disentangling subject-specific from abnormal pattern-specific gait features based on a multiencoder autoencoder architecture. To validate the proposed methodology, we obtained multimodal gait data based on a multicamera motion capture system along with synchronized recordings of electromyography (EMG) data and 4-D skeleton data extracted from a single RGBD camera. Classification accuracy was improved significantly in both Mocap and noisy modalities.

Egocentric vision has gained increasing popularity recently, opening new avenues for human-centric applications. However, the use of the egocentric fisheye cameras allows wide angle coverage but image distortion is introduced along with strong human body self-occlusion imposing significant challenges in data processing and model reconstruction. Unlike previous work only leveraging synthetic data for model training, this paper presents a new real-world EgoCentric Human Pose (ECHP) dataset. To tackle the difficulty of collecting 3D ground truth using motion capture systems, we simultaneously collect images from a head-mounted egocentric fisheye camera as well as from two third-person-view cameras, circumventing the environmental restrictions. By using self-supervised learning under multi-view constraints, we propose a simple yet effective framework, namely EgoFish3D, for egocentric 3D pose estimation from a single image in different real-world scenarios. The proposed EgoFish3D incorporates three main modules. 1) The third-person-view module takes two exocentric images as input and estimates the 3D pose represented in the third-person camera frame; 2) the egocentric module predicts the 3D pose in the egocentric camera frame; and 3) the interactive module estimates the rotation matrix between the third-person and the egocentric views. Experimental results on our ECHP dataset and existing benchmark datasets demonstrate the effectiveness of the proposed EgoFish3D, which can achieve superior performance to existing methods.

For gait analysis, especially for the detection of subtle gait abnormalities, the collected datasets involve high variability across subjects due to inherent biometric traits and movement behaviors, leading to limited detection accuracy and poor generalizability. To address this, we propose a novel deep multi-source Unsupervised Domain Adaptation (UDA) approach, namely Maximum Cross-Domain Classifier Discrepancy (MCDCD), which aims to improve the classification performance on the test subject (target domain) by leveraging the information from multiple labelled training subjects (source domains). Specifically, the proposed model consists of a feature extractor and a domain-specific category classifier per source domain. The former feature extractor learns to generate discriminative gait features. For the latter classifiers, we minimize the cross-entropy loss to accurately classify source samples, and simultaneously maximize a novel cross-domain discrepancy loss between any two category classifiers to minimize domain shift between multiple sources and the target domain. To validate the proposed MCDCD for detecting gait abnormalities on novel subjects, we collected both high-quality Motion capture (Mocap) and noisy Electromyography (EMG) data from eighteen subjects with both normal and imitated abnormal gaits. Experiment results using both data modalities demonstrate that the proposed approach can achieve superior performance in abnormal gait classification compared to baseline deep models and state-of-the-art UDA methods.

The integration of machine/deep learning and sensing technologies is transforming healthcare and medical practice. However, inherent limitations in healthcare data, namely scarcity , quality , and heterogeneity , hinder the effectiveness of supervised learning techniques which are mainly based on pure statistical fitting between data and labels. In this paper, we first identify the challenges present in machine learning for pervasive healthcare and we then review the current trends beyond fully supervised learning that are developed to address these three issues. Rooted in the inherent drawbacks of empirical risk minimization that underpins pure fully supervised learning, this survey summarizes seven key lines of learning strategies, to promote the generalization performance for real-world deployment. In addition, we point out several directions that are emerging and promising in this area, to develop data-efficient, scalable, and trustworthy computational models, and to leverage multi-modality and multi-source sensing informatics, for pervasive healthcare.

With the increasing development of brain imaging and sensing technologies, a wide variety of medical signals in multiple modalities have facilitated a better understanding of mental health. The brain-computer interface (BCI) is a technology capable of detecting mental states automatically by employing various brain sensing and machine learning methods. This contributes to efforts involving neurological disease management and restoration of cognitive function. In this study, we present an independent decision path fusion (IDPF) method by developing a bimodal asynchronous BCI based on electroencephalographs (EEGs) and functional near-infrared spectroscopy (fNIRS) to discriminate multiple mental states. The proposed IDPF method generates several independent decision paths, each of which is capable of analyzing cerebral information with respect to a specific aspect, thus interpreting the brain state from multiple points of view. Moreover, in one particular decision path for the fNIRS analysis of the IDPF method, we modified the EEG-based common spatial pattern (CSP) algorithm according to the characteristics of fNIRS. The results show that the modified common spatial pattern (MCSP) significantly outperforms CSP in fNIRS-based BCIs. Through validation on an open-access EEG-fNIRS dataset and comparison with recent studies, we found that our IDPF method achieves a high accuracy of 70.32% for a four-class classification problem (left hand motor imagery, right hand motor imagery, mental arithmetic, and resting state).