The essential human gait parameters are briefly reviewed, followed by a detailed review of the state of the art in deep learning for the human gait analysis. The modalities for capturing the gait data are grouped according to the sensing technology: video sequences, wearable sensors, and floor sensors, as well as the publicly available datasets. The established artificial neural network architectures for deep learning are reviewed for each group, and their performance are compared with particular emphasis on the spatiotemporal character of gait data and the motivation for multi-sensor, multi-modality fusion. It is shown that by most of the essential metrics, deep learning convolutional neural networks typically outperform shallow learning models. In the light of the discussed character of gait data, this is attributed to the possibility to extract the gait features automatically in deep learning as opposed to the shallow learning from the handcrafted gait features.

Floor Sensors (FS) are used to capture information from the force induced on the contact surface by feet during gait. On the other hand, the Ambulatory Inertial Sensors (AIS) are used to capture the velocity, acceleration and orientation of the body during different activities. In this paper, fusion of the stated modalities is performed to overcome the challenge of gait classification from wearable sensors on the lower portion of human body not in contact with ground as in FS. Deep learning models are utilized for the automatic feature extraction of the ground reaction force obtained from a set of 116 FS and body movements from AIS attached at 3 different locations of lower body, which is novel. Spatio-temporal information of disproportionate inputs obtained from the two modalities is balanced and fused within deep learning network layers whilst reserving the categorical content for each gait activity. Our approach of fusion compensates the degradation in spatio-temporal accuracies in individual modalities and makes the overall classification outcomes more accurate. Further assessment of multi-modality based results show significant improvements in f-scores using different deep learning models i.e., LSTM (99.90%), 2D-CNN (88.73%), 1D-CNN (94.97%) and ANN (89.33%) respectively.

Human gait has been acquired and studied through modalities such as video cameras, inertial sensors and floor sensors etc. Due to many environmental constraints such as illumination, noise, drifts over extended periods or restricted environment, the classification f-score of gait classifications is highly dependent on the usage scenario. This is addressed in this work by proposing sensor fusion of data obtained from 1) ambulatory inertial sensors (AIS) and 2) plastic optical fiber-based floor sensors (FS). Four gait activities are executed by 11 subjects on FS whilst wearing AIS. The proposed sensor fusion method achieves classification f-scores of 88% using artificial neural network (ANN) and 91% using convolutional neural network (CNN) by learning the best data representations from both modalities.

In a case study of gait classification from floor and ambulatory sensors, we compare results with data from each modality. The automatic extraction of features is achieved by Principle Component Analysis and Canonical Correlation Analysis, the latter performing better even with a reduced number of components used. Non-linear classifiers are most efficient for fused features. With a Kernel Support Vector Machine around 94% accuracy is demonstrated, improving over the 87% and 79% accuracies obtained with separate floor and ambulatory sensor data, respectively.

Gait activity classifications from single-modality data, e.g. acquired by separate vision, pressure, sound and inertial measurements, can be improved by complementary multi-modality fusion to capture a larger set of distinctive gait activity features. We demonstrate a feature-level based sensor fusion of spatio-temporal data obtained from a set of 116 collaborative floor sensors for spatio-temporal sampling of the ground reaction force and ambulatory inertial sensors at 3 positions on the human body. Principle Component Analysis and Canonical Correlation Analysis are used for automatic feature extraction. Fusion at feature level elucidates the balance between otherwise disproportional number of inputs from the two modalities, while reducing the overall number of inputs for classification without degrading substantially the information content. Improvement in the classification is achieved using K-Nearest Neighbor and Kernel Support Vector Machine, manifesting f-scores of 0.95 and 0.94 respectively.