In skeleton-based abnormal gait recognition, using original skeleton data decreases the recognition performance because they contain noise and irrelevant information. Instead of feeding original skeletal gait data to a recognition model, features extracted from the skeleton data are normally used. However, existing feature extraction methods might include laborious processes and it is hard for them to minimize the irrelevant information while preserving the important information. To solve this problem, an automatic feature extraction method using a recurrent neural network (RNN)-based Autoencoder (AE) is proposed in this paper. We extracted features from skeletal gait data by using two RNN AEs: a long shortterm memory (LSTM)-based AE (LSTM AE) and a gated recurrent unit (GRU)-based AE (GRU AE). The features of the RNN AEs are compared to the original skeleton data and other existing features. We evaluated the features by feeding them to various discriminative models (DMs) and comparing the recognition performances. The features extracted by using the RNN AEs are more easily recognized and robust than the original skeleton data and other existing features. In particular, the LSTM AE shows a better performance than the GRU AE. Compared to single DMs fed with the original skeleton directly, hybrid models where the features of the RNN AEs are fed to DMs show a higher recognition accuracy with fewer training epochs and learning parameters. Therefore, the proposed automatic feature extraction method improves the performance of skeleton-based abnormal gait recognition by reducing laborious processes and increasing the recognition accuracy effectively.

With the development of depth sensors and skeleton tracking algorithms, many skeleton-based pathological gait classification methods have recently been proposed. However, these methods classify only simple gait patterns, and there is no approach to classify complicated gait patterns. In this paper, we classify 1 normal and 5 pathological gaits (antalgic, stiff-legged, lurching, steppage, and Trendelenburg gaits) by using a gated recurrent unit (GRU)-based classifier and 3D skeleton data. We collected skeleton datasets for 1 normal and 5 pathological gaits by using a multiperspective Kinect system. We developed the GRU classifier to classify the pathological gaits and compared its performance with that of other machine learning-based classifiers. Furthermore, we considered various joint groups to identify important and irrelevant joints for pathological gait classification and to improve the performance of the GRU classifier. When all skeleton data are used, the GRU classifier achieves a classification accuracy of 90.13%. A long short-term memory (LSTM)-based classifier achieves the next highest accuracy of 87.25%. The classification accuracy of the GRU classifier depends on the joint groups considered, and the classification accuracy increases to 93.67% when only leg joints are considered. This study indicates that various pathological gaits can be classified by using skeleton data and the GRU classifier. The proposed method can be used to support medical and clinical decisions. Furthermore, the results for various joint groups can be used to develop other methods in cases where only specific joint data are available because of environmental limitations.

Classification of pathological gaits has an important role in finding a weakened body part and diagnosing a disease. Many machine learning-based approaches have been proposed that automatically classify abnormal gait patterns using various sensors, such as inertial sensors, depth cameras and foot pressure plates. In this paper, we present a deep learning-based abnormal gait classification method employing both a 3D skeleton (obtained with a depth camera) and plantar foot pressure. We collected skeleton and foot pressure data simultaneously for 1 normal and 5 pathological (antalgic, lurching, steppage, stiff-legged, and Trendelenburg) gaits and classified them by using a multimodal hybrid model fed both data types together. In the proposed method, we fed the sequential skeleton and average foot pressure data into recurrent neural network (RNN)-based encoding layers and convolutional neural network (CNN)-based encoding layers, respectively, to effectively extract features from different data types. Their output features were concatenated and fed to fully connected layers for classification. The pressure-based and skeleton-based single-modal models achieved classification accuracies of 68.82% and 93.40%, respectively. The proposed multimodal hybrid model showed improved performance with an accuracy of 95.66%. We fine-tuned the hybrid model by applying a 3-step training methodology and ultimately increased the accuracy to 97.60%. This study indicates that the integrated features of the skeleton and foot pressure data represent both the spatiotemporal motion information and weight distribution, so data fusion can generate a positive effect in pathological gait classification.

In visual speech recognition (VSR), speech is transcribed using only visual information to interpret tongue and teeth movements. Recently, deep learning has shown outstanding performance in VSR, with accuracy exceeding that of lipreaders on benchmark datasets. However, several problems still exist when using VSR systems. A major challenge is the distinction of words with similar pronunciation, called homophones; these lead to word ambiguity. Another technical limitation of traditional VSR systems is that visual information does not provide sufficient data for learning words such as “a”, “an”, “eight”, and “bin” because their lengths are shorter than 0.02 s. This report proposes a novel lipreading architecture that combines three different convolutional neural networks (CNNs; a 3D CNN, a densely connected 3D CNN, and a multi-layer feature fusion 3D CNN), which are followed by a two-layer bi-directional gated recurrent unit. The entire network was trained using connectionist temporal classification. The results of the standard automatic speech recognition evaluation metrics show that the proposed architecture reduced the character and word error rates of the baseline model by 5.681% and 11.282%, respectively, for the unseen-speaker dataset. Our proposed architecture exhibits improved performance even when visual ambiguity arises, thereby increasing VSR reliability for practical applications.

Skeleton data, which is often used in the HCI field, is a data structure that can efficiently express human poses and gestures because it consists of 3D positions of joints. The advancement of RGB-D sensors, such as Kinect sensors, enabled the easy capture of skeleton data from depth or RGB images. However, when tracking a target with a single sensor, there is an occlusion problem causing the quality of invisible joints to be randomly degraded. As a result, multiple sensors should be used to reliably track a target in all directions over a wide range. In this paper, we proposed a new method for combining multiple inaccurate skeleton data sets obtained from multiple sensors that capture a target from different angles into a single accurate skeleton data. The proposed algorithm uses density-based spatial clustering of applications with noise (DBSCAN) to prevent noise-added inaccurate joint candidates from participating in the merging process. After merging with the inlier candidates, we used Kalman filter to denoise the tremble error of the joint’s movement. We evaluated the proposed algorithm’s performance using the best view as the ground truth. In addition, the results of different sizes for the DBSCAN searching area were analyzed. By applying the proposed algorithm, the joint position accuracy of the merged skeleton improved as the number of sensors increased. Furthermore, highest performance was shown when the searching area of DBSCAN was 10 cm.

The identification of attention deficit hyperactivity disorder (ADHD) in children, which is increasing every year worldwide, is very important for early diagnosis and treatment. However, since ADHD is not a simple disease that can be diagnosed with a simple test, doctors require a large period of time and substantial effort for accurate diagnosis and treatment. Currently, ADHD classification studies using various datasets and machine learning or deep learning algorithms are actively being conducted for the screening diagnosis of ADHD. However, there has been no study of ADHD classification using only skeleton data. It was hypothesized that the main symptoms of ADHD, such as distraction, hyperactivity, and impulsivity, could be differentiated through skeleton data. Thus, we devised a game system for the screening and diagnosis of children’s ADHD and acquired children’s skeleton data using five Azure Kinect units equipped with depth sensors, while the game was being played. The game for screening diagnosis involves a robot first travelling on a specific path, after which the child must remember the path the robot took and then follow it. The skeleton data used in this study were divided into two categories: standby data, obtained when a child waits while the robot demonstrates the path; and game data, obtained when a child plays the game. The acquired data were classified using the RNN series of GRU, RNN, and LSTM algorithms; a bidirectional layer; and a weighted cross-entropy loss function. Among these, an LSTM algorithm using a bidirectional layer and a weighted cross-entropy loss function obtained a classification accuracy of 97.82%.