In skeleton-based abnormal gait recognition, using original skeleton data decreases the recognition performance because they contain noise and irrelevant information. Instead of feeding original skeletal gait data to a recognition model, features extracted from the skeleton data are normally used. However, existing feature extraction methods might include laborious processes and it is hard for them to minimize the irrelevant information while preserving the important information. To solve this problem, an automatic feature extraction method using a recurrent neural network (RNN)-based Autoencoder (AE) is proposed in this paper. We extracted features from skeletal gait data by using two RNN AEs: a long shortterm memory (LSTM)-based AE (LSTM AE) and a gated recurrent unit (GRU)-based AE (GRU AE). The features of the RNN AEs are compared to the original skeleton data and other existing features. We evaluated the features by feeding them to various discriminative models (DMs) and comparing the recognition performances. The features extracted by using the RNN AEs are more easily recognized and robust than the original skeleton data and other existing features. In particular, the LSTM AE shows a better performance than the GRU AE. Compared to single DMs fed with the original skeleton directly, hybrid models where the features of the RNN AEs are fed to DMs show a higher recognition accuracy with fewer training epochs and learning parameters. Therefore, the proposed automatic feature extraction method improves the performance of skeleton-based abnormal gait recognition by reducing laborious processes and increasing the recognition accuracy effectively.

With the development of depth sensors and skeleton tracking algorithms, many skeleton-based pathological gait classification methods have recently been proposed. However, these methods classify only simple gait patterns, and there is no approach to classify complicated gait patterns. In this paper, we classify 1 normal and 5 pathological gaits (antalgic, stiff-legged, lurching, steppage, and Trendelenburg gaits) by using a gated recurrent unit (GRU)-based classifier and 3D skeleton data. We collected skeleton datasets for 1 normal and 5 pathological gaits by using a multiperspective Kinect system. We developed the GRU classifier to classify the pathological gaits and compared its performance with that of other machine learning-based classifiers. Furthermore, we considered various joint groups to identify important and irrelevant joints for pathological gait classification and to improve the performance of the GRU classifier. When all skeleton data are used, the GRU classifier achieves a classification accuracy of 90.13%. A long short-term memory (LSTM)-based classifier achieves the next highest accuracy of 87.25%. The classification accuracy of the GRU classifier depends on the joint groups considered, and the classification accuracy increases to 93.67% when only leg joints are considered. This study indicates that various pathological gaits can be classified by using skeleton data and the GRU classifier. The proposed method can be used to support medical and clinical decisions. Furthermore, the results for various joint groups can be used to develop other methods in cases where only specific joint data are available because of environmental limitations.

Existing methods for fall detection may not detect a fall when it occurs or may generate a false alarm when a fall does not occur. In order to overcome these limitations and detect falls with 100% accuracy, a double-check method for fall detection in elderly people via an inertial measurement unit-location (IMU-L) sensor and a red-green-blue (RGB) camera is proposed. The IMU-L sensor is a combination of an IMU sensor (accelerometer and gyroscope) and an ultrawideband signal-based location sensor; the RGB sensor is mounted on a robot. The proposed method involves detecting and confirming the fall of an elderly individual via the IMU-L sensor and an RGB image, respectively. The IMU-L sensor is worn on the body to detect falls. When a potential fall occurs, the individual's location information is synchronized with the motion data. During detection, because of the sequential nature of IMU data, a deep learning technique called a recurrent neural network (RNN) is trained to classify falls. When the IMU indicates a suspected fall situation, the robot moves to the corresponding location and confirms whether a fall has occurred. During the confirmation stage, a convolutional neural network-based technique is applied to the RGB image data to recognize and confirm the fall. Repeated confirmed fall detections using this method classified falls more accurately than existing methods that use only an IMU sensor. We conducted a real-time experiment to validate our method using a dataset developed in a laboratory and achieved 100% accuracy in our experimental environment.

Classification of pathological gaits has an important role in finding a weakened body part and diagnosing a disease. Many machine learning-based approaches have been proposed that automatically classify abnormal gait patterns using various sensors, such as inertial sensors, depth cameras and foot pressure plates. In this paper, we present a deep learning-based abnormal gait classification method employing both a 3D skeleton (obtained with a depth camera) and plantar foot pressure. We collected skeleton and foot pressure data simultaneously for 1 normal and 5 pathological (antalgic, lurching, steppage, stiff-legged, and Trendelenburg) gaits and classified them by using a multimodal hybrid model fed both data types together. In the proposed method, we fed the sequential skeleton and average foot pressure data into recurrent neural network (RNN)-based encoding layers and convolutional neural network (CNN)-based encoding layers, respectively, to effectively extract features from different data types. Their output features were concatenated and fed to fully connected layers for classification. The pressure-based and skeleton-based single-modal models achieved classification accuracies of 68.82% and 93.40%, respectively. The proposed multimodal hybrid model showed improved performance with an accuracy of 95.66%. We fine-tuned the hybrid model by applying a 3-step training methodology and ultimately increased the accuracy to 97.60%. This study indicates that the integrated features of the skeleton and foot pressure data represent both the spatiotemporal motion information and weight distribution, so data fusion can generate a positive effect in pathological gait classification.

Skeleton data, which is often used in the HCI field, is a data structure that can efficiently express human poses and gestures because it consists of 3D positions of joints. The advancement of RGB-D sensors, such as Kinect sensors, enabled the easy capture of skeleton data from depth or RGB images. However, when tracking a target with a single sensor, there is an occlusion problem causing the quality of invisible joints to be randomly degraded. As a result, multiple sensors should be used to reliably track a target in all directions over a wide range. In this paper, we proposed a new method for combining multiple inaccurate skeleton data sets obtained from multiple sensors that capture a target from different angles into a single accurate skeleton data. The proposed algorithm uses density-based spatial clustering of applications with noise (DBSCAN) to prevent noise-added inaccurate joint candidates from participating in the merging process. After merging with the inlier candidates, we used Kalman filter to denoise the tremble error of the joint’s movement. We evaluated the proposed algorithm’s performance using the best view as the ground truth. In addition, the results of different sizes for the DBSCAN searching area were analyzed. By applying the proposed algorithm, the joint position accuracy of the merged skeleton improved as the number of sensors increased. Furthermore, highest performance was shown when the searching area of DBSCAN was 10 cm.

The identification of attention deficit hyperactivity disorder (ADHD) in children, which is increasing every year worldwide, is very important for early diagnosis and treatment. However, since ADHD is not a simple disease that can be diagnosed with a simple test, doctors require a large period of time and substantial effort for accurate diagnosis and treatment. Currently, ADHD classification studies using various datasets and machine learning or deep learning algorithms are actively being conducted for the screening diagnosis of ADHD. However, there has been no study of ADHD classification using only skeleton data. It was hypothesized that the main symptoms of ADHD, such as distraction, hyperactivity, and impulsivity, could be differentiated through skeleton data. Thus, we devised a game system for the screening and diagnosis of children’s ADHD and acquired children’s skeleton data using five Azure Kinect units equipped with depth sensors, while the game was being played. The game for screening diagnosis involves a robot first travelling on a specific path, after which the child must remember the path the robot took and then follow it. The skeleton data used in this study were divided into two categories: standby data, obtained when a child waits while the robot demonstrates the path; and game data, obtained when a child plays the game. The acquired data were classified using the RNN series of GRU, RNN, and LSTM algorithms; a bidirectional layer; and a weighted cross-entropy loss function. Among these, an LSTM algorithm using a bidirectional layer and a weighted cross-entropy loss function obtained a classification accuracy of 97.82%.