The feature extraction of human activity recognition (HAR) based on sensor data has been studied as a hand-crafted method. The significant feature extraction ability is a key factor in improving the accuracy of HAR. Recently, deep learning methods have been employed for feature extraction. In this paper, we review previous studies on deep learning methods in HAR and discuss suitable models for feature extraction. First, we applied various convolutional neural networks to clarify the effective architecture for HAR. Afterward, we developed advanced models by embedding submodules, such as self-attention and recurrent neural networks, often adopted in recent studies. Comparative experiments on HASC, UCI, and WISDM public datasets showed that Inception-V3, which used cross-channel multi-size convolution transformation, outperformed other backbones. Through comparative experiments after embedding submodules, submodules do not always have a positive effect on accuracy. Compared with other submodules, SENet has a positive effect. We conclude that it is essential to select an appropriate backbone model before applying the submodules, and submodules are unnecessary in some cases.

Transfer learning (TL) is an essential technique for human activity recognition (HAR) since collecting a large amount of labeled sensor data is cost-intensive. In the image recognition field, parameter-based TL using ImageNet has become the defacto standard; however, no multipurpose TL method has been established in HAR thus far. Therefore, we proposed a simple and novel technique, i.e., class augmentation (CA), which augments the diversity of the class categories of source domains based on the hypothesis that HAR datasets used as source domains have low diversity in class categories. Data augmentation (DA) enhances the diversity of input data, whereas CA enhances the diversity of output labels. We evaluated three types of CA using DA and HAR-specific auxiliary information (gender and sensor position). Our experiments using public HAR datasets revealed that using CA seems to enhance the transfer performance. Moreover, we confirmed that CA closes the interdomain distance between the source and target domains.

In sensor-based human activity recognition, many methods based on convolutional neural networks (CNNs) have been proposed. In the typical CNN-based activity recognition model, each class is treated independently of others. However, actual activity classes often have hierarchical relationships. It is important to consider an activity recognition model that uses the hierarchical relationship among classes to improve recognition performance. In image recognition, branch CNNs (B-CNNs) have been proposed for classification using class hierarchies. B-CNNs can easily perform classification using hand-crafted class hierarchies, but it is difficult to manually design an appropriate class hierarchy when the number of classes is large or there is little prior knowledge. Therefore, in our study, we propose a class hierarchy-adaptive B-CNN, which adds a method to the B-CNN for automatically constructing class hierarchies. Our method constructs the class hierarchy from training data automatically to effectively train the B-CNN without prior knowledge. We evaluated our method on several benchmark datasets for activity recognition. As a result, our method outperformed standard CNN models without considering the hierarchical relationship among classes. In addition, we confirmed that our method has performance comparable to a B-CNN model with a class hierarchy based on human prior knowledge.