Human activity recognition (HAR) is a popular and challenging research topic, driven by a variety of applications. More recently, with significant progress in the development of deep learning networks for classification tasks, many researchers have made use of such models to recognise human activities in a sensor-based manner, which have achieved good performance. However, sensor-based HAR still faces challenges; in particular, recognising similar activities that only have a different sequentiality and similarly classifying activities with large inter-personal variability. This means that some human activities have large intra-class scatter and small inter-class separation. To deal with this problem, we introduce a margin mechanism to enhance the discriminative power of deep learning networks. We modified four kinds of common neural networks with our margin mechanism to test the effectiveness of our proposed method. The experimental results demonstrate that the margin-based models outperform the unmodified models on the OPPORTUNITY, UniMiB-SHAR, and PAMAP2 datasets. We also extend our research to the problem of open-set human activity recognition and evaluate the proposed method’s performance in recognising new human activities.

Human activity recognition (HAR) using body-worn sensors is an active research area in human-computer interaction and human activity analysis. The traditional methods use hand-crafted features to classify multiple activities, which is both heavily dependent on human domain knowledge and results in shallow feature extraction. Rapid developments in deep learning have caused most researchers to switch to deep learning methods, which extract features from raw data automatically. Most of the existing works on human activity recognition tasks involve multimodal sensor data, and these networks mainly focus on the top representation extracted from bottom-up feedforward process without reusing other features from bottom layers. In this paper, we present a novel hybrid deep learning network for human activity recognition that also employs multimodal sensor data; however, our proposed model is a ConvLSTM pipeline that makes full use of the information in each layer extracted along the temporal domain. Thus, we propose a dense connection module (DCM) to ensure maximum information flow between the network layers. Furthermore, we employ a multilayer feature aggregation module (MFAM) to extract features along the spatial domain, and we aggregate the features obtained from every convolutional layer according to the importance of features in different spatial locations. The output of the MFAM is input into two LSTM layers to further model the temporal dependencies. Finally, a fully connected layer and a softmax function are used to compute the probability of each class. We demonstrate the effectiveness of our proposed model on two benchmark datasets: Opportunity and UniMiB-SHAR. The results illustrate that our designed network outperforms the state-of-the-art models. We also conduct experiments on efficiency, multimodal fusion and different hyperparameters to analyze our proposed network. Finally, we carry out ablation and visualization experiments to reveal the effectiveness of the two proposed modules.

Human activity recognition (HAR) has become a hot research topic, due to its wide application prospect. Fusion-based methods can be used to complement single sensing modality methods. This paper presents the simultaneous utilization of skeleton data and inertial signals—which are captured at the same time using a Kinect depth camera and ten wearable inertial sensors —within a fusion framework, in order to achieve more robust human action recognition, compared to situations where each sensing modality is used individually. Skeleton data captured by the Kinect depth camera are transformed into a weighted front-view skeleton motion map (WF-SMM), a weighted multi-view skeleton motion map (WM-SMM), and a 3D weighted skeleton motion map (3DW-SMM), which are then fed as inputs into a convolutional neural network. Meanwhile, the inertial data are transformed into 2D inertial images, then fed into a 2D dilated convolutional neural network. Two types of fusion are considered: decision-level fusion and feature-level fusion. Experiments were conducted using the publicly available Changzhou University Multimodal Human Action Data set (CZU-MHAD), in which simultaneous skeleton sequence and inertial signals were captured for a total of 22 actions. The results obtained indicate that both the decision- and feature-level fusion approaches generate higher recognition accuracies, compared to the approaches where each sensing modality is used individually. The highest accuracy (of 98.90%) was obtained with the decision-level fusion approach using 3DW-SMM. In addition, some experiments are conducted on the continuous action streams generated based on the CZU-MHAD with different score threshold. And the highest f1 score 82.05 % is obtained with the threshold of 0.4.

Human activity recognition (HAR) based on wearable sensors is a promising research direction. The resources of handheld terminals and wearable devices limit the performance of recognition and require lightweight architectures. With the development of deep learning, the neural architecture search (NAS) has emerged in an attempt to minimize human intervention. We propose an approach for using NAS to search for models suitable for HAR tasks, namely, HARNAS. The multi-objective search algorithm NSGA-II is used as the search strategy of HARNAS. To make a trade-off between the performance and computation speed of a model, the F1 score and the number of floating-point operations (FLOPs) are selected, resulting in a bi-objective problem. However, the computation speed of a model not only depends on the complexity, but is also related to the memory access cost (MAC). Therefore, we expand the bi-objective search to a tri-objective strategy. We use the Opportunity dataset as the basis for most experiments and also evaluate the portability of the model on the UniMiB-SHAR dataset. The experimental results show that HARNAS designed without manual adjustments can achieve better performance than the best model tweaked by humans. HARNAS obtained an F1 score of 92.16% and parameters of 0.32 MB on the Opportunity dataset.

Network traffic classification technology plays an important role in network security management. However, the inherent limitations of traditional methods have become increasingly obvious, and they cannot address existing traffic classification tasks. Very recently, neural architecture search (NAS) has aroused widespread interest as a tool to automate the manual architecture construction process. To this end, this paper proposes NAS based on multiobjective evolutionary algorithms (MOEAs) to classify malicious network traffic. The main purpose is to simplify the search space by reducing the spatial ratio and number of channels of the model. In addition, the search strategy is changed in the effective search space, and the utilized strategies include EAs with the nondominated sorting genetic algorithm with the elite retention strategy (NSGA-II), strength Pareto evolutionary algorithm (SPEA-II) and multiobjective particle swarm optimization (MOPSO) to solve the formulated multiobjective NAS. Through comprehensive comparison of the population convergence times, model accuracies, Pareto optimality sets, model complexities and running speeds of the strategies, it is concluded that the model based on NSGA-II search has the best performance. The experimental results of the current machine learning algorithms and artificial learning methods based on the network are compared, showing that our method achieved better classification performance on two public datasets with a lower computational complexity, as mainly measured by FLOPs. Our approach is able to achieve 99.806% and 99.369% F1-score with 11.501 MB and 4.718 MB FLOPs on both IDS2012 and ISCX VPN dataset respectively.