Nowadays, deep learning has achieved remarkable results in many computer vision related tasks, among which the support of big data is essential. In this paper, we propose a full stage data augmentation framework to improve the accuracy of deep convolutional neural networks, which can also play the role of implicit model ensemble without introducing additional model training costs. Simultaneous data augmentation during training and testing stages can ensure network optimization and enhance its generalization ability. Augmentation in two stages needs to be consistent to ensure the accurate transfer of specific domain information. Furthermore, this framework is universal for any network architecture and data augmentation strategy and therefore can be applied to a variety of deep learning based tasks. Finally, experimental results about image classification on the coarse-grained dataset CIFAR-10 (93.41%) and fine-grained dataset CIFAR-100 (70.22%) demonstrate the effectiveness of the framework by comparing with state-of-the-art results.

With the widespread deployment of surveillance cameras, automatic violence detection has attracted extensive attention from industry and academia. Though researchers have made great progress in video-based violence detection, it is still a challenging task to realize accurate violence detection in real time, especially with limited computing resources. In this paper, we propose a lightweight 2D CNN-based violence detection scheme, which takes advantage of frame-grouping to reduce data redundancy greatly and, meanwhile, enable short-term temporal modeling. In particular, a lightweight 2D CNN, named improved EfficientNet-B0, is constructed by integrating our proposed bi-directional long-term motion attention (Bi-LTMA) module and a temporal shift module (TSM) into the original EfficientNet-B0. The Bi-LTMA takes both spatial and channel dimensions into consideration and captures motion features in both forward and backward directions. The TSM is adopted to realize temporal feature interaction. Moreover, an auxiliary classifier is designed and employed to improve the classification capability and generalization performance of the proposed model. Experiment results demonstrate that the computational cost of the proposed model is 1.21 GFLOPS. Moreover, the proposed scheme achieves accuracies of 100%, 98.5%, 91.67%, and 90.25% on the Movie Fight dataset, the Hockey Fight dataset, the Surveillance Camera dataset, and the RWF-2000 dataset, respectively.

Recognizing emotional states from speech is essential for human-computer interaction. It is a challenging task to realize effective speech emotion recognition (SER) on platforms with limited memory capacity and computing power. In this paper, we propose a lightweight multi-scale deep neural network architecture for SER, which takes Mel Frequency Cepstral Coefficients (MFCCs) as input. In order to realize effective multi-scale feature extraction, we propose a new Inception module, named A_Inception. A_Inception combines the merits of Inception module and attention-based rectified linear units (AReLU) and thus can learn multi-scale features adaptively with low computational cost. Meanwhile, to extract most important emotional information, we propose a new multiscale cepstral attention and temporal-cepstral attention (MCA-TCA) module. The idea of MCA-TCA module is to focus on the key cepstral components and the key temporal-cepstral positions. Furthermore, a loss function combining Softmax loss and Center loss is adopted to supervise the model training so as to enhance the model’s discriminative power. Experiments have been carried out on IEMOCAP, EMODB and SAVEE datasets to verify the performance of the proposed model and compare with the state-of-the-art SER models. Numerical results reveal that the proposed model has a small number of parameters (0.82M) and much lower computational cost (81.64 MFLOPs) than compared models, and achieves impressive accuracy on all datasets considered.

Facial expression recognition (FER) is an important task in human emotion analysis. Though researchers have made great progress in FER, the semantic information lying in facial landmarks has not been fully exploited. In this paper, we propose a landmark-based adaptive graph convolutional network (LBAGCN) for FER. It takes facial landmarks as input in both phases of model training and inference and thus preserves sensitive privacy information well. Based on the facial landmark sequence extracted from a video clip, we define landmark motion sequence, muscle sequence and muscle motion sequence. Correspondingly, we construct four facial graphs with learnable topologies, named landmark graph, landmark motion graph, muscle graph and muscle motion graph. Each of the facial graphs contains useful semantic information relevant to facial expressions. The proposed LBAGCN consists of four parallel adaptive graph convolutional networks (AGCNs), each of which employs well-designed adaptive graph convolution blocks and multi-branch temporal convolutional network blocks to learn the spatial and temporal features from a facial graph automatically. The class scores of AGCNs are fused to boost the FER accuracy. Extensive experiments have been carried out to validate the proposed LBAGCN on the recognized CK+ dataset and Oulu CASIA dataset. Compared with state-of-the-art methods, the proposed LBAGCN achieves competitive FER accuracy especially on the challenging Oulu CASIA dataset. Testing results on practical platforms confirm that the proposed LBAGCN can be a feasible solution for real-time FER applications.

Variations in the quantity of plankton impact the entire marine ecosystem. It is of great significance to accurately assess the dynamic evolution of the plankton for monitoring the marine environment and global climate change. In this paper, a novel method is introduced for deep-sea plankton community detection in marine ecosystem using an underwater robotic platform. The videos were sampled at a distance of 1.5 m from the ocean floor, with a focal length of 1.5–2.5 m. The optical flow field is used to detect plankton community. We showed that for each of the moving plankton that do not overlap in space in two consecutive video frames, the time gradient of the spatial position of the plankton are opposite to each other in two consecutive optical flow fields. Further, the lateral and vertical gradients have the same value and orientation in two consecutive optical flow fields. Accordingly, moving plankton can be accurately detected under the complex dynamic background in the deep-sea environment. Experimental comparison with manual ground-truth fully validated the efficacy of the proposed methodology, which outperforms six state-of-the-art approaches.