18 September 2024 in IEEE Access
Facial expression recognition (FER) is an important task in human emotion analysis. Though researchers have made great progress in FER, the semantic information lying in facial landmarks has not been fully exploited. In this paper, we propose a landmark-based adaptive graph convolutional network (LBAGCN) for FER. It takes facial landmarks as input in both phases of model training and inference and thus preserves sensitive privacy information well. Based on the facial landmark sequence extracted from a video clip, we define landmark motion sequence, muscle sequence and muscle motion sequence. Correspondingly, we construct four facial graphs with learnable topologies, named landmark graph, landmark motion graph, muscle graph and muscle motion graph. Each of the facial graphs contains useful semantic information relevant to facial expressions. The proposed LBAGCN consists of four parallel adaptive graph convolutional networks (AGCNs), each of which employs well-designed adaptive graph convolution blocks and multi-branch temporal convolutional network blocks to learn the spatial and temporal features from a facial graph automatically. The class scores of AGCNs are fused to boost the FER accuracy. Extensive experiments have been carried out to validate the proposed LBAGCN on the recognized CK+ dataset and Oulu CASIA dataset. Compared with state-of-the-art methods, the proposed LBAGCN achieves competitive FER accuracy especially on the challenging Oulu CASIA dataset. Testing results on practical platforms confirm that the proposed LBAGCN can be a feasible solution for real-time FER applications.
23 July 2024 in IEEE Access
Recognizing emotional states from speech is essential for human-computer interaction. It is a challenging task to realize effective speech emotion recognition (SER) on platforms with limited memory capacity and computing power. In this paper, we propose a lightweight multi-scale deep neural network architecture for SER, which takes Mel Frequency Cepstral Coefficients (MFCCs) as input. In order to realize effective multi-scale feature extraction, we propose a new Inception module, named A_Inception. A_Inception combines the merits of Inception module and attention-based rectified linear units (AReLU) and thus can learn multi-scale features adaptively with low computational cost. Meanwhile, to extract most important emotional information, we propose a new multiscale cepstral attention and temporal-cepstral attention (MCA-TCA) module. The idea of MCA-TCA module is to focus on the key cepstral components and the key temporal-cepstral positions. Furthermore, a loss function combining Softmax loss and Center loss is adopted to supervise the model training so as to enhance the model’s discriminative power. Experiments have been carried out on IEMOCAP, EMODB and SAVEE datasets to verify the performance of the proposed model and compare with the state-of-the-art SER models. Numerical results reveal that the proposed model has a small number of parameters (0.82M) and much lower computational cost (81.64 MFLOPs) than compared models, and achieves impressive accuracy on all datasets considered.