Facial Expression Recognition (FER) aims to identify emotional expressions in human faces, and it is a fundamental task in computer vision. Recently, some methods apply Vision Transformer (ViT) to FER and have achieved promising results. However, FER still suffers from two key issues: inter-class similarity and intra-class discrepancy. To address the issues, in this letter, we propose a Multi-Scale Attention Learning Network (MALN) based on ViT, which can learn facial expression embeddings in a multi-scale manner. Specifically, we adopt a multi-branch ViT architecture to adaptively explore multi-scale correlations without self-attention. Furthermore, we also design a Scale Distinction Loss (SDL) to dynamically regulate facial embeddings from multiple branches, which can guide ViT to capture discriminative facial regions. Experimental results on three public datasets (inluding RAF-DB, AffectNet and FERPlus) demonstrate the effectiveness of our proposed MALN for FER

Noisy labels make a profoundly negative impact on Facial Expression Recognition (FER) due to inter-class similarity and subjective annotation. Recent works mainly focus on distinguishing the clean samples or mining the latent truth, which not only need extra computational overhead, but also have the error risk of relabeling. In this work, we propose a SNEFER model without elaborately discriminating noisy labels, which can adaptively stop the negative effect by a novel contrastive regularization term. Specifically, we establish two branches: class prediction and contrastive regularization, in a parallel manner. Though the class prediction branch learns facial expression representations over the noisy labels, it keeps the ability to measure the similarity of two facial images but is not robust for FER. If the similarity exceeds a given threshold, the regularization branch pulls them tightly in the embedding space through a Bilateral Contrastive Regularization (BCR) loss, which prevents noisy labels from degrading FER model adaptively. Experimental results demonstrate that our SNEFER model outperforms the state-of-the-art methods on three FER benchmarks under different noisy levels.

In the fields of health care and clinical monitoring, vision-based Facial Expression Recognition (FER) has achieved significantly progress, but it still faces the challenge of poor generalization ability under unconstrained conditions of occlusions and pose variation. Recently, Vision-Language Model (VLM) has greatly advanced the FER task. However, the existing VLM-based FER methods typically leverage a hard-crafted prompt (e.g.“, a photo of class”) and only focus on the holistic semantic alignment, which may suffer from modal heterogeneity. In this work, we propose a fine-grained vision-language model via Prompt Masking for FER (PMFER). Specifically, for each expression, we first create fine-grained prompts using facial action units to guide the image encoder to learn discriminative representations. Further, to finely align text prompts and visual action units, we randomly drop a phrase description in the prompts and then predict the dropped phrase by conducting modal cross attention, implicitly promoting fine-grained vision-language alignment. In addition, we also design a modal-adversarial strategy to holistically eliminate the modal difference between visual and textual embeddings in a common latent space. Experimental results demonstrate that our PMFER model outperforms the state-of-the-art methods on several FER benchmarks, especially under the conditions of occlusions and pose variations. Note to Practitioners —Facial expression recognition is very important in health care and clinical monitoring, which provides an useful tool to assess the psychological and physiological conditions of patients. Although FER has made significant progress with the development of deep learning technologies, it still faces problems in the complex environments (e.g., occlusions and pose variations). To address the above issues, we propose a novel FER method in this work based on the recent vision-language model. It takes RGB image and text prompts as input and finally predicts the expression classification. Different from the existing methods, the proposed PMFER can enable fine-grained modal alignment for facial key units. Compared with the state-of-the-art methods on the public datasets, it can achieve better results, especially under the conditions of occlusions and pose variations. Also, we evaluate the proposed method on a real-world pain dataset, and the results demonstrate that PMFER has a good generalization and can be applied to health care.

Facial Expression Recognition (FER) aims to categorize emotional expressions depicted on a human face, and is a challenging task under unconstrained conditions, such as face occlusions and pose variations. Recent methods usually adopt self attention or cross attention to explore global or local relationships among different level features. However, these methods are inclined to focus on the redundant facial regions, causing model overfitting. To address this problem, we propose a new FER model named JADFER, which drops the joint attention in the weight matrix to adaptively enhance facial expression representations. Specifically, our JADFER model consists of three components: Spatial Branch (SB), Contextual Branch (CB), and Spatial-Contextual Interaction (SCI). Firstly, SB runs $N$ paths in parallel, where a Variety loss is designed to guide the paths of SB to focus on different discriminative regions. Meanwhile, CB abstracts the contextual facial representations using self attention with Joint Attention Dropping (JAD). Then, the SCI adopts the spatial features from SB to query the contextual representations from CB through cross attention with JAD, which regulates the attention weights by dropping the similar activations to further enhance the facial embeddings. Experimental results demonstrate that the proposed model outperforms the state-of-the-art methods on several FER benchmarks.

Emotion Recognition in Conversation (ERC) plays an important role in driving the development of human-machine interaction. After the extensive exploration of the text modality, visual and audio information has attracted considerable attention. Most of the existing approaches adopt either the attention mechanism or graph neural networks to conduct multi-modal fusion by directly utilizing pre-extracted single modal features, but they ignore the inherent priors and characteristics (highly related to emotions) contained in different modalities. For example, for visual modality, facial expression can well reveal a person's emotional state, and the intonation, speech rate and volume of audio modality also reflect emotional fluctuation. Thus, in this work, we aim to promote context fusion among multi-modal features by exploring inherent modal priors. Firstly, we create textual descriptions for different emotion categories belonging to different modalities, denoted as multimodal prompts which can be generated either from common sense or using large language models (e.g., ChatGPT). Then, we propose an adaptive gating fusion module which dynamically learns the weights between unimodal features and prompts, and allows the multimodal prompts to participate in encoding and enriching modal information. Finally, to facilitate multimodal fusion, we design a multimodal progressive encoder to learn inter-modal interactions among conversational utterances. Experimental results show that our model outperforms state-of-the-art models in ERC on two popular benchmark datasets.