23 November 2023 in IEEE Signal Processing Letters
Facial Expression Recognition (FER) aims to identify emotional expressions in human faces, and it is a fundamental task in computer vision. Recently, some methods apply Vision Transformer (ViT) to FER and have achieved promising results. However, FER still suffers from two key issues: inter-class similarity and intra-class discrepancy. To address the issues, in this letter, we propose a Multi-Scale Attention Learning Network (MALN) based on ViT, which can learn facial expression embeddings in a multi-scale manner. Specifically, we adopt a multi-branch ViT architecture to adaptively explore multi-scale correlations without self-attention. Furthermore, we also design a Scale Distinction Loss (SDL) to dynamically regulate facial embeddings from multiple branches, which can guide ViT to capture discriminative facial regions. Experimental results on three public datasets (inluding RAF-DB, AffectNet and FERPlus) demonstrate the effectiveness of our proposed MALN for FER
5 September 2024 in IEEE Transactions on Affective Computing
Facial Expression Recognition (FER) aims to categorize emotional expressions depicted on a human face, and is a challenging task under unconstrained conditions, such as face occlusions and pose variations. Recent methods usually adopt self attention or cross attention to explore global or local relationships among different level features. However, these methods are inclined to focus on the redundant facial regions, causing model overfitting. To address this problem, we propose a new FER model named JADFER, which drops the joint attention in the weight matrix to adaptively enhance facial expression representations. Specifically, our JADFER model consists of three components: Spatial Branch (SB), Contextual Branch (CB), and Spatial-Contextual Interaction (SCI). Firstly, SB runs $N$ paths in parallel, where a Variety loss is designed to guide the paths of SB to focus on different discriminative regions. Meanwhile, CB abstracts the contextual facial representations using self attention with Joint Attention Dropping (JAD). Then, the SCI adopts the spatial features from SB to query the contextual representations from CB through cross attention with JAD, which regulates the attention weights by dropping the similar activations to further enhance the facial embeddings. Experimental results demonstrate that the proposed model outperforms the state-of-the-art methods on several FER benchmarks.
24 July 2025 in IEEE Transactions on Cybernetics
human-object interaction (HOI) detection tackles the problem of joint localization and classification of HOIs. Recent HOI detection methods are mainly based on transformer networks, where the explicit priors at the object level (e.g., scene layout, object appearance, or category) are usually fed into the transformer to improve the object query ability. Though these methods have achieved remarkable results, they did not pay enough attention to the implicit action-level information, which is the fundamental element of HOI. In this work, we propose an interaction-aware transformer network (IATN) to obtain the interaction-aware query, by jointly utilizing implicit action-level priors and explicit object-level priors. Specifically, we design an action-aware module (AAM) to aggregate implicit action priors from the scene level and instance level, respectively. Then, we design an action-oriented graph (AOG), where human feature and object feature are graph nodes and action semantics represent graph edges, to aggregate priors jointly from action level and object level. Afterwards, the interaction-aware query is acquired and finally adopted to obtain the HOI predictions. Besides, we leverage knowledge distillation to enhance the action-level priors by transferring the final HOI predictions to the intermediate features. Extensive experiments on HICO-DET and V-COCO datasets verify the effectiveness of our proposed interaction-aware model.