5 July 2023 in IEEE Transactions on Image Processing
With the development of deep learning technology, the performance of facial expression recognition (FER) has been significantly improved. The current main challenge comes from the confusion of facial expressions caused by the highly nonlinear changes of facial expressions. However, the existing FER methods based on Convolutional Neural Networks (CNN) often ignore the underlying relationship between expressions which is crucial to meliorate the performance of recognition for confusable expressions. And the methods based on Graph Convolutional Networks (GCN) can capture the relationship between vertices, but the aggregation degree of subgraphs generated by these methods is low. They are easy to include unconfident neighbors, which increases the learning difficulty of the network. To solve the above problems, this paper proposes a method to recognize facial expressions on the high aggregation subgraphs (HASs) by combing the advantages of CNN extracting features and GCN modeling complex graph patterns. Specifically, we formulate FER as a vertex prediction problem. Considering the importance of high-order neighbors and higher efficiency, we utilize vertex confidence to find high-order neighbors. Then we construct the HASs based on the top embedding features of these high-order neighbors. And we utilize the GCN to perform reasoning and infer the class of vertices for HASs without a large number of overlapping subgraphs. Our method captures the underlying relationship between expressions on the HASs and improves the accuracy and efficiency of FER. Experimental results on both the in-the-lab datasets and the in-the-wild datasets show that our method achieves higher recognition accuracy than several state-of-the-art methods. This highlights the benefit of the underlying relationship between expressions for FER.
17 February 2025 in IEEE Transactions on Multimedia
Many supervised learning-based facial expression recognition (FER) methods achieve good performance with the assistance of expression labels and a complex framework. However, there are inconsistent annotations in different expression datasets, making the above methods disadvantageous for new expression datasets or datasets with limited training data. The objective of this paper is to learn self-supervised facial expression features that enable the FER model not to rely on the annotation consistency of the different datasets. Most current self-supervised learning algorithms based on contrastive learning learn the representation by forcing different augmented views of the same image close in the embedding space, but they cannot cover all variances within a semantic class. We propose a heatmap neighbor contrastive learning (HNCL) method for FER. It treats the images corresponding to the heatmap nearest neighbors of expressions as other positives, providing more semantic variations than pre-defined augmented transformations. Therefore, our HNCL can learn better expression features covering more intraclass variances, improving the performance of the FER model based on self-supervised learning. After fine-tuning, HNCL with a simple framework achieves top-three performance on the inthe-lab datasets and even matches the performance of state-ofthe-art supervised learning methods on the in-the-wild datasets.
30 November 2023 in Pattern Recognition
16 October 2023 in Multimedia Tools and Applications
21 May 2026 in IEEE Transactions on Circuits and Systems for Video Technology
Arbitrary-oriented object detection plays a crucial role in numerous vision applications. A key challenge lies in accurate orientation estimation, which suffers from angular periodicity and boundary discontinuity, leading to unstable learning and reduced accuracy. To address this issue, we propose a Coarse-to-Fine Structured Angle Prediction (CSAP) approach that reformulates angle estimation as a sequence of structured sub-tasks. Specifically, CSAP first discretizes continuous angles into coarse orientation intervals to ensure robust global prediction. Then, fine-grained angular components are modeled in a two-dimensional structured space, where elliptical Gaussian heatmaps capture local variations. An axis-rescaling strategy is further introduced to alleviate peak-shift errors and enhance localization precision. In addition, an optional multimodal contrastive alignment module is incorporated to provide semantic regularization by embedding category and orientation information into a shared representation space. This design provides a lightweight semantic regularization signal that can improve feature consistency and robustness to rotation variations. Extensive experiments across multiple benchmarks demonstrate that CSAP consistently outperforms state-of-the-art methods in both accuracy and robustness.