Facial Expression Recognition (FER) in the wild is extremely challenging due to occlusions, variant head poses, face deformation and motion blur under unconstrained conditions. Although substantial progresses have been made in automatic FER in the past few decades, previous studies were mainly designed for lab-controlled FER. Real-world occlusions, variant head poses and other issues definitely increase the difficulty of FER on account of these information-deficient regions and complex backgrounds. Different from previous pure CNNs based methods, we argue that it is feasible and practical to translate facial images into sequences of visual words and perform expression recognition from a global perspective. Therefore, we propose the Visual Transformers with Feature Fusion (VTFF) to tackle FER in the wild by two main steps. First, we propose the attentional selective fusion (ASF) for leveraging discriminative information with the global-local attention. Second, we propose to model relationships between these visual words with the global self-attention. The proposed method is evaluated on three public in-the-wild facial expression datasets (RAF-DB, FERPlus and AffectNet). Extensive experiments demonstrate that our method shows superior performance over other methods, setting new state of the art on RAF-DB with 88.14%, FERPlus with 88.81% and AffectNet with 61.85%.

Hyperspectral images (HSI) contain rich spatial and spectral detail information, while light detection and ranging (LiDAR) data can provide the elevation information. Thus, the fusion of HSI and LiDAR data can help for more accurate image classification, which becomes a hot research topic. However, it is difficult to capture complex local and global spatial-spectral associations, meanwhile, how to build an effective interaction between multi-modal data is another important issue. To this end, a novel global-local transformer network (GLT-Net) is proposed for the joint classification of HSI and LiDAR data, in this paper. The main idea is to fully exploit the advantage of the convolution operator in characterizing locally correlated features and the promising capability of transformer architecture in learning long-range dependencies. Moreover, multi-scale feature fusion and probabilistic decision fusion strategies are also designed in one framework, in order to further improve classification performance. Here, the proposed GLT-Net mainly consists of multi-scale local spatial feature learning, global spectral feature learning, and global-local feature fusion classification. In specific, multi-modal image cubes of different sizes are firstly extracted and sent into convolutional neural networks (CNNs) to learn local spatial features, which is followed by multi-modal information propagation and spatial-attention guided multi-scale feature fusion. Afterwards, by considering spectral feature channels from a sequential perspective, vision transformers are introduced to model the global spectral dependencies. Finally, multiple class estimations based on local and global features are integrated via a probabilistic decision fusion strategy. In this way, complementary information of multi-modal data as well as local/global spectral-spatial information can be fully mined and jointly utilized. Extensive experiments on three popular HSI and LiDAR datasets demonstrate that the proposed method performs superiority over state-of-the-art methods. The source code of the proposed method will be made publicly available at https://github.com/Ding-Kexin/GLT-Net.

Facial expression recognition (FER) in the wild is extremely challenging due to occlusions, variant head poses under unconstrained conditions and incorrect annotations (e.g., label noise). In this paper, we aim to improve the performance of in-the-wild FER with Transformers and online label correction. Different from pure CNNs based methods, we propose a Transformer-augmented network (TAN) to dynamically capture the relationships within each facial patch and across the facial patches. Specifically, the TAN translates a number of facial patch images into a set of visual feature sequences by a backbone convolutional neural network. The intra-patch Transformer is subsequently utilized to capture the most discriminative features within each visual feature sequence. The position-disentangled attention mechanism of the intra-patch Transformer is proposed to better incorporate the positional information for feature sequences. Furthermore, we propose the inter-patch Transformer to model the dependencies across these feature sequences. More importantly, we present the online label correction (OLC) framework to correct suspicious hard labels and accumulate soft labels based on the predictions of the model, which strengthens the robustness of our model against label noise. We validate our method on several widely-used datasets (RAF-DB, FERPlus, AffectNet), realistic occlusion and pose variation datasets, and synthetic noisy datasets. Extensive experiments on these benchmarks demonstrate that the proposed method performs favorably against state-of-the-art methods. The source code will be made publicly available.

Remote sensing image super-resolution (SR) aims to boost the image resolution while recovering rich high-frequency details. Currently, most of the SR methods are based on an assumption that the degradation kernel is a specific downsampler. However, the degradation kernel is unknown and sophisticated for real remote sensing scenes, leading to a severe performance drop. To alleviate this problem, we propose a multilayer degradation representation-guided blind SR method for remote sensing images, which mainly consists of three key steps. First, an unsupervised representation learning is exploited to learn the degradation representation from low-resolution images. Then, a degradation-guided deep residual module is designed to model high-order features across different scales from the original images. Finally, a multilayer degradation-aware feature fusion mechanism is proposed to restore the finer details. Experiments on synthetic and real datasets demonstrate that the proposed method can achieve promising performance with respect to other state-of-the-art SR approaches.

Detecting eye contact is essential for embodied robots to engage in natural interactions with humans, enhancing the intuitiveness and comfort of these exchanges. However, eye contact detection often presents a significant challenge due to a variety of factors, such as low contrast and various forms of occlusions. Existing methods incorporate convolutional neural networks (CNNs) or Transformers to learn discriminative representations, but usually ignore the influence of noisy or less relevant regions in facial images. To address this gap, we propose the deep feature selection and fusion network (FSFNet) for eye contact detection in multi-party conversations. Our proposed method adaptively selects fine-grained visual features and reduces the impacts of irrelevant features. Specifically, we present a local feature selection scheme that leverages the attention scores to progressively concentrate on the most informative features. By integrating the carefully selected features into the multi-head self-attention module, we can maintain the superior properties of Transformers while simultaneously reducing the overall computational demands. We evaluate the proposed method on the official eye contact detection datasets, which achieves promising results of 0.8174 and 0.79 on the validation and test sets, respectively. We have made the source code publicly accessible in https://github.com/ma-hnu/FSFNet.

Facial Expression Recognition (FER) in the wild is extremely challenging due to occlusions, variant head poses, face deformation and motion blur under unconstrained conditions. Although substantial progresses have been made in automatic FER in the past few decades, previous studies were mainly designed for lab-controlled FER. Real-world occlusions, variant head poses and other issues definitely increase the difficulty of FER on account of these information-deficient regions and complex backgrounds. Different from previous pure CNNs based methods, we argue that it is feasible and practical to translate facial images into sequences of visual words and perform expression recognition from a global perspective. Therefore, we propose the Visual Transformers with Feature Fusion (VTFF) to tackle FER in the wild by two main steps. First, we propose the attentional selective fusion (ASF) for leveraging two kinds of feature maps generated by two-branch CNNs. The ASF captures discriminative information by fusing multiple features with the global-local attention. The fused feature maps are then flattened and projected into sequences of visual words. Second, inspired by the success of Transformers in natural language processing, we propose to model relationships between these visual words with the global self-attention. The proposed method is evaluated on three public in-the-wild facial expression datasets (RAF-DB, FERPlus and AffectNet). Under the same settings, extensive experiments demonstrate that our method shows superior performance over other methods, setting new state of the art on RAF-DB with 88.14%, FERPlus with 88.81% and AffectNet with 61.85%. The cross-dataset evaluation on CK+ shows the promising generalization capability of the proposed method.