Facial Expression Recognition (FER) in the wild is extremely challenging due to occlusions, variant head poses, face deformation and motion blur under unconstrained conditions. Although substantial progresses have been made in automatic FER in the past few decades, previous studies were mainly designed for lab-controlled FER. Real-world occlusions, variant head poses and other issues definitely increase the difficulty of FER on account of these information-deficient regions and complex backgrounds. Different from previous pure CNNs based methods, we argue that it is feasible and practical to translate facial images into sequences of visual words and perform expression recognition from a global perspective. Therefore, we propose the Visual Transformers with Feature Fusion (VTFF) to tackle FER in the wild by two main steps. First, we propose the attentional selective fusion (ASF) for leveraging discriminative information with the global-local attention. Second, we propose to model relationships between these visual words with the global self-attention. The proposed method is evaluated on three public in-the-wild facial expression datasets (RAF-DB, FERPlus and AffectNet). Extensive experiments demonstrate that our method shows superior performance over other methods, setting new state of the art on RAF-DB with 88.14%, FERPlus with 88.81% and AffectNet with 61.85%.

Hyperspectral images (HSIs) are often degraded by a mixture of various types of noise during the imaging process, including Gaussian noise, impulse noise, stripes, etc. Such complex noise could plague the subsequent HSIs processing. Generally, most of HSI denoising methods formulate sparsity optimization problems with convex norm constraints, which over-penalize large entries of vectors, and may result in a biased solution. In this paper, a nonconvex regularized low-rank and sparse matrix decomposition (NonRLRS) method is proposed for HSIs denoising, which can simultaneously remove the Gaussian noise, impulse noise, dead lines, and stripes. The NonRLRS aims to decompose the degraded HSI, expressed in a matrix form, into low-rank and sparse components with a robust formulation. To enhance the sparsity in both the intrinsic low-rank structure and the sparse corruptions, a novel nonconvex regularizer named as normalized ε-penalty, is presented, which can adaptively shrink each entry. In addition, an effective algorithm based on the majorization minimization (MM) is developed to solve the resulting nonconvex optimization problem. Specifically, the MM algorithm first substitutes the nonconvex objective function with the surrogate upper-bound in each iteration, and then minimizes the constructed surrogate function, which enables the nonconvex problem to be solved in the framework of reweighted technique. The experimental results on both simulated and real data demonstrate the effectiveness of the proposed method.

Temporal answer grounding in instructional video (TAGV) is a new task naturally derived from temporal sentence grounding in general video (TSGV). Given an untrimmed instructional video and a text question, this task aims at locating the frame span from the video that can semantically answer the question, i.e., visual answer. Existing methods tend to solve the TAGV problem with a visual span-based predictor, taking visual information to predict the start and end frames in the video. However, due to the weak correlations between the semantic features of the textual question and visual answer, current methods using the visual span-based predictor do not work well in the TAGV task. In this paper, we propose a visual-prompt text span localization (VPTSL) method, which introduces the timestamped subtitles for a text span-based predictor. Specifically, the visual prompt is a learnable feature embedding, which brings visual knowledge to the pre-trained language model. Meanwhile, the text span-based predictor learns joint semantic representations from the input text question, video subtitles, and visual prompt feature with the pre-trained language model. Thus, the TAGV is reformulated as the task of the visual-prompt subtitle span localization for the visual answer. Extensive experiments on five instructional video datasets, namely MedVidQA, TutorialVQA, VehicleVQA, CrossTalk and Coin, show that the proposed method outperforms several state-of-the-art (SOTA) methods by a large margin in terms of mIoU score, which demonstrates the effectiveness of the proposed visual prompt and text span-based predictor.

Glioblastoma (GBM) tumor is the most common primary brain malignant tumor. The precise identification of GBM tumors is very important for diagnosis and treatment. Hyperspectral imaging is a fast, noncontact, accurate, and safe modern medical detection technology, which is expected to be a new tool of intraoperative diagnosis. In order to make full use of the spectral and spatial information of hyperspectral images (HSIs) to achieve accurate GBM tumor identification, a method based on the fusion of multiple deep models is proposed for in vivo human brain HSI classification. The proposed method includes the following major steps: 1) spectral phasor analysis and data oversampling; 2) 1-D deep neural network (1D-DNN)-based spectral HSI feature extraction and classification; 3) 2-D convolution neural network (2D-CNN)-based spectral–spatial HSI feature extraction and classification; 4) edge-preserving filtering-based classification result fusion and optimization; and 5) fully convolutional network (FCN)-based background segmentation. To verify the capabilities of the proposed method, experiments are performed on two real human brain hyperspectral datasets, including 36 in vivo HSIs captured from 16 different patients. The proposed method can achieve an overall accuracy of 96.69% for four-class classification and overall accuracy of 96.34% for GBM tumor identification. Experimental results demonstrate that the proposed method exhibits competitive classification performance and can generate satisfactory thematic maps of the location of the GBM tumor, which can provide the surgeon with guidance on successful and precise tumor resection.

Semisupervised semantic segmentation is an effective way to reduce the expensive manual annotation cost and take advantage of the unlabeled data for remote sensing (RS) image interpretation. Recent related research has mainly adopted two strategies: self-training and consistency regularization. Self-training tries to acquire accurate pseudo-labels to explicitly expand the train set. However, the existing methods cannot accurately identify false pseudo-labels, suffering from their negative impact on model optimization. The consistency regularization constrains the model by producing consistent predictions robust to the perturbations introduced in the sample or feature domain but requires a sufficient number of training data. Therefore, we propose a strategy for the semisupervised semantic segmentation of the RS images. The proposed model in the generative adversarial network (GAN) framework is optimized by consistency self-training, learning the distributions of both labeled and unlabeled data. The discriminator is optimized by accurate pixel-level training labels instead of the image-level ones, thereby assessing the confidence for the prediction of each pixel, which is then used to reweight the loss of the unlabeled data in self-training. The generator is optimized with the consistency constraint with respect to all random perturbations on the unlabeled data, which increases the sample diversity and prompts the model to learn the underlying distribution of the unlabeled data. Experimental results on the the large-scale and densely annotated Instance Segmentation in Aerial Images Dataset (iSAID) datasets and the International Society for Photogrammetry and Remote Sensing (ISPRS) datasets show that our framework outperforms several state-of-the-art semisupervised semantic segmentation methods.

Facial expression recognition (FER) in the wild is extremely challenging due to occlusions, variant head poses under unconstrained conditions and incorrect annotations (e.g., label noise). In this paper, we aim to improve the performance of in-the-wild FER with Transformers and online label correction. Different from pure CNNs based methods, we propose a Transformer-augmented network (TAN) to dynamically capture the relationships within each facial patch and across the facial patches. Specifically, the TAN translates a number of facial patch images into a set of visual feature sequences by a backbone convolutional neural network. The intra-patch Transformer is subsequently utilized to capture the most discriminative features within each visual feature sequence. The position-disentangled attention mechanism of the intra-patch Transformer is proposed to better incorporate the positional information for feature sequences. Furthermore, we propose the inter-patch Transformer to model the dependencies across these feature sequences. More importantly, we present the online label correction (OLC) framework to correct suspicious hard labels and accumulate soft labels based on the predictions of the model, which strengthens the robustness of our model against label noise. We validate our method on several widely-used datasets (RAF-DB, FERPlus, AffectNet), realistic occlusion and pose variation datasets, and synthetic noisy datasets. Extensive experiments on these benchmarks demonstrate that the proposed method performs favorably against state-of-the-art methods. The source code will be made publicly available.