Recently, vision transformers (ViTs) have been investigated in fine-grained visual recognition (FGVC) and are now considered state of the art. However, most ViT-based works ignore the different learning performances of the heads in the multi-head self-attention (MHSA) mechanism and its layers. To address these issues, in this paper, we propose a novel internal ensemble learning transformer (IELT) for FGVC. The proposed IELT involves three main modules: multi-head voting (MHV) module, cross-layer refinement (CLR) module, and dynamic selection (DS) module. To solve the problem of the inconsistent performances of multiple heads, we propose the MHV module, which considers all of the heads in each layer as weak learners and votes for tokens of discriminative regions as cross-layer feature based on the attention maps and spatial relationships. To effectively mine the cross-layer feature and suppress the noise, the CLR module is proposed, where the refined feature is extracted and the assist logits operation is developed for the final prediction. In addition, a newly designed DS module adjusts the token selection number at each layer by weighting their contributions of the refined feature. In this way, the idea of ensemble learning is combined with the ViT to improve fine-grained feature representation. The experiments demonstrate that our method achieves competitive results compared with the state of the art on five popular FGVC datasets. Source code has been released and can be found at https://github.com/mobulan/IELT .

Hierarchical deep features can provide multilevel abstractions of target objects, which play an important role in target localization and classification. Determining how to effectively aggregate abstract information from different levels in RGB and thermal modalities is the key to exploiting their complementary advantages for robust RGBT tracking. However, existing RGBT tracking algorithms either focus on the semantic information of the last layer or aggregate hierarchical deep features from each modal using simple operations (e.g., summation and concatenation), which limit the capability of the multimodal tracker. To address these issues, in this paper, we propose a novel multimodal cross-layer bilinear pooling network for RGBT tracking. In our network, firstly, to boost the performance of the tracker, we use a channel attention mechanism to implement the adaptive calibration of feature channels for all convolutional layer features before realizing hierarchical feature fusion. Then, a bilinear pooling operation is performed on any two layers through the cross product, which is a second-order computation that effectively aggregates the deep semantic and shallow texture information of the target. Finally, a quality-aware fusion module is designed to aggregate the bilinear pooling features of different layer interactions between different modalities in an adaptive manner. The results of a large number of experiments on two public benchmark datasets demonstrate the effectiveness of our tracker compared with other state-of-the-art tracking methods.

3D convolutional neural networks (CNNs) have been demonstrated to be a powerful tool in hyperspectral images (HSIs) classification. However, using the conventional 3D CNNs to extract the spectral–spatial feature for HSIs results in too many parameters as HSIs have plenty of spatial redundancy. To address this issue, in this paper, we first design multiscale convolution to extract the contextual feature of different scales for HSIs and then propose to employ the octave 3D CNN which factorizes the mixed feature maps by their frequency to replace the normal 3D CNN in order to reduce the spatial redundancy and enlarge the receptive field. To further explore the discriminative features, a channel attention module and a spatial attention module are adopted to optimize the feature maps and improve the classification performance. The experiments on four hyperspectral image data sets demonstrate that the proposed method outperforms other state-of-the-art deep learning methods.

Fine-grained visual classification aims to classify similar sub-categories with the challenges of large variations within the same sub-category and high visual similarities between different sub-categories. Recently, methods that extract semantic parts of the discriminative regions have attracted increasing attention. However, most existing methods extract the part features via rectangular bounding boxes by object detection module or attention mechanism, which makes it difficult to capture the rich shape information of objects. In this paper, we propose a novel Multi-Granularity Part Sampling Attention (MPSA) network for fine-grained visual classification. First, a novel multi-granularity part retrospect block is designed to extract the part information of different scales and enhance the high-level feature representation with discriminative part features of different granularities. Then, to extract part features of various shapes at each granularity, we propose part sampling attention, which can sample the implicit semantic parts on the feature maps comprehensively. The proposed part sampling attention not only considers the importance of sampled parts but also adopts the part dropout to reduce the overfitting issue. In addition, we propose a novel multi-granularity fusion method to highlight the foreground features and suppress the background noises with the assistance of the gradient class activation map. Experimental results demonstrate that the proposed MPSA achieves state-of-the-art performance on four commonly used fine-grained visual classification benchmarks. The source code is publicly available at https://github.com/mobulan/MPSA.

Single hyperspectral image (HSI) super-resolution (SR) is an important topic in remote sensing field. However, existing HSI SR methods mainly use the feed-forward upsampling technique and convolutional neural network (CNN) to learn the feature representation, failing to learn the complex mapping relationship between low-resolution (LR) and high-resolution (HR) and long-range joint spectral and spatial features. To address this issue, in this paper, we propose the Spatial-Spectral Interactive Transformer U-Net with Alternating Sampling (AS ³ ITransUNet) for the HSI SR task. In this method, to mitigate the computational burden resulting from the high spectral dimension of HSI, a group reconstruction strategy is adopted. To effectively explore the hierarchical features of HSI, the U-Net with alternating upsampling and downsampling is designed that allocates the task of learning the complex mapping relationship to each stage of U-Net. To fully extract the spatial-spectral features of HSI, we propose the spatial-spectral interactive transformer (SSIT) block and integrate it into the encoder and decoder of U-Net. The SSIT block contains a cross-branch bidirectional interaction module, which further captures the complementary information between spatial and spectral dimensions. Moreover, the multi-stage complementary information learning (MFEL) is proposed to capture the complementary information in the adjacent HSI groups for recovering the absent details in the current HSI group. The experiments on the three benchmark datasets demonstrate that the proposed AS ³ ITransUNet can effectively improve the spatial resolution and preserve the spectral information at different scales. Models and code are available at https://github.com/liushiji666/AS3-ITransUNet.