Visible-infrared person re-identification (VI-ReID) is a cross-modality retrieval problem, which aims at matching the same pedestrian between the visible and infrared cameras. Due to the existence of pose variation, occlusion, and huge visual differences between the two modalities, previous studies mainly focus on learning image-level shared features. Since they usually learn a global representation or extract uniformly divided part features, these methods are sensitive to misalignments. In this paper, we propose a structure-aware positional transformer (SPOT) network to learn semantic-aware sharable modality features by utilizing the structural and positional information. It consists of two main components: attended structure representation (ASR) and transformer-based part interaction (TPI). Specifically, ASR models the modality-invariant structure feature for each modality and dynamically selects the discriminative appearance regions under the guidance of the structure information. TPI mines the part-level appearance and position relations with a transformer to learn discriminative part-level modality features. With a weighted combination of ASR and TPI, the proposed SPOT explores the rich contextual and structural information, effectively reducing cross-modality difference and enhancing the robustness against misalignments. Extensive experiments indicate that SPOT is superior to the state-of-the-art methods on two cross-modal datasets. Notably, the Rank-1/mAP value on the SYSU-MM01 dataset has improved by 8.43%/6.80%.

Matching the daytime visible and nighttime infrared person images, namely visible infrared person re-identification (VI-ReID), is a challenging cross-modality retrieval problem. Due to the difficulty of data collection and annotation in nighttime surveillance, VI-ReID usually suffers from noise problems, making it challenging to directly learn part discriminative features. In order to improve the discriminability and enhance the robustness against noisy images, this paper proposes a novel dynamic tri-level relation mining (DTRM) framework by simultaneously exploring channel-level, part-level intra-modality, and graph-level cross-modality relation cues. To address the misalignment within the person images, we design an intra-modality weighted-part attention (IWPA) to construct part-aggregated representation. It adaptively integrates the body part relation into the local feature learning with a residual batch normalization (RBN) connection scheme. Besides, a cross-modality graph structured attention (CGSA) is incorporated to improve the global feature learning by utilizing the contextual relation between images from two modalities. This module reduces the negative effects of noisy images. To seamlessly integrate two components, a parameter-free dynamic aggregation strategy is designed in a progressive joint learning manner. To further improve the performance, we additionally design a simple yet effective channel-level learning strategy by exploiting the rich channel information of visible images, which significantly reinforces the performance without modifying the network structure or changing the training process. Extensive experiments on two visible infrared re-identification datasets have verified the effectiveness under various settings. Code is available at: https://github.com/mangye16/DDAG

This paper introduces a simple yet powerful channel augmentation for visible- infrared re-identification. Most existing augmentation operations designed for single-modality visible images do not fully consider the imagery properties in visible to infrared matching. Our basic idea is to homogeneously generate color-irrelevant images by randomly exchanging the color channels. It can be seamlessly integrated into existing augmentation operations, consistently improving the robustness against color variations. For cross-modality metric learning, we design an enhanced channel-mixed learning strategy to simultaneously handle the intra- and cross-modality variations with squared difference for stronger discriminability. Besides, a weak-and-strong augmentation joint learning strategy is further developed to explicitly optimize the outputs of augmented images, which mutually integrates the channel augmented images (strong) and the general augmentation operations (weak) with consistency regularization. Furthermore, by conducting the label association between the channel augmented images and infrared modalities with modality-specific clustering, a simple yet effective unsupervised learning baseline is designed, which significantly outperforms existing unsupervised single-modality solutions. Extensive experiments with insightful analysis on two visible- infrared recognition tasks show that the proposed strategies consistently improve the accuracy. Without auxiliary information, the Rank-1/mAP achieves 71.48%/68.15% on the large-scale SYSU-MM01 dataset.

Video-based person re-identification (ReID) matches the same people across the video sequences with rich spatial and temporal information in complex scenes. It is highly challenging to capture discriminative information when occlusions and pose variations exist between frames. A key solution to this problem rests on extracting the temporal invariant features of video sequences. In this paper, we propose a novel method for discovering temporal coherence by designing a region-level saliency and granularity mining network (SGMN). Firstly, to address the varying noisy frame problem, we design a temporal spatial-relation module (TSRM) to locate frame-level salient regions, adaptively modeling the temporal relations on spatial dimension through a probe-buffer mechanism. It avoids the information redundancy between frames and captures the informative cues of each frame. Secondly, a temporal channel-relation module (TCRM) is proposed to further mine the small granularity information of each frame, which is complementary to TSRM by concentrating on discriminative small-scale regions. TCRM exploits a one-and-rest difference relation on channel dimension to enhance the granularity features, leading to stronger robustness against misalignments. Finally, we evaluate our SGMN with four representative video-based datasets, including iLIDS-VID, MARS, DukeMTMC-VideoReID, and LS-VID, and the results indicate the effectiveness of the proposed method.

Matching hand-drawn sketches with photos (a.k.a sketch-photo recognition or re-identification) faces the information asymmetry challenge due to the abstract nature of the sketch modality. Existing works tend to learn shared embedding spaces with CNN models by discarding the appearance cues for photo images or introducing GAN for sketch-photo synthesis. The former unavoidably loses discriminability, while the latter contains ineffaceable generation noise. In this paper, we start the first attempt to design an information-aligned sketch transformer (Sketch Trans+) viacross-modal disentangled prototype learning, while the transformer has shown great promise for discriminative visual modelling. Specifically, we design an asymmetric disentanglement scheme with a dynamic updatable auxiliary sketch (A-sketch) to align the modality representations without sacrificing information. The asymmetric disentanglement decomposes the photo representations into sketch-relevant and sketch-irrelevant cues, transferring sketch-irrelevant knowledge into the sketch modality to compensate for the missing information. Moreover, considering the feature discrepancy between the two modalities, we present a modality-aware prototype contrastive learning method that mines representative modality-sharing information using the modality-aware prototypes rather than the original feature representations. Extensive experiments on categoryand instance-level sketch-based datasets validate the superiority of our proposed method under various metrics. Code is available at https://github.com/ccq195/SketchTrans

Infrared-Visible person Re-IDentification(IV-ReID) is an emerging subject, which has important research significance for nighttime monitoring. Existing works focus on reducing cross-modality discrepancies, but the cross-modality discrepancy cannot be completely eliminated. Therefore, we concentrate on excavating cross-modality commonalities to handle the task. Since similar features between two modalities are possessed of cross-modality commonalities, our goal is to find more similar features in infrared and visible images. A novel Dual-stream Multi-layer Corresponding Fusion Network(DMCF) is proposed to explore more similar features between two modalities in this paper. It mainly contains three aspects. 1) We explore more similar features between two modalities by learning low-level features, Meanwhile, we also propose a method that the same level features between two modalities are correspondingly fused to reduce cross-modality discrepancies. 2) We adopt different Multi-granularity dividing methods for multi-layer features, so that it can improve the ability of the model to perceive feature details. 3) We separately calculate the loss for different-layer features. Therefore, we learn different weighting factors for the loss of different hierarchical features through Multi-task Learning, so that each branch can be fully optimized. Extensive experiments on two datasets demonstrate the superior performance compared to the state-of-the-arts.