31 August 2021 in IEEE Transactions on Neural Networks and Learning Systems
Visible-Infrared person reidentification (VI-ReID) is a challenging matching problem due to large modality variations between visible and infrared images. Existing approaches usually bridge the modality gap with only feature-level constraints, ignoring pixel-level variations. Some methods employ a generative adversarial network (GAN) to generate style-consistent images, but it destroys the structure information and incurs a considerable level of noise. In this article, we explicitly consider these challenges and formulate a novel spectrum-aware feature augmentation network named SFANet for cross-modality matching problem. Specifically, we put forward to employ grayscale-spectrum images to fully replace RGB images for feature learning. Learning with the grayscale-spectrum images, our model can apparently reduce modality discrepancy and detect inner structure relations across the different modalities, making it robust to color variations. At feature level, we improve the conventional two-stream network by balancing the number of specific and sharable convolutional blocks, which preserve the spatial structure information of features. Additionally, a bidirectional tri-constrained top-push ranking loss (BTTR) is embedded in the proposed network to improve the discriminability, which efficiently further boosts the matching accuracy. Meanwhile, we further introduce an effective dual-linear with batch normalization identification (ID) embedding method to model the identity-specific information and assist BTTR loss in magnitude stabilizing. On SYSU-MM01 and RegDB datasets, we conducted extensively experiments to demonstrate that our proposed framework contributes indispensably and achieves a very competitive VI-ReID performance.
4 January 2023 in IEEE Journal of Selected Topics in Signal Processing
Visible-infrared person re-identification (VI-ReID) is a challenging and essential task that aims to retrieve a set of person images over visible and infrared camera views. In order to mitigate the impact of large modality discrepancy existing in heterogeneous images, previous methods attempt to apply generative adversarial network (GAN) to generate the modality-consisitent data. However, due to severe color variations between the visible domain and infrared domain, the generated fake cross-modality samples often fail to possess good qualities to fill the modality gap between synthesized scenarios and target real ones, which leads to sub-optimal feature representations. In this work, we address the cross-modality matching problem with Aligned Grayscale Modality (AGM), an unified dark-line spectrum that reformulates visible-infrared dual-mode learning as a gray-gray single-mode learning problem. Specifically, we generate the grayscale modality from the homogeneous visible images. Then, we train a style tranfer model to transfer infrared images into homogeneous grayscale images. In this way, the modality discrepancy is remarkably reduced in the image space. In order to reduce the remaining appearance discrepancy, we further introduce a multi-granularity feature extraction network to conduct feature-level alignment. Rather than relying on the global information, we propose to exploit local (head-shoulder) features to assist person Re-ID, which complements each other to form a stronger feature descriptor. Comprehensive experiments implemented on the mainstream evaluation datasets include SYSU-MM01 and RegDB indicate that our method can remarkably enhance cross-modality retrieval performance against the state of the art methods.
20 February 2024 in IEEE Transactions on Knowledge and Data Engineering
Unsupervised visible-infrared person re-identification (USVI-ReID) is a valuable yet under-explored task that addresses the challenge of person retrieval from different modalities without annotations. USVI-ReID is faced with large intra and inter modality discrepancy as well as pseudo label noises. In this paper, we propose a novel learning approach termed Spectrum-Cam Aware and Refined Cross-Pair learning strategy (SCA-RCP), crafted to concurrently tackle these issues. The Dual-Spectrum Augmentation (DSA) method is first presented to mine inter-modality knowledge between the visible and infrared modalities by expanding the diversity of spectra for both modalities. Concurrently, to learn intra-modality knowledge, we divide the clusters into camera-based proxies and introduce the Cross-modal Cam-Aware Proxy learning method (CCAP) to pull together camera-based proxies that correspond to the same identity across modalities. Finally, to mitigate clustering noise, the innovative Refined Cross-Pair learning strategy (RCP) is devised comprising the Intra-modality Label Refinement (ILR) and Bi-directional Cross Pairing (BCP) method. ILR calculates maximum-likely label for each instance by employing the intra-modal cluster memories as a classifier, while BCP establish dependable cross-paired labels in a bidirectional manner. Extensive experiments on the cross-modality datasets demonstrate the superior performance of our model over the state-of-art method.
23 June 2021 in Multimedia Tools and Applications
15 September 2025 in IEEE Transactions on Systems, Man, and Cybernetics: Systems
Visible–infrared person re-identification (VI-ReID) focuses on accurately matching individuals across different imaging modalities. Existing studies focus on generating modality-consistent images at the pixel level through the use of generative adversarial networks (GANs) to mitigate the impact of modality discrepancies. However, these methods face significant challenges in overcoming the limitation that synthesized samples from different modalities may suffer from semantic distortion. In this work, we propose an online one-stage style fuzzy normalization (SFN) method to generate modality-fuzzy features in the latent space while regularizing the model’s predictions. Specifically, SFN adaptively mixes the feature statistics of two random modality instances of the same identity in a single forward pass during training. In this process, to enhance the richness of modality interaction information, we design a novel causality balance loss, which enforces the generated fuzzy features to be independent of their initial modality while simultaneously encouraging them to align more closely with the other modality. Furthermore, we introduce an identity-aware consistency loss to regularize the predictions between the original and SFN-generated features to ensure semantic consistency. In contrast to prior work, SFN is a plug-and-play module that does not rely on any generative-based models, making it highly adaptable to various network architectures. Extensive experiments were performed on three public cross-modality datasets to ensure fair and reliable comparisons. The empirical results demonstrate the clear superiority of our method over previous state-of-the-art methods.
30 December 2022 in IEEE Internet of Things Journal
Visible-thermal person re-identification (VT-ReID) is an emerging cross-modality matching problem, which aims to identify the same person across the daytime visible modality and nighttime thermal modality in the Internet of Things. Existing cutting-edge approaches consistently attempts to exploit image generation technique to generate cross-modality images or design various feature-level constraints to align feature distribution of heterogeneous data. However, color variations originating from the different imaging processes of spectrum cameras remains unsolved, which leads to sub-optimal feature representations. In this paper, we present a simple but very effective data augmentation method named Random Gray for the cross-modality matching task. Given a training sample, Random Gray randomly selects a rectangular region and translates it to grayscale. In this process, training images with fusing various levels of visible and grayscale information are generated, thereby reducing the risk of overfitting and making the model robust to color variations. Besides, we introduce a novel pooling method called softpooling to retain more information in the reduced activation maps. With softpooling layer, the network can learn more discriminative person features and further boost its retrieval performance. We conduct extensive experiments on publicly available cross-modality Re-ID datasets (SYSU-MM01 and RegDB) to demonstrate the effectiveness of our proposed method. Experimental results show that Random Gray and softpooling strategies yield siginificant accurancy improvement, and they can be utilized as training tricks for further VT-ReID research.