12 May 2022 in IEEE Transactions on Multimedia
Person re-identification is still a challenging task when moving objects or another person occludes the probe person. Mainstream methods based on even partitioning apply an off-the-shelf human semantic parsing to highlight the non-collusion part. In this paper, we apply an attention branch to learn the human semantic partition to avoid misalignment introduced by even partitioning. In detail, we propose a semantic attention branch to learn 5 human semantic maps. We also note that some accessories or belongings, such as a hat, bag, may provide more informative clues to improve the person Re-ID. Human semantic parsing, however, usually treats non-human parts as distractions and discards them. To fetch the missing clues, we design a branch to capture the salient non-human parts. Finally, we merge the semantic and saliency attention to build an end-to-end network, named as S2-Net. Specifically, to further improve Re-ID, we develop a trade-off weighting scheme between semantic and saliency attention and set the right weight with the actual scene. The extensive experiments show that S2-Net gets the competitive performance. S2-Net achieves 87.4% mAP on Market1501 and obtains 79.3%/56.1% rank-1/mAP on MSMT17 without semantic supervision. The source codes are available at https://github.com/upgirlnana/S2Net.
6 November 2023 in IEEE Transactions on Circuits and Systems for Video Technology
Current methods for detecting deep fakes concentrate on specific patterns of forgery like noise characteristics, local textures, or frequency statistics. These approaches assume training and test sets exhibit similar data distributions, which bring severe performance drops and further limit broader applications when migrating unseen domains. Existing works show that reconstruction learning is effective in capturing unseen forgery clues. However, 2D reconstruction is insufficient and can not handle non-frontal face reconstruction, while 3D reconstruction provides more critical details of facial structure and finds accurate forgery regions. In this paper, we propose a bi-source reconstruction based classification network (BRCNet) to incorporate 2D and 3D reconstruction as the supervisions and learn the optimal feature representation. In detail, we employ an encoder-decoder architecture to facilitate reconstruction learning, enhancing the learned representations to detect forgery patterns that are unknown. To further capture forgery evidence across multiple scales, instead of using encoder features from the reconstruction network only, we build a feature improvement network to combine feature details from encoder and decoder features in a multi-scale fashion. In addition, we use the reconstruction difference to supervise the feature aggregation, which enables detecting the subtle and trivial discrepancies between fake and real video frames. Extensive experiments are conducted to validate the performance of our proposed method on several deep fake benchmarks. The results demonstrate the efficacy of our approach, offering promising results and showcasing its potential for practical applications. The source code is available at https://github.com/cccvl/BRCNet.
25 April 2024 in IEEE Transactions on Circuits and Systems for Video Technology
Image-text matching aims to bridge vision and language areas, which is a crucial task in multi-modal intelligence. The core idea is to learn features of each modality and aggregate learned features as holistic representations to measure image-text relevance. Most existing methods involve cross-modal interaction during feature learning by modeling fine-grained relationships between two modalities for better results. However, these methods may obtain wrong attention scores when directly computing similarities between regions and words. Besides, current methods mainly rely on simple pooling operations for feature aggregation, which introduces interference from redundant information, resulting in inaccurate matching results. To alleviate these issues, we propose a novel reference-aware adaptive network for image-text matching by jointly using a reference attention module for feature learning and an adaptive aggregation module for feature aggregation. The proposed model enjoys several merits. First, the designed reference attention module effectively reduces wrong attention scores by introducing a set of references during cross-modal interaction. Second, the proposed adaptive aggregation module highlights useful information adaptively while suppressing redundant information during aggregation. Extensive experiments on two standard benchmarks demonstrate that our method performs favorably against state-of-the-art methods.
1 January 2025 in IEEE Transactions on Image Processing
In this paper, we propose a novel Transformer based approach, namely Cross-modal Contrastive Masked AutoEncoder (C2MAE), to Self-Supervised Learning (SSL) on compressed videos. A unified Transformer encoder is employed to discover relationships of visual tokens from RGBs, motion vectors and residuals. A hybrid SSL framework is proposed, which combines the complementary advantages of Masked Image Modeling (MIM) and Contrastive Learning (CL) pretext tasks, for powerful representation learning. The MIM branch extends VideoMAE by a new Fine-Grained Motion-aware Masking (FGMM) strategy and a modified Multi-modal Reconstruction (MR) task, where FGMM computes motion saliency maps as motion priors to guide the masks so that it well fits for the data properties in the compressed domain and the MR task highlights the reconstruction of raw videos by joint representations from corresponding compressed videos in addition to that in each single modality. The CL branch introduces the Contrastive Cross-modal Learning (CCL) module, and the features from a compressed video clip and the ones from its raw video counterpart are compared instead of widely used augmented data. Due to these designs, C2MAE significantly enhances interactions across modalities to compensate the sparsity of I-frames and the coarse and noisy nature of P-frames, thus delivering much stronger pre-trained models. Extensive experiments are conducted on the UCF-101, HMDB-51 and Kinetics-400 benchmarks with state-of-the-art results reported, demonstrating its effectiveness.