12 June 2024 in Chinese Journal of Information Fusion
The rapid advancement of Internet technology, driven by social media and e-commerce platforms, has facilitated the generation and sharing of multimodal data, leading to increased interest in efficient cross-modal retrieval systems. Cross-modal image-text retrieval, encompassing tasks such as image query text (IqT) retrieval and text query image (TqI) retrieval, plays a crucial role in semantic searches across modalities. This paper presents a comprehensive survey of cross-modal image-text retrieval, addressing the limitations of previous studies that focused on single perspectives such as subspace learning or deep learning models. We categorize existing models into single-tower, dual-tower, real-value representation, and binary representation models based on their structure and feature representation. A key focus is placed on the fusion of modalities to enhance retrieval performance across diverse data types. Additionally, we explore the impact of multimodal Large Language Models (MLLMs) on cross-modal fusion and retrieval. Our study also provides a detailed overview of common datasets, evaluation metrics, and performance comparisons of representative methods. Finally, we identify current challenges and propose future research directions to advance the field of cross-modal image-text retrieval.
27 February 2026 in Data Science and Engineering
Cross-modal hashing retrieval faces fundamental challenges from modality-modality (M-M) and modality-label (M-L) inconsistencies inherent in multimodal data. Existing methods rely on coarse-grained disentanglement to address these inconsistencies, but suffer from inaccurate semantic separation and modality-common semantic information loss during cross-modal alignment. Through comprehensive analysis, we demonstrate that coarse-grained approaches fail to effectively alleviate modality inconsistencies. Our validation experiments show that incorporating fine-grained features yields up to \(6\%\) accuracy improvements over coarse-grained methods, confirming that fine-grained semantic components are critical for robust cross-modal retrieval. However, existing fine-grained methods require extensive pre-training and lack seamless integration into end-to-end frameworks. In this paper, we propose Inconsistency Alleviated Fine-Grained (IAFG) cross-modal hashing retrieval, a novel framework that enables semantic component-level disentanglement and alignment without extensive pre-training. Our approach introduces two key innovations: Semantic Component Disentanglement (SCD) that achieves fine-grained separation of modality-common and modality-unique information using learnable query vectors and competitive feature routing, and Fine-grained Semantic Alignment (FSA) that realizes accurate cross-modal alignment at the component level while preserving semantic details through component-level cross-attention and cross-modal triplet alignment. Extensive experiments on benchmark datasets demonstrate that our method achieves state-of-the-art performance with significant improvements in retrieval accuracy across different modalities.
11 April 2025 in Proceedings of the AAAI Conference on Artificial Intelligence
Video Moment Retrieval (VMR) aims to identify a temporal segment in an untrimmed video that best matches a given textual query. Bias in VMR is a critical issue, where the model achieves favorable results even if disregarding the video input. Existing evaluation methods, such as Resplitting, have attempted to address bias by creating out-of-distribution (OOD) datasets. However, these methods provide an incomplete definition of bias and do not quantify bias. To this end, we provide a comprehensive definition of bias in VMR, encompassing both data bias and model bias. Besides, our evaluation metrics can analyze the magnitude of these biases better. To address both data and model biases comprehensively, we introduce Reverse Distribution based VMR (ReDis-VMR). This novel approach dynamically generates datasets with inverse distributions tailored to different models based on Gaussian kernel estimation. As a result, it enables a more accurate evaluation of model performance. Building on ReDis-VMR, we further propose the Dynamic Expandable Adjustment (DEA) pipeline. DEA incrementally expands the model structure to enhance its focus on video and text features, and it incorporates a fair loss to minimize the influence of concentrated data distributions. The experimental results on bias ratio demonstrate that our ReDis method achieves state-of-the-art performance in bias elimination, while the results on moment retrieval confirm the effectiveness of our DEA framework across three evaluation methods, two datasets, and three baselines.