26 November 2023 in Information Processing & Management
23 August 2025 in Information Processing & Management
7 June 2024 in IEEE Transactions on Circuits and Systems for Video Technology
To meet users’ demands for video retrieval, text-video cross-modal retrieval technology continues to evolve. Methods based on pre-trained models and transfer learning are widely employed in designing cross-modal retrieval models, significantly enhancing the accuracy of video retrieval. However, these methods exhibit shortcomings when it comes to studying the relationships between video frames, preventing the model from fully establishing the hidden semantic relationships within video features. To further deduce the implicit semantic relationships among video frames, we propose a cross-modal retrieval model based on graph convolutional networks (GCN) and visual semantic inference (GVSI). The GCN is utilized to establish relationships between video frame features, facilitating the mining of hidden semantic information across video frames. In order to use text semantic features to help the model to infer temporal and implicit semantic information between video frames, we introduce a semantic mining and temporal space (SM&TS) inference module. Additionally, we design semantic alignment modules (SA_M) to align explicit and implicit object features present in both video and text. Finally, we analyze and validate the effectiveness of the model using MSR-VTT, MSVD, and LSMDC datasets.
14 January 2025 in IEEE Transactions on Circuits and Systems for Video Technology
Video-text cross-modal retrieval is widely studied to improve retrieval accuracy. However, the security of video-text cross-modal retrieval models receives little attention. If attackers exploit the security vulnerabilities in these models, it poses a significant threat to the retrieval models. Thus, identifying security flaws in video-text cross-modal retrieval models becomes the focus of our research. We are the first to design a video poisoning model to uncover security vulnerabilities in retrieval models. Existing poisoning models have certain limitations when it comes to exploiting vulnerabilities in retrieval models. These include failing to comprehensively embed malicious information into the original video and being unable to maintain visual consistency between the original and poisoned videos. These limitations can result in unsuccessful attacks on retrieval models and an inability to effectively identify security flaws within them. To address these shortcomings, we design an efficient poisoning model that embeds malicious information thoroughly into the original clean data to attack video-text cross-modal retrieval models. We are the first to use a poisoning model to attack retrieval models, thereby uncovering their security vulnerabilities. Second, we introduce a bi-level poisoning module to ensure that malicious information is thoroughly embedded into the original video, thereby enhancing the attack capability of the poisoning model. Finally, we design an adversarial module to improve visual consistency between the original and poisoned videos, thus enhancing the concealment of malicious information within the training data of retrieval models. Our poisoning model can identify security flaws in video-text cross-modal retrieval models, providing insights into improving the security of retrieval models. The effectiveness of our model is validated on the MSR-VTT, LSMDC, and MSVD datasets.
4 June 2025 in IEEE Transactions on Circuits and Systems for Video Technology
Video, as an information carrier, provides a vast amount of important information to people. Therefore, the method of obtaining video becomes particularly important, which drives the research on text-video cross-modal retrieval technology. However, current text-video cross-modal retrieval models still face several issues. First, these models do not fully utilize the powerful reasoning and generative capabilities of large models to address the issues of missing critical objects and insufficient high-quality video-text paired training data. Second, existing retrieval models do not adequately research the bidirectional cross-modal semantic interaction and reasoning mechanism, which hinders the ability to fully capture and learn the implicit semantic features between different modalities. To address these issues, we propose an innovative bidirectional semantic reasoning and large model data augmentation cross-modal retrieval model (BiSeR-LMA). This model first leverages the strong reasoning and generative capabilities of large models to perform semantic reasoning on the textual descriptions of videos, then generates multiple semantically rich video frames, thereby compensating for the missing critical objects in the original video and improving the quality of video-text paired training data. Second, we design a bidirectional text-video semantic reasoning module, which uses features from one modality as auxiliary information to assist the model in reasoning the implicit semantic information of another modality. This enhances the model’s capability to establish semantic relationships and perform reasoning on implicit semantics, promoting text-video semantic alignment. Finally, we verify the effectiveness of the proposed cross-modal retrieval model on the MSR-VTT, LSMDC, and MSVD datasets. The source codes of our method are available at: https://anonymous.4open.science/r/BiSeR_LMA.