13 October 2023 in IEEE Transactions on Instrumentation and Measurement
This paper tackles the challenging yet significant task of grounding a natural language query to the corresponding region onto an image. The main challenge in visual grounding is to model the correspondence between visual context and semantic concept referred by the language expression, that is, multi-modal fusion. Nevertheless, there is an inherent deficiency in the current fusion module designs, which makes visual and linguistic feature embeddings cannot be unified into the same semantic space. To address the issue, we present a novel and effective visual grounding framework based on joint multi-modal representation and interaction (JMRI). Specifically, we propose to perform image-text alignment in a multi-modal embedding space learned by a large-scale foundation model, so as to obtain semantically unified joint representations. Furthermore, the transformer-based deep interactor is designed to capture intra-modal and inter-modal correlations, rendering our model to highlight the localization-relevant cues for accurate reasoning. By freezing the pre-trained vision-language foundation model and updating the other modules, we achieve the best performance with the lowest training cost. Extensive experimental results on five benchmark datasets with quantitative and qualitative analysis show that the proposed method performs favorably against the state-of-the-arts.
27 December 2024 in IEEE Transactions on Intelligent Transportation Systems
Vision-language tracking is a new rising topic in intelligent transportation systems, particularly significant in autonomous driving and road surveillance. It is a task that aims to combine visual and auxiliary linguistic modalities to co-locate the target object in a video sequence. Currently, multi-modal data scarcity and burdensome modality fusion have become two major factors in limiting the development of vision-language tracking. To tackle the issues, we propose an efficient and effective one-stage vision-language tracking framework (CPIPTrack) that unifies feature extraction and multi-modal fusion by interactive prompt learning. Feature extraction is performed by the high-performance vision-language foundation model CLIP, resulting in the impressive generalization ability inherited from the large-scale model. Modality fusion is simplified to a few lightweight prompts, leading to significant savings in computational resources. Specifically, we design three types of prompts to dynamically learn the layer-wise feature relationships between vision and language, facilitating rich context interactions while enabling the pre-trained CLIP adaptation. In this manner, discriminative target-oriented visual features can be extracted by language and template guidance, which are used for subsequent reasoning. Due to the elimination of extra heavy modality fusion, the proposed CPIPTrack shows high efficiency in both training and inference. CPIPTrack has been extensively evaluated on three benchmark datasets, and the experimental results demonstrate that it achieves a good performance-speed balance with an AUC of 66.0% on LaSOT and a runtime of 51.7 FPS on RTX2080 Super.
2 May 2025 in Pattern Recognition Letters
6 June 2023 in IEEE Access
Recent advances in deep learning have enabled state-of-the-art performance in detecting medium and large-size objects. However, small object detection remains challenging primarily due to the scarcity of information. This paper proposes an end-to-end fusion network that integrates deep and hand-crafted features to address this limitation. A fusion module based on semantic context information is designed to enhance feature discrimination ability. Additionally, we introduce a kind of feature-contrast loss to incorporate prior knowledge into the learning of deep feature according to contrastive learning. Experiments on the MS COCO dataset (34.4% AP S ) demonstrate that our approach achieves improved detection accuracy over previous methods, especially for small objects.