13 October 2023 in IEEE Transactions on Instrumentation and Measurement
This paper tackles the challenging yet significant task of grounding a natural language query to the corresponding region onto an image. The main challenge in visual grounding is to model the correspondence between visual context and semantic concept referred by the language expression, that is, multi-modal fusion. Nevertheless, there is an inherent deficiency in the current fusion module designs, which makes visual and linguistic feature embeddings cannot be unified into the same semantic space. To address the issue, we present a novel and effective visual grounding framework based on joint multi-modal representation and interaction (JMRI). Specifically, we propose to perform image-text alignment in a multi-modal embedding space learned by a large-scale foundation model, so as to obtain semantically unified joint representations. Furthermore, the transformer-based deep interactor is designed to capture intra-modal and inter-modal correlations, rendering our model to highlight the localization-relevant cues for accurate reasoning. By freezing the pre-trained vision-language foundation model and updating the other modules, we achieve the best performance with the lowest training cost. Extensive experimental results on five benchmark datasets with quantitative and qualitative analysis show that the proposed method performs favorably against the state-of-the-arts.
30 September 2024 in IEEE Transactions on Signal Processing
This paper considers the design of constant-envelope waveforms for multiple-input-multiple-output (MIMO) dual-function radar communication (DFRC) systems. The purpose is to improve the angle estimation performance of the multiple signal classification (MUSIC) algorithm through minimizing its asymptotic estimation error bound. To guarantee the communication performance, we enforce an energy constraint as well as a cosine similarity-based constraint on the synthesized communication signals. Additionally, we constrain the energy transmitted toward each potential eavesdropper to a low level, which prevents the waveforms from being intercepted. To tackle the encountered non-convex optimization problem, we develop two iterative algorithms. The first algorithm is based on the minorization-maximization method, wherein we construct a quadratic surrogate to minorize the approximate objective function. Then we use the alternating direction method of multipliers (ADMM) to tackle the quadratic programming problem. In the second algorithm, we directly tackle the waveform design problem by the ADMM method. Numerical examples are provided to show that through elaborate waveform design, the MIMO DFRC system is capable to realize high-performance target localization and covert communications simultaneously.
27 December 2024 in IEEE Transactions on Intelligent Transportation Systems
Vision-language tracking is a new rising topic in intelligent transportation systems, particularly significant in autonomous driving and road surveillance. It is a task that aims to combine visual and auxiliary linguistic modalities to co-locate the target object in a video sequence. Currently, multi-modal data scarcity and burdensome modality fusion have become two major factors in limiting the development of vision-language tracking. To tackle the issues, we propose an efficient and effective one-stage vision-language tracking framework (CPIPTrack) that unifies feature extraction and multi-modal fusion by interactive prompt learning. Feature extraction is performed by the high-performance vision-language foundation model CLIP, resulting in the impressive generalization ability inherited from the large-scale model. Modality fusion is simplified to a few lightweight prompts, leading to significant savings in computational resources. Specifically, we design three types of prompts to dynamically learn the layer-wise feature relationships between vision and language, facilitating rich context interactions while enabling the pre-trained CLIP adaptation. In this manner, discriminative target-oriented visual features can be extracted by language and template guidance, which are used for subsequent reasoning. Due to the elimination of extra heavy modality fusion, the proposed CPIPTrack shows high efficiency in both training and inference. CPIPTrack has been extensively evaluated on three benchmark datasets, and the experimental results demonstrate that it achieves a good performance-speed balance with an AUC of 66.0% on LaSOT and a runtime of 51.7 FPS on RTX2080 Super.
22 July 2024 in IEEE Transactions on Aerospace and Electronic Systems
Reconfigurable intelligent surface (RIS) refers to a signal reflection surface composed of numerous low-cost passive reflecting elements. RIS can improve the performance of radar and communication systems by dynamically modulating the wireless channels. In this paper, we consider the co-design for improving the co-existence between multiple-input-multiple-output (MIMO) radar and MIMO communication system with the assistance of RIS. The design objective is to improve the radar detection performance while ensuring reliable communication capabilities. The optimization problem associated with this co-design task is challenging due to the nonconvex nature resulting from the unimodular constraint on RIS coefficients and the constant-envelope constraint on the radar transmit waveforms. To address this problem, we propose a cyclic method that combines minorization-maximization, semi-definite programming, and alternating direction method of multipliers. Numerical examples are provided to validate the effectiveness of the proposed algorithm.
17 February 2025 in IEEE Transactions on Aerospace and Electronic Systems
Analyzing and recognizing the modulation types of radar pulse repeated intervals (PRIs) is a critical aspect of electronic support measurement (ESM), as it is closely associated with the operational modes of radar emitters. This study proposes the histogram-informed dual branches network (HIDB-Net) to recognize modulation types of PRI by learning the histogram patterns of PRI and DPRI simultaneously. To this end, vectorized histogram features of PRI and DPRI sequences are extracted and utilized as the input for the network. Subsequently, these features are separately processed by dual branches' one-dimension convolution neural network (1DCNN) layers and fused as a whole for modulation recognition through the multi-head self-attention (MHSA) module. Finally, HIDB-Net infers the recognition result via a fully connected (FC) layer followed by softmax operation. In contrast to previous methods, our approach is capable of handling PRI sequences of arbitrary length with fewer feature dimensions, at the same time achieving superior recognition performance. Experimental results validate the robustness and effectiveness of our method.
4 December 2024 in IEEE Transactions on Circuits and Systems for Video Technology
Vision-Language Tracking (VLT) aims to predict the target state in video sequences using two types of heterogeneous information: 1) the static text description detailing main characteristics of the tracked object, and 2) the dynamic image patches containing the target and its surroundings. However, as the tracking proceeds, inconsistencies may arise between the linguistic information embedded in the text description and the visual representations stored in the search images. In such cases, the direct fusion of vision and language could result in conflicts. To tackle this issue, we propose MugTracker, which integrates image-to-text generation into the VLT framework and attempts a generative updating way to mitigate the effects of inconsistencies. Specifically, we design two branch tasks: multi-modal understanding for reasoning and multi-modal generation for updating. We develop a dynamic text generator based on the hybrid architecture of the pre-trained foundation model BLIP and adaptively update the text reference as the context varies for more accurate target modeling. The semantically consistent visual and linguistic representations are then aligned and associated by the reasoning branch built on the BLIP dual-encoder to infer the target state. To better transfer the foundation model to build a strong tracker, we introduce the proposed TE-Adapter in the visual components for target enhancement and Text-Adapter in the linguistic components to strengthen the learning of discriminative semantics. Our MugTracker has been extensively evaluated on three datasets, and the superior performance compared to the state-of-the-arts demonstrates its effectiveness.