14 August 2024 in IEEE Transactions on Intelligent Vehicles
Feature-level fusion methods have demonstrated superior performance for visible- infrared object detection due to the deep exploration of visible and infrared features. However, most existing feature-level fusion methods utilize multiple convolutional layers with fixed parameters to extract bimodal features, leading to low adaptivity to diverse data distributions. This paper proposes a Cross-scale Dynamic Convolution-driven YOLO Fusion (CDC-YOLOFusion) network, which introduces a novel Cross-scale Dynamic Convolution Fusion (CDCF) module to adaptively extract and fuse bimodal features concerning on data distribution. Technically, CDC-YOLOFusion first designs a novel data augmentation strategy “Cross-modal Data Swapping” (CDS) to exchange local regions between visible and infrared images, effectively capturing cross-modal correlations within local regions. Building on this, the proposed CDCF utilizes cross-scale enhanced features to assist dynamic convolution prediction by introducing a disparity attention mask, which emphasizes the extraction of disparate features between two modalities. Our CDCF is effectively guided by a novel cross-modal kernel interaction loss, aiming the learned kernels to simultaneously focus on common salient features and unique features of each modality for comprehensive feature generation. Extensive experiments on three representative detection datasets demonstrate that CDCF can be easily plugined into the existing pipelines, obtaining consistent performance improvements. Moreover, our approach yields SOTA performance with about 2% to 3% mAP improvements as compared to the state-of-the-art methods. The code has been published in https://github.com/Morph-3800/CDC_YOLOFusion .
10 July 2025 in IEEE Transactions on Circuits and Systems for Video Technology
Visible-infrared image pairs provide complementary information, enhancing the reliability and robustness of object detection applications in real-world scenarios. However, most existing methods face challenges in maintaining robustness under complex weather conditions, which limits their applicability. Meanwhile, the reliance on attention mechanisms in modality fusion introduces significant computational complexity and storage overhead, particularly when dealing with high-resolution images. To address these challenges, we propose the Cross-modality Fusion Mamba with Weather-removal (CFMW) to augment stability and cost-effectiveness under adverse weather conditions. Leveraging the proposed Perturbation-Adaptive Diffusion Model (PADM) and Cross-modality Fusion Mamba (CFM) modules, CFMW is able to reconstruct visual features affected by adverse weather, enriching the representation of image details. With efficient architecture design, CFMW is 3 times faster than Transformer-style fusion (e.g., CFT). To bridge the gap in relevant datasets, we construct a new Severe Weather Visible-Infrared (SWVI) dataset, encompassing diverse adverse weather scenarios such as rain, haze, and snow. The dataset contains 64, 281 paired visible-infrared images, providing a valuable resource for future research. Extensive experiments on public datasets (i.e., M3FD and LLVIP) and the newly constructed SWVI dataset conclusively demonstrate that CFMW achieves state-of-the-art detection performance. Both the dataset and source code will be made publicly available at https://github.com/lhy-zjut/CFMW.
6 September 2024 in IEEE Transactions on Intelligent Transportation Systems
Temporal information plays a pivotal role in Bird’s-Eye-View (BEV) driving scene understanding, which can alleviate the visual information sparsity. However, the indiscriminate temporal fusion method will cause the barrier of feature redundancy when constructing vectorized High-Definition (HD) maps. In this paper, we revisit the temporal fusion of vectorized HD maps, focusing on temporal instance consistency and temporal map consistency learning. To improve the representation of instances in single-frame maps, we introduce a novel method, DTCLMapper. This approach uses a dual-stream temporal consistency learning module that combines instance embedding with geometry maps. In the instance embedding component, our approach integrates temporal Instance Consistency Learning (ICL), ensuring consistency from vector points and instance features aggregated from points. A vectorized points pre-selection module is employed to enhance the regression efficiency of vector points from each instance. Then aggregated instance features obtained from the vectorized points preselection module are grounded in contrastive learning to realize temporal consistency, where positive and negative samples are selected based on position and semantic information. The geometry mapping component introduces Map Consistency Learning (MCL) designed with self-supervised learning. The MCL enhances the generalization capability of our consistent learning approach by concentrating on the global location and distribution constraints of the instances. Extensive experiments on well-recognized benchmarks indicate that the proposed DTCLMapper achieves state-of-the-art performance in vectorized mapping tasks, reaching $61.9\%$ and $65.1\%$ mAP scores on the nuScenes and Argoverse datasets, respectively. The source code is available at https://github.com/lynn-yu/DTCLMapper.
2 January 2025 in IEEE Transactions on Instrumentation and Measurement
As an essential component of Vision-Based Measurement (VBM), Referring Multi-Object Tracking (RMOT) involves localizing and tracking specific objects in video frames using linguistic prompts as references. To enhance the effectiveness of linguistic prompts when training, we introduce a novel Multi-Granularity Localization Transformer with collaborative understanding, termed MGLT. Unlike previous methods focused on visual-language fusion and post-processing, MGLT reevaluates RMOT by preventing linguistic clues from attenuating during propagation. MGLT comprises two key components: Multi-Granularity Implicit Query Bootstrapping (MGIQB) and Multi-Granularity Track-Prompt Alignment (MGTPA). MGIQB ensures that tracking and linguistic features are preserved in later layers of network propagation by bootstrapping the model to generate text-relevant and temporal-enhanced track queries. Simultaneously, MGTPA with multi-granularity linguistic prompts enhances the model’s localization ability by understanding the relative positions of different referred objects within a frame. Extensive experiments on well-recognized benchmarks demonstrate that MGLT achieves state-of-the-art performance. Notably, it shows significant improvements on Refer-KITTI dataset of 2.73%, 7.95%, and 3.18% in HOTA, AssA, and IDF1, respectively. The code will be available at https://github.com/JiajunChern/MGLT.
1 January 2026 in IEEE Open Journal of Intelligent Transportation Systems