14 August 2024 in IEEE Transactions on Intelligent Vehicles
Feature-level fusion methods have demonstrated superior performance for visible- infrared object detection due to the deep exploration of visible and infrared features. However, most existing feature-level fusion methods utilize multiple convolutional layers with fixed parameters to extract bimodal features, leading to low adaptivity to diverse data distributions. This paper proposes a Cross-scale Dynamic Convolution-driven YOLO Fusion (CDC-YOLOFusion) network, which introduces a novel Cross-scale Dynamic Convolution Fusion (CDCF) module to adaptively extract and fuse bimodal features concerning on data distribution. Technically, CDC-YOLOFusion first designs a novel data augmentation strategy “Cross-modal Data Swapping” (CDS) to exchange local regions between visible and infrared images, effectively capturing cross-modal correlations within local regions. Building on this, the proposed CDCF utilizes cross-scale enhanced features to assist dynamic convolution prediction by introducing a disparity attention mask, which emphasizes the extraction of disparate features between two modalities. Our CDCF is effectively guided by a novel cross-modal kernel interaction loss, aiming the learned kernels to simultaneously focus on common salient features and unique features of each modality for comprehensive feature generation. Extensive experiments on three representative detection datasets demonstrate that CDCF can be easily plugined into the existing pipelines, obtaining consistent performance improvements. Moreover, our approach yields SOTA performance with about 2% to 3% mAP improvements as compared to the state-of-the-art methods. The code has been published in https://github.com/Morph-3800/CDC_YOLOFusion .
21 March 2025 in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
Visible-infrared object detection has been widely applied in multimodal remote sensing image perception tasks due to the strong complementarity between the two modalities. However, visible-infrared remote sensing data often exhibits long-tail distribution characteristics, where some categories have sparse samples, resulting in insufficient training and poor detection performance for tail categories. To address this issue, this paper proposes an “Equalized Multi-modal Cues Fusion Transformer” (EMCFormer), incorporating an innovative “Multi-modal Heterogeneous Cues Aggregation” (MHCA) module and “Equalized-Adaptive Focal Loss” (EAFL). Specifically, MHCA leverages a cross-modal self-attention mechanism with Gumbel Softmax to generate fused features and enhance the learning of tail category features. By introducing Gumbel random noise, MHCA effectively increases attention on sparse data from tail categories, thereby producing robust fused features that enhance detection performance for these categories. Additionally, EAFL dynamically amplifies the contribution of tail categories by using a dynamic focusing factor, improving performance for tail category detection. Extensive experiments on well-recognized datasets demonstrate that EMCFormer effectively improves detection accuracy for tail categories and mitigates the challenges posed by long-tail data distribution.
1 January 2026 in IEEE Transactions on Geoscience and Remote Sensing