26 September 2024 in Expert Systems with Applications
19 October 2023 in IEEE Transactions on Instrumentation and Measurement
Deep learning has become a popular technique for infrared and visible image fusion due to its powerful feature representation abilities. Existing methods often employ the CNN, Transformer, or mixed CNN-Transformer to treat different modalities indiscriminately. However, we observe that these homogeneous fusion networks (i.e., same network for cross-modal) struggle to handle the specific characteristics of each modality, leading to fused images being contaminated with interference or lacking modality-specific information. To alleviate this issue, we propose a heterogeneous Dual-branch multi-cascade fusion network based on CNN and Transformer, named DCTNet, aims independently to maintain the focus of respective modalities, such as radiation intensity of infrared images and texture details of visible images. In DCTNet, the CNN branch is specifically designed to exploit local features in visible images by utillizing stacked residual dense CNNs, while the Transformer branch is tailored to model the long-range dependencies, capturing the overall temperature distribution and thermal patterns of a scene in infrared images by cascading residual Transformers. In addition, we introduce an Adaptive Fusion Interaction Module (AFIM) that leverages attention mechanisms to adaptively high-light informative regions in the fused feature maps. The module assigns weights to these regions based on their contributions and dynamically merges features from the dual-branch at multiple levels. Experimental results on three public datasets demonstrate that the proposed method outperforms state-of-the-art methods in terms of quantitative and qualitative evaluations. Moreover, we showcase the promising performance of DCTNet in downstream object detection and semantic segmentation applications on a widely-accepted benchmark.
16 February 2026 in IEEE Transactions on Multimedia
CNN-Transformer hybrid architecture have demonstrated remarkable success in image fusion. However, existing studies often oversimplify the respective strengths of CNNs and Transformers, attributing them solely to their abilities to capture local and global information. This characterization is neither precise nor reflective of their fundamental differences, leading to network designs that rely heavily on empirical practices rather than theoretical insights. To address this limitation and enhance the interpretability of fusion networks, we conduct a Fourier analysis to elucidate the distinct roles of CNNs and Transformers from a novel frequency perspective. Our analysis reveals that CNNs and Transformers exhibit contrasting properties in the frequency domain. However, prior CNN-Transformer hybrid architectures have failed to leverage these characteristics, resulting in a loss of low frequency contextual information. Based on this finding, we propose the Frequency Complementary learning and Rebalancing Fusion Network (FCRNet), which leverages the complementary behaviors of CNNs and Transformers to optimize image fusion. Firstly, a frequency complementary learning mechanism is utilized to integrate the strengths of CNNs and Transformers in representation learning, effectively capturing essential low frequency features and rich high frequency details. Secondly, a frequency rebalancing module is designed to address the imbalance between high and low frequency components in the synthesized feature maps, ensuring their frequency distribution aligns with that of natural images. Extensive experiments on multiple benchmarks validate the effectiveness and superiority of our proposed FCRNet. Additionally, experiments on downstream tasks validates the practicality of FCRNet.
15 December 2025 in IEEE Transactions on Circuits and Systems for Video Technology
Fully sparse fusion makes an excellent balance between efficiency and accuracy in multi-modal 3D object detection. However, most existing methods focus on foreground objects while overlooking background context. This oversight compromises detection robustness, especially for occluded or small-sized objects, leading to suboptimal detection performance. To address this limitation, we propose a novel fully sparse fusion framework (PLPFusion), which introduces a hierarchical Plane-Line-Pixel representation to progressively model the object-context relationships. PLPFusion comprises three key modules: the Plane Enhancement Module (PEM), the Line Alignment Module (LAM) and the Pixel-Level Aggregation Module (PLAM). Firstly, PEM utilizes geometric cues from LiDAR feature planes to generate spatially-aware object queries. Secondly, LAM further refines these queries with geometric priors for semantic awareness. Lastly, PLAM aggregates pixel-level context to enhance discriminative completeness by leveraging the semantically-aware object queries. On the nuScenes benchmark, PLPFusion achieves 71.9% mAP and 74.0% NDS, outperforming the baseline method FUTR3D by +2.5% mAP and +1.9% NDS, respectively. On the KITTI benchmark, it achieves 72.68% BEV mAP and 67.39% 3D mAP. These results confirm its robustness and effectiveness in diverse multi-modal 3D scenarios. The code of PLPFusion is available on the https://github.com/Text357/PLPFusion.
30 July 2026 in SSRN Electronic Journal
Multi-Modality Image Fusion (MMIF) aims to reconstruct an informative latent image from complementary sensor observations and is commonly formulated as an inverse problem within a Bayesian framework. Due to its ill-posed nature, the recovery process critically depends on prior knowledge to constrain the solution space and preserve underlying structural characteristics. However, current methods primarily rely on hand-crafted, task-specific priors, which not only introduce subjective biases but also undermine generalizability and robustness across diverse real-world scenarios. To address this issue, we reformulate MMIF within a bias-free Bayesian framework. Each modality is modeled as an independent degradation of the fused image, transforming conventional predefined priors into cross-modality conditional priors and enabling a fully data-driven fusion process. Building on this foundation, we propose the Symmetric Degradation Complementarity Unrolling Network (SDCUNet), which explicitly unrolls the optimization process into a symmetric and interpretable architecture. Each stage of SDCUNet comprises a Degradation Awareness Module (DAM) to model the modality-specific degradation, and a Complementarity Enhancement Module (CEM) to extract structural guidance from cross-modality conditional priors. Additionally, we introduce a Local Gradient-Guided (LGG) pixel-level loss function to adaptively quantify each modality's contribution, further enhancing the model's robustness against complex environmental degradations. Extensive experiments on five infrared-visible and two medical image datasets demonstrate that SDCUNet consistently achieves superior visual quality and quantitative metrics, exhibiting remarkable robustness even under challenging degradation conditions, while improving the downstream detection performance. The code is available at https://github.com/op-dog/SDCUNet.