30 April 2025 in IEEE Transactions on Multimedia
The process of fusing low-resolution multispectral (LRMS) and high-resolution panchromatic (PAN) imagery, commonly referred to as pansharpening, is intended to generate high-resolution multispectral (HRMS) imagery. Typically, most pre-existing pansharpening frameworks mainly emphasize the straightforward learning of the mapping relationship among PAN and LRMS images to HRMS images. However, a key limitation of these frameworks is their potential overemphasis on spatial information, particularly the enhancement of low-frequency components. As a result, such an oversight potentially hinders the model's ability to simultaneously restore both spectral and spatial details. To address this issue, we propose a novel pansharpening model based on the denoising diffusion probabilistic model (DDPM), dubbed FrDiff. Specifically, we build a framelet-based conditional diffusion model that leverages the generative power of diffusion models to produce more refine results. Different from conventional methods directly inferring HRMS images, our strategy is designed to project their framelet coefficients, utilizing the available PAN and LRMS images as resources. This approach enables the separation of high-frequency and low-frequency components through framelet transformation, which are subsequently recombined to create a novel set of conditional embeddings that feed into the diffusion process. At the same time, the powerful predictive power of the diffusion model is exploited to simultaneously recover the high-frequency and low-frequency components of the HRMS. Moreover, we introduce a framelet-oriented cross-attention module dedicated to honing spectral fidelity. This module is crucial for improving the spectral precision of the HRMS images, ensuring a balanced emphasis on both spatial and spectral enhancements. Quantitative and qualitative experiments on multiple benchmark datasets demonstrate that the proposed method achieves more robustness and high-quality results than other state-of-the-art pansharpening methods.
1 November 2023 in IEEE Transactions on Geoscience and Remote Sensing
Pansharpening aims to produce a high-resolution multispectral (HRMS) image by combining a low-resolution multispectral (LRMS) image with a high-resolution panchromatic (PAN) image through a fusion process. Deep learning (DL)-based pansharpening methods have demonstrated impressive results in generating high-quality HRMS images. However, they suffer from a lack of interpretability due to their black-box network architectures. Recently, model-based deep unrolling networks have been proposed to improve the interpretability of networks. Among these approaches, the multi-source convolutional sparse coding (MCSC)-based models stand out by effectively learning common and unique features from both LRMS and PAN images, showing promising results. As the LRMS image provides limited information in MCSC-based models, it can result in weak feature response and even lead to incorrect fusion outcomes. To address this issue, we propose a novel deep MCSC-based method that enhances the robustness and performance. Specifically, we build an optimization model that integrates MCSC with a degradation model and a deep prior, which can sufficiently capture the common information shared by the latent HRMS images and PAN images, thereby enabling the recovery of more accurate spectral information. To optimize the proposed model, we adopt an iterative optimization strategy that unfolds the iterative solution into networks. Moreover, we propose an enhanced version of our method that utilizes multi-scale dictionaries to capture common and unique features at different scales, thereby facilitating the extraction of more abundant spectral and spatial details. We evaluate the effectiveness of our proposed method on multiple benchmark datasets. Experiment results demonstrate its effectiveness in improving the robustness and performance of MCSC-based models.
17 April 2025 in Information Fusion
15 March 2026 in Pattern Recognition
27 February 2025 in IEEE Transactions on Computational Imaging
Recovering a dense depth image from sparse inputs is inherently challenging. Image-guided depth completion has become a prevalent technique, leveraging sparse depth data alongside RGB images to produce detailed depth maps. Although deep learning-based methods have achieved notable success, many state-of-the-art networks operate as black boxes, lacking transparent mechanisms for depth recovery. To address this, we introduce a novel model-guided depth recovery method. Our approach is built on a maximum a posterior (MAP) framework and features an optimization model that incorporates a non-local cross-modality regularizer and a deep image prior. The cross-modality regularizer capitalizes on the inherent correlations between depth and RGB images, enhancing the extraction of shared information. Additionally, the deep image prior captures local characteristics between the depth and RGB domains effectively. To counter the challenge of high heterogeneity leading to degenerate operators, we have integrated an implicit data consistency term into our model. Our model is then realized as a network using the half-quadratic splitting algorithm. Extensive evaluations on the NYU-Depth V2 and SUN RGB-D datasets demonstrate that our method performs competitively with current deep learning techniques. Code will be released at https://github.com/Jackzhang13/NLCMR.