4 August 2021 in IEEE Transactions on Computational Imaging
This paper presents a mosaic convolution-attention network (MCAN) for demosaicing spectral mosaic images captured using multispectral filter array (MSFA) imaging sensors. MSFA-based multispectral imaging systems acquire multispectral information of a scene in a single snap-shot operation. A complete multispectral image is reconstructed by demosaicing an MSFA-based spectral mosaic image. To avoid aliasing and artifacts in demosaicing, we utilize joint spatial-spectral correlation in a raw mosaic image. The proposed MCAN includes a mosaic convolution module (MCM) and a mosaic attention module (MAM). The MCM extracts features via a learning approach with a margin between splitting the periodic spectral mosaic and keeping the underlying spatial information of the raw image. Based on the strategy of position-sensitive weight sharing, MCM assigns the same weight to pixels with the same relative position in an MSFA. The MAM uses a position-sensitive feature aggregation strategy to describe the loading of mosaic patterns within the feature maps, which gradually reduces mosaic distortion through the attention mechanism. The experimental results on synthetic as well as real-world data show that the proposed scheme outperforms state-of-the-art methods in terms of spatial details and spectral fidelity.
22 February 2024 in IEEE Transactions on Image Processing
This paper presents a deep learning-based spectral demosaicing technique trained in an unsupervised manner. Many existing deep learning-based techniques relying on supervised learning with synthetic images, often underperform on real-world images, especially as the number of spectral bands increases. This paper presents a comprehensive unsupervised spectral demosaicing (USD) framework based on the characteristics of spectral mosaic images. This framework encompasses a training method, model structure, transformation strategy, and a well-fitted model selection strategy. To enable the network to dynamically model spectral correlation while maintaining a compact parameter space, we reduce the complexity and parameters of the spectral attention module. This is achieved by dividing the spectral attention tensor into spectral attention matrices in the spatial dimension and spectral attention vector in the channel dimension. This paper also presents Mosaic 25, a real 25-band hyperspectral mosaic image dataset featuring various objects, illuminations, and materials for benchmarking purposes. Extensive experiments on both synthetic and real-world datasets demonstrate that the proposed method outperforms conventional unsupervised methods in terms of spatial distortion suppression, spectral fidelity, robustness, and computational cost. Our code and dataset are publicly available at https://github.com/polwork/Unsupervised-Spectral-Demosaicing.
13 May 2024 in IEEE Transactions on Circuits and Systems for Video Technology
This paper presents a hyperspectral image (HSI) reconstruction technique based on physics-driven optimization of multispectral filter array (MSFA) patterns. The encoding of HSIs using an MSFA and their decoding through deep learning has gained increasing attention. However, previous studies have seldom explored pattern optimization from a physical perspective during the encoding process. In this paper, we apply a spectral sensitivity function (SSF) response model to generate the MSFA, and the goal of encoder optimization extends from SSF to physical structural parameters. To fully utilize spatial and spectral information in the decoding process, we design an end-to-end dual-branch spatial-spectral fusion network (DSFNet). By jointly optimizing the MSFA with the SSF response model and DSFNet, the proposed method significantly improves the reconstruction accuracy of HSI. When compared with existing HSI reconstruction methods, our proposed approach achieves state-of-the-art performance in both metric and visual quality.
20 November 2023 in Optics and Lasers in Engineering
20 February 2025 in IEEE Transactions on Geoscience and Remote Sensing
Hyperspectral Images (HSIs) play a pivotal role in fields like medical diagnosis and agriculture. However, it often contends with significant noise stemming from narrow-band spectral filtering. Existing denoising techniques have their limitations: model-driven methods rely on manual priors and hyperparameters, while learning-based methods struggle to discern intrinsic noise patterns, as they require paired images with specific example noise for training, fail to capture critical noise distribution information, leading to unrobust denoising results. This work addresses the issue by presenting a Degradation-Noise-Aware Unfolding Network (DNA-Net). Unlike training directly with simulated noise, DNA-Net initially models general sparse and Gaussian noise through statistic distributions. It then explicitly represents image priors with a customized spectral transformer. The model is subsequently unfolded into an end-to-end network, with hyperparameters adaptively estimated from noisy HSI and degradation models, effectively regulating each iteration. Furthermore, a novel U-Shaped Local-Non-local-Spectral Transformer (U-LNSA) is introduced, simultaneously capturing spectral correlations, local features, and non-local dependencies. The integration of U-LNSA into DNA-Net establishes the first Transformer-based deep unfolding method for HSI denoising. Experimental results on synthetic and real noise validate DNA-Net’s superior performance over state-of-the-art (SOTA) methods. Moreover, the DNA-Net, trained exclusively on mixed Gaussian noise and impulse noise, demonstrates the ability to generalize to unseen noise present in real images. Code and models will be released at https://github.com/NavyZeng/DNA-Net.