Fusing a low spatial resolution hyperspectral image with a high spatial resolution multispectral image has become popular for generating a high spatial resolution hyperspectral image (HR-HSI). Most methods assume that the degradation information from high resolution to low resolution is known in spatial and spectral domains. Conversely, this information is often limited or unavailable in practice, restricting their performance. Furthermore, existing fusion methods still face the problem of insufficient exploration of the cross-interaction between the spatial and spectral domains in the HR-HSI, leaving scope for further improvement. This paper proposes an unsupervised Hybrid Network of Transformer and CNN (uHNTC) for blind HSI-MSI fusion. The uHNTC comprises three subnetworks: a transformer-based feature fusion subnetwork (FeafusFomer) and two CNN-based degradation subnetworks (SpaDNet and SpeDNet). Considering the strong multi-level spatio-spectral correlation between the desired HR-HSI and the observed images, we design a Multi-level Cross-feature Attention (MCA) mechanism in FeafusFormer. By incorporating the hierarchical spatio-spectral feature fusion into the attention mechanism in the transformer, the MCA globally keeps a high spatio-spectral cross-similarity between the recovered HR-HSI and observed images, thereby ensuring the high cross-interaction of the recovered HR-HSI. Subsequently, the characteristics of degradation information are utilized to guide the design of the SpaDNet and SpeDNet, which helps FeafusFormer accurately recover the desired HR-HSI in complex real-world environments. Through an unsupervised joint training of the three subnetworks, uHNTC recovers the desired HR-HSI without pre-known degradation information. Experimental results on three public datasets and a WorldView-2 images show that the uHNTC outperforms ten state-of-the-art fusion methods. Code available: https://github.com/Caoxuheng/HIFtool.

Recently, it has become popular to obtain a high spatial resolution hyperspectral image (HR-HSI) by fusing a low spatial resolution hyperspectral image (LR-HSI) with a high spatial resolution RGB image (HR-RGB). Existing HSI super-resolution methods are designed based on a known spatial degeneration. In practice, it is difficult to obtain correct spatial degradation, which restricts the performance of existing methods. Therefore, we propose a multi-stage scheme without employing the spatial degradation model. The multi-stage scheme consists of three stages: initialization, modification, and refinement. According to the angle similarity between the HR-RGB pixel and LR-HSI spectra, we first initialize a spectrum for each HR-RGB pixel. Then, we propose a polynomial function to modify the initialized spectrum so that the RGB color values of the modified spectrum are the same as the HR-RGB. Finally, the modified HR-HSI is refined by a proposed optimization model, in which a novel, to the best of our knowledge, spectral-spatial total variation (SSTV) regularizer is investigated to keep the spectral and spatial structure of the reconstructed HR-HSI. The experimental results on two public datasets and our real-world images demonstrate our method outperforms eight state-of-the-art existing methods in terms of both reconstruction accuracy and computational efficiency.

Existing hyperspectral image (HSI) super-resolution methods fusing a high-resolution RGB image (HR-RGB) and a low-resolution HSI (LR-HSI) always rely on spatial degradation and handcrafted priors, which hinders their practicality. To address these problems, we propose a novel, to the best of our knowledge, method with two transfer models: a window-based linear mixing (W-LM) model and a feature transfer model. Specifically, W-LM initializes a high-resolution HSI (HR-HSI) by transferring the spectra from the LR-HSI to the HR-RGB. By using the proposed feature transfer model, the HR-RGB multi-level features extracted by a pre-trained convolutional neural network (CNN) are then transferred to the initialized HR-HSI. The proposed method fully exploits spectra of LR-HSI and multi-level features of HR-RGB and achieves super-resolution without requiring the spatial degradation model and any handcrafted priors. The experimental results for 32 × super-resolution on two public datasets and our real image set demonstrate the proposed method outperforms eight state-of-the-art existing methods.

Unsupervised spectral reconstruction (SR) aims to recover the hyperspectral image (HSI) from corresponding RGB images without annotations. Existing SR methods achieve it from a single RGB image, hindered by the significant spectral distortion. Although several deep learning-based methods increase the SR accuracy by adding RGB images, their networks are always designed for other image recovery tasks, leaving huge room for improvement. To overcome this problem, we propose a novel, to our knowledge, approach that reconstructs the HSI from a pair of RGB images captured under two illuminations, significantly improving reconstruction accuracy. Specifically, an SR iterative model based on two illuminations is constructed at first. By unfolding the proximal gradient algorithm solving this SR model, an interpretable unsupervised deep network is proposed. All the modules in the proposed network have precise physical meanings, which enable our network to have superior performance and good generalization capability. Experimental results on two public datasets and our real-world images show the proposed method significantly improves both visually and quantitatively as compared with state-of-the-art methods.

By fusing a low spatial resolution hyperspectral image (LR-HSI) and a high spatial resolution RGB image (HR-RGB), hybrid-resolution hyperspectral imaging has been a popular framework for acquiring high spatial resolution hyperspectral images (HR-HSIs). Existing fusion methods always employ a known spectral response function (SRF) of the RGB camera to reconstruct the HR-HSI. The SRF is often limited or unavailable in practice, restricting the performance of existing methods. To address this problem, we propose a color space transfer-based fusion strategy that obtains HR-HSIs based on a hybrid resolution hyperspectral imaging system without measuring the SRF. Specifically, using a clustered-based backpropagation neural network, the HR-RGB is mapped into the CIE XYZ color space, and the HR-XYZ is obtained. In the CIE XYZ color space, its SRF is known; thus, the the SRF measurement is successfully skipped. To efficiently fuse the HR-XYZ and the LR-HSI, we propose a polynomial fusion model that estimates the ratio matrix between the target HR-HSI and the upsampled LR-HSI. Finally, the target HR-HSI is reconstructed by combining the ratio matrix and the unsampled LR-HSI. Experimental results on two public data sets and our real-world data sets show that the proposed method outperforms five state-of-the-art fusion methods.