Depending on a large-scale paired dataset of low-resolution hyperspectral images (LrHSI), high-resolution multispectral images (HrMSI), and corresponding high-resolution hyperspectral images (HrHSI), the supervised paradigm has achieved impressive performance in the hyperspectral image super-resolution (HISR). However, the intrinsic data-intensive manner hinders its further application in real scenarios. Fortunately, deep image prior (DIP) allows us to achieve unsupervised SR by solely utilizing degraded observations. However, its potential to accurately model complicated hyperspectral priors is still not fully exploited owing to the following two factors: i) Existing methods tend to reconstruct the unknown HrHSI directly from a randomly generated noise, leaving it hard to leverage the scene-relevant information for prior learning; ii) The vanilla architecture is handcrafted for the generator network, which shows limitations in feature representation and hence fails to characterize the complicated image properties. To unleash the potential of DIP for the HISR task, we propose an enhanced DIP network, called EDIP-Net, by addressing the aforementioned impediments. Specifically, EDIP-Net is built with a two-stage four-component scheme, with a zero-shot learning (ZSL) stage for input image establishment and a deep image generation stage for prior learning. First, we exploit the cross-scale spectral relationship inside the observations and hence design a degradation learning network to generate paired training samples from the observations themselves. As such, two image-coarse estimations are derived in a ZSL manner by learning an interactive spectral learning network. By replacing random noise with two estimations, we design a double U-shape architecture for the generator network to capture their hyperspectral prior, each independently generating one HrHSI candidate. Under this premise, we further propose a degradation-aware decision fusion strategy to integrate the optimal results in a pixel-to-pixel manner. Extensive experiments demonstrate our superiority in achieving high-quality SR performance. The code will be available at https://github.com/JiaxinLiCAS.

By fusing a low-resolution hyperspectral image (LrMSI) with an auxiliary high-resolution multispectral image (HrMSI), hyperspectral image super-resolution (HISR) can generate a high-resolution hyperspectral image (HrHSI) economically. Despite the promising performance achieved by deep learning (DL), there are still two challenges remaining to be solved. First, most DL-based methods heavily rely on large-scale training triplets, which reduces them to limited generalization and poor practicability in real-world scenarios. Second, existing methods pursue higher performance by designing complex structures from off-the-shelf components while ignoring inherent information from the degradation model, hence leading to insufficient integration of domain knowledge and lower interpretability. To address those drawbacks, we propose a model-informed multistage unsupervised network, M2U-Net for short, by leveraging both deep image prior (DIP) and degradation model information. Generally, M2U-Net is built with a three-stage scheme, i.e., degradation information learning (DIL), initialized image establishment (IIE), and deep image generation (DIG) stages. The first stage is to exploit the deep information of the degradation model via a tiny network whose parameters and outputs will serve as guidance for the following two stages. Instead of feeding uninformed noise as input for stage three, the IIE stage aims to establish an initialized input with expressive HrHSI-relevant information by resorting to a spectral mapping learning network (SML-Net), thus facilitating the extraction of prior information and further magnifying the potential of DIP for high-quality reconstruction. Finally, we propose a dual U-shape network (Dual U-Net) as a powerful regularizer to capture image statistics, in which two U-Nets are coupled together by a cross-attention guidance (CAG) module to separately achieve spatial feature extraction and final image generation. The CAG module can incorporate abundant spatial information into the reconstruction process and, hence, guide the network toward a more plausible generation. Extensive experiments demonstrate the effectiveness of our proposed M2U-Net in terms of quantitative evaluation and visual quality. The code will be available at https://github.com/JiaxinLiCAS .

Hyperspectral image super-resolution can compensate for the incompleteness of single-sensor imaging and provide desirable products with both high spatial and spectral resolution. Among them, unmixing-inspired networks have drawn considerable attention owing to their straightforward unsupervised paradigm. However, most do not fully capture and utilize the multi-modal information due to their limited representation ability of constructed networks, hence leaving large room for further improvement. To this end, we propose an X-shaped interactive autoencoders network with cross-modality mutual learning between hyperspectral and multispectral data, XINet for short, to cope with this problem. Generally, it employs a coupled structure equipped with two autoencoders, aiming at deriving latent abundances and corresponding endmembers from input correspondence. Inside the network, a novel X-shaped interactive architecture is designed by coupling two disjointed U-Nets together via a parameter-shared strategy, which not only enables sufficient information flow between two modalities but also leads to informative spatial-spectral features. Considering the complementarity across each modality, a cross-modality mutual learning module is constructed to further transfer knowledge from one modality to another, allowing for better utilization of multi-modal features. Moreover, a joint self-supervised loss is proposed to effectively optimize our proposed XINet, enabling an unsupervised manner without external triplets supervision. Extensive experiments, including super-resolved results in four datasets, robustness analysis, and extension to other applications, are conducted, and the superiority of our method is demonstrated.

Due to the limitations of hyperspectral imaging systems, hyperspectral imagery (HSI) often suffers from poor spatial resolution, thus hampering many applications of the imagery. Hyperspectral super resolution refers to fusing HSI and MSI to generate an image with both high spatial and high spectral resolutions. Recently, several new methods have been proposed to solve this fusion problem, and most of these methods assume that the prior information of the point spread function (PSF) and spectral response function (SRF) are known. However, in practice, this information is often limited or unavailable. In this work, an unsupervised deep learning-based fusion method--HyCoNet--that can solve the problems in HSI-MSI fusion without the prior PSF and SRF information is proposed. HyCoNet consists of three coupled autoencoder nets in which the HSI and MSI are unmixed into endmembers and abundances based on the linear unmixing model. Two special convolutional layers are designed to act as a bridge that coordinates with the three autoencoder nets, and the PSF and SRF parameters are learned adaptively in the two convolution layers during the training process. Furthermore, driven by the joint loss function, the proposed method is straightforward and easily implemented in an end-to-end training manner. The experiments performed in the study demonstrate that the proposed method performs well and produces robust results for different data sets and arbitrary PSFs and SRFs.

Hyperspectral images (HSIs) usually have finer spectral resolution but coarser spatial resolution than multispectral images (MSIs). To obtain a desired HSI with higher spatial resolution, great research attention has been paid to achieving hyperspectral super-resolution by fusing the observed HSI with an auxiliary MSI of the same scene. However, most of the existing HSI-MSI fusion methods rely either on prior knowledge of the degradation model or on sufficient training data, hindering their practicality and interpretability. In this letter, we propose a novel unsupervised HSI-MSI fusion network with the ability of degradation adaptive learning, namely, UDALN. Specifically, we propose three modules to straightly encode the spatial and spectral transformations across resolutions, i.e., SpaDnet, SpeUnet, and SpeDnet. Through an elaborately designed three-stage unsupervised training strategy, the estimated network parameters can exhibit clear physical meanings of degradation processes and therefore help guarantee a faithful reconstruction of the desired HSI. The experimental results on two widely used hyperspectral datasets demonstrate the effectiveness of our method in comparison to the state-of-the-art HSI-MSI fusion models. (Code available at https://github.com/JiaxinLiCAS/UDALN_GRSL .)

Fusing a low-resolution hyperspectral image (LrHSI) with an auxiliary high-resolution multispectral image (HrMSI) is a burgeoning technique to realize hyperspectral image super-resolution, in which learning-based methods have dominated the mainstream direction. However, the underutilization of degradation models and strong dependence on large-scale training triplets severely impedes their applicability and performance. Considering these issues, we reformulate the fusion task as a spectral mapping problem and hence propose an unsupervised model-guided coarse-to-fine fusion network. Specifically, degradation knowledge learning is first performed to fully excavate latent model information, which will serve as guidance for better mapping learning. Following that, a coarse-to-fine fusion network is constructed with a multi-scale attentional fusion module in the head and a coarse-to-fine structure in the tail. The former is deployed to achieve a more informative compression, and the latter is adopted to capture the spectral relationship, including a spectral degradation-guided subnetwork for group-by-group coarse reconstruction and a refinement subnetwork for inter-group correlation and dependencies. Finally, high-resolution HSI can be recovered via established spectral mapping. Extensive experiments on simulated and real datasets verify the superiority of our proposed method. The code is available at https://github.com/JiaxinLiCAS/UMC2FF_GRSL.