In order to preserve the spatial and spectral information of the original panchromatic and multi-spectral images, this study designs a loss function suitable for pan-sharpening and a four-layer convolution neural network that could adequately extract spectral and spatial features from original source images. The major advantage of this study is that the designed loss function does not need the reference fused image, and then the proposed pan-sharpening method does not need to make the simulation data for training. This is the big difference from most existing pan-sharpening methods. Moreover, the loss function takes into account the characteristics of remote sensing images, including the spatial and spectral evaluation indicators. We also add the feature enhancement layer in convolution neural network, thus, the proposed four-layer network contains feature extraction, feature enhancement, linear mapping and reconstruction. In order to evaluate the effectiveness and universality of the proposed fusion model, we selected thousands of remote sensing images which include different sensors, different times and different land-cover types to make the training dataset. By evaluating the performance on the WorldView-2, Pleiades and Gaofen-1 experimental data, the results show that the proposed method achieves optimal performance in terms of both the subjective visual effect and the object assessment.The code is available at https://github.com/Zhangxi-Xiong/pan-sharpening

Recently, deep learning has become one of the most popular tools for pansharpening, many relevant methods have been investigated and reflected great performance. However, a non-negligible problem is the absence of ground-truth (GT). A common solution is using degraded images as training input and the original images are employed as GT. The learned mapping between low resolution (LR) and high resolution (HR) is simulated, is not real, which may cause spectral distortion or insufficient spatial texture enhancement of fused images. In order to address the drawback, a novel unsupervised attention pansharpening net (UAP-Net) is proposed. The proposed UAP-Net mainly contains two major components: 1) the deep residual network (DRN) and 2) spatial texture attention block (STAB). The DRN aims to extract spectral features and spatial details features from low-resolution multi-spectral (LRMS) and panchromatic (PAN), and to fuse those features to make them more representative. The designed STAB adopts the high-frequency component of corresponding input PAN as the weight to enhance the spatial details of the residual block output features. Moreover, a new loss function including two spatial losses and two spectral losses are established. The losses are calculated in the spatial domain and the frequency domain, respectively. Experiments on Gaofen-2 and Worldview-2 remote sensing data demonstrate that the proposed UAP-Net could fuse PAN and LRMS images effectively without the help of high-resolution multi-spectral (HRMS). The proposed framework is fully general and can be used for many multisource remote sensing image fusion, and achieves optimal performance in terms of both the subjective visual effect and the quantitative evaluation.

While snapshot hyperspectral cameras are cheaper and faster than imagers based on pushbroom or whiskbroom spatial scanning, the output imagery from a snapshot camera typically has different spectral bands mapped to different spatial locations in a mosaic pattern, requiring a demosaicing process to be applied to generate the desired hyperspectral image with full spatial and spectral resolution. However, many existing demosaicing algorithms suffer common artifacts such as periodic striping or other forms of noise. To ameliorate these issues, a hyperspectral demosaicing framework that couples a preliminary demosaicing network with a separate multi-stage progressive denoising network is proposed, with both networks employing transformer and attention mechanisms. A multi-term loss function permits supervised network training to monitor not only performance of the preliminary demosaicing but also denoising at each stage. An extensive collection of experimental results demonstrate that the proposed approach produces demosaiced images with not only fewer visual artifacts but also improved performance with respect to several quantitative measures as compared to other state-of-the-art demosaicing methods from recent literature.

Most of the existing fusion algorithms are not robust to unregistered input images. Even after image registration, nonlinear nonregistration may persist in the local areas of the images, leading to poor quality in the fused image. So, as to tackle these challenges, a progressive remote sensing image registration and fusion network is proposed in this article, and named PRF-Net, which is particularly useful when two images are from different platforms. First, a registration network is designed to register the input image patches, which includes a global spatial transform network (GSTN) and a local spatial warp network (LSWN). The GSTN is primarily used for coarse registration, applying rigid transformation to globally align the input images. After coarse registration, the preliminarily registered moving image is input into the LSWN for local fine-tuning to maximize correlation between the input image patches. Subsequently, the fine registered images are degraded and input into the fusion network to generate the fused image. To maintain sufficient spectral and spatial information of the fused image, a multiscale feature extraction (MSFE) block with a highly interpretable spatial details attention (SDA) block is designed, which can enhance the ability of fusion network to extract and preserve spatial details and spectral information. Three groups of experiments conducted on four types of remote sensing images give evidence of that the proposed PRF-Net exhibits excellent performance in both reduced and full resolutions, showcasing its outstanding registration and fusion quality.

In order to adequately preserve the spatial information of panchromatic (PAN) image and the spectral information of multispectral (MS) image, this study proposes a method to learn the spectral information of MS based on the PAN image. In the spectral learning model, the PAN is degraded to the spatial resolution of the MS. Then, a deep convolutional neural network (CNN) is adopted to learn the spectral information between the degraded PAN and the original MS. In addition, the spectral evaluation index—spectral angle mapper (SAM) is used for controlling the spectral losses. Finally, the high spatial resolution MS (HMS) can be obtained through the trained model by using the original PAN as the test input. Besides, seven representative pan-sharpening algorithms and seven widely recognized objective fusion metrics are used to compare and evaluate the performance on the WorldView-2 experimental data, respectively. The results show that the proposed method achieves the purpose of pan-sharpening well, especially maintaining the optimal spectral information.

In order to overcome the lack of the multispectral image (MS) and adequately preserve the spatial information of panchromatic (PAN) image and the spectral information of MS image, this study proposes a method which adds the spectral information of the prior MS to the prior PAN during training, and only the posterior PAN is needed for predicting. Firstly, we introduce the autoencoder model based on image colorization and discuss its feasibility in the field of multi-band remote sensing image pan-sharpening. Then, the image quality evaluation functions including spatial and spectral indexes are formed as the loss function to control the image colorization model. Because the loss function contains spatial and spectral evaluation indexes, it could directly calculate the loss between the network output and the label considering characteristics of remote sensing images. Besides, the training data in our model is original PAN, this means that it is not necessary to make the simulated degraded MS and PAN data for training which is a big difference from most existing deep learning pan-sharpening methods. The new loss function including the spectral and spatial quality instead of the general MSE (mean square error), only the original PAN instead of the simulated degraded MS + PAN to be inputted, only the spectral feature instead of the direct fusion result to be learned, these three aspects change the current learning framework and optimization rule of deep learning pan-sharpening. Finally, thousands of remote sensing images from different scenes are adopted to make the training dataset to verify the effectiveness of the proposed method. In addition, we selected seven representative pan-sharpening algorithms and four widely recognized objective fusion metrics to evaluate and compare the performance on the WorldView-2 experimental data. The results show that the proposed method achieves optimal performance in terms of both the subjective visual effect and the object assessment.