Infrared and visible image fusion (IVIF) expects to obtain images that retain thermal radiation information from infrared images and texture details from visible images. In this paper, a model-based convolutional neural network (CNN) model, referred to as Algorithm Unrolling Image Fusion (AUIF), is proposed to overcome the shortcomings of traditional CNN-based IVIF models. The proposed AUIF model starts with the iterative formulas of two traditional optimization models, which are established to accomplish two-scale decomposition, i.e., separating low-frequency base information and high-frequency detail information from source images. Then the algorithm unrolling is implemented where each iteration is mapped to a CNN layer and each optimization model is transformed into a trainable neural network. Compared with the general network architectures, the proposed framework combines the model-based prior information and is designed more reasonably. After the unrolling operation, our model contains two decomposers (encoders) and an additional reconstructor (decoder). In the training phase, this network is trained to reconstruct the input image. While in the test phase, the base (or detail) decomposed feature maps of infrared/visible images are merged respectively by an extra fusion layer, and then the decoder outputs the fusion image. Qualitative and quantitative comparisons demonstrate the superiority of our model, which can robustly generate fusion images containing highlight targets and legible details, exceeding the state-of-the-art methods. Furthermore, our network has fewer weights and faster speed.

The emergence of generative models has revolutionized the field of remote sensing (RS) image generation. Despite generating high-quality images, existing methods are limited in relying mainly on text control conditions, and thus do not always generate images accurately and stably. In this paper, we propose CRS-Diff, a new RS generative framework specifically tailored for RS image generation, leveraging the inherent advantages of diffusion models while integrating more advanced control mechanisms. Specifically, CRS-Diff can simultaneously support text-condition, metadata-condition, and image-condition control inputs, thus enabling more precise control to refine the generation process. To effectively integrate multiple condition control information, we introduce a new conditional control mechanism to achieve multi-scale feature fusion, thus enhancing the guiding effect of control conditions. To our knowledge, CRS-Diff is the first multiple-condition controllable RS generative model. Experimental results in single-condition and multiple-condition cases have demonstrated the superior ability of our CRS-Diff to generate RS images both quantitatively and qualitatively compared with previous methods. Additionally, our CRS-Diff can serve as a data engine that generates high-quality training data for downstream tasks, e.g., road extraction. The code is available at https://github.com/Sonettoo/CRS-Diff.

The fusion of hyperspectral image (HSI) and multispectral image (MSI) is one of the most significant topics in remote sensing image processing. Recently, deep learning (DL) has emerged as an important tool for this task. However, existing DL-based methods have two drawbacks, that is, limited ability for feature extraction and suffering from spectral distortion. To address these issues, this article presents a novel neural network, where sophisticated techniques are employed, including network-in-network convolutional unit, batch normalization, and skip connection. To make full use of the MSI, the proposed model fuses HSI and MSI at different scales. Besides, this article presents a new loss function, called RMSE, angle and Laplacian (RAP) loss (the combination of the relative mean squared error, angle loss, and Laplacian loss), to deal with both spatial and spectral distortions. Experiments conducted on four data sets have verified the rationality of network structure and the proposed loss function and demonstrated that the proposed novel model outperforms state-of-the-art counterparts.

Object detection is a fundamental and important task in the analysis of remote sensing images (RSIs), and existing deep learning-based object detection models in this literature strongly rely on predefined anchor boxes and encounter redesigned difficulties related to anchors. In addition, they often ignore the scene-contextual information that objects are usually closely related to their surrounding scene. To deal with these problems, we propose an anchor-free network, referred to as scene-relevant anchor-free network (SRAF-Net), for object detection in RSIs. The SRAF-Net first captures the scene-contextual features of objects by using a designed scene-enhanced feature pyramid network (SE-FPN) and then performs more accurate detection by implementing a scene auxiliary detection head (SADH), which can predict the existence of the objects with the help of the scene-contextual features extracted from the SE-FPN. To deal with insufficient scene diversity in the training stage, a simple yet effective data augmentation module, termed balanced mixup data augment (BMDA), is introduced by linearly expanding the training dataset to improve the generalization of SRAF-Net. Comprehensive experiments on three publicly available challenging remote sensing datasets demonstrate the effectiveness of the proposed method. The codes will be made publicly available at https://github.com/Complicateddd/SRAF-Net .

3D object recognition is one of the most important tasks in 3D data processing, and has been extensively studied recently. Researchers have proposed various 3D recognition methods based on deep learning, among which a class of view-based approaches is a typical one. However, in the view-based methods, the commonly used view pooling layer to fuse multi-view features causes a loss of visual information. To alleviate this problem, in this paper, we construct a novel layer called Dynamic Routing Layer (DRL) by modifying the dynamic routing algorithm of capsule network, to more effectively fuse the features of each view. Concretely, in DRL, we use rearrangement and affine transformation to convert features, then leverage the modified dynamic routing algorithm to adaptively choose the converted features, instead of ignoring all but the most active feature in view pooling layer. We also illustrate that the view pooling layer is a special case of our DRL. In addition, based on DRL, we further present a Dynamic Routing Convolutional Neural Network (DRCNN) for multi-view 3D object recognition. Our experiments on three 3D benchmark datasets show that our proposed DRCNN outperforms many state-of-the-arts, which demonstrates the efficacy of our method.

Remote sensing image change captioning (RSICC) aims to automatically generate sentences that describe content differences in remote sensing bitemporal images. Recently, attention-based transformers have become a prevalent idea for capturing the features of global change. However, existing transformer-based RSICC methods face challenges, e.g., high parameters and high computational complexity caused by the self-attention operation in the transformer encoder component. To alleviate these issues, this paper proposes a Sparse Focus Transformer (SFT) for the RSICC task. Specifically, the SFT network consists of three main components, i.e. a high-level features extractor based on a convolutional neural network (CNN), a sparse focus attention mechanism-based transformer encoder network designed to locate and capture changing regions in dual-temporal images, and a description decoder that embeds images and words to generate sentences for captioning differences. The proposed SFT network can reduce the parameter number and computational complexity by incorporating a sparse attention mechanism within the transformer encoder network. Experimental results on various datasets demonstrate that even with a reduction of over 90% in parameters and computational complexity for the transformer encoder, our proposed network can still obtain competitive performance compared to other state-of-the-art RSICC methods. The code is available at Lite_Chag2cap.