In recent years, there have been many multimodal works in the field of remote sensing, and most of them have achieved good results in the task of land-cover classification. However, multi-scale information is seldom considered in the multi-modal fusion process. Secondly, the multimodal fusion task rarely considers the application of attention mechanism, resulting in a weak representation of the fused feature. In order to better use the multimodal data and reduce the losses caused by the fusion of different modalities, we proposed a TRMSF (Transformer and Multi-scale fusion) network for land-cover classification based on HSI (hyperspectral images) and LiDAR (Light Detection and Ranging) images joint classification. The network enhances multimodal information fusion ability by the method of attention mechanism from Transformer and enhancement using multi-scale information to fuse features from different modal structures. The network consists of three parts: multi-scale attention enhancement module (MSAE), multimodality fusion module (MMF) and multi-output module (MOM). MSAE enhances the ability of feature representation from extracting different multi-scale features of HSI, which are used to fuse with LiDAR feature, respectively. MMF integrates the data of different modalities through attention mechanism, thereby reducing the loss caused by the data fusion of different modal structures. MOM optimizes the network by controlling different outputs and enhances the stability of the results. The experimental results show that the proposed network is effective in multimodality joint classification.

The joint utilization of multi-source data is of great significance in geospatial observation applications, such as urban planning, disaster assessment, and military applications. However, this approach is confronted with challenges including inconsistent data structures, irrelevant physical properties, scarce training data, insufficient utilization of information and an imperfect feature fusion method. Therefore, this paper proposes a novel binary-tree Transformer network (BTRF-Net), which is used to fuse heterogeneous information and utilize complementarity among multi-source remote sensing data to enhance the joint classification performance of hyperspectral image (HSI) and light detection and ranging (LiDAR) data. Firstly, a hyperspectral network (HSI-Net) is employed to extract spectral and spatial features of hyperspectral images, while the elevation information of LiDAR data is extracted using the LiDAR network (LiDAR-Net). Secondly, a multi-source transformer complementor (MSTC) is designed that utilizes the complementarity and cooperation among multi-modal feature information in remote sensing images to better capture their correlation. The multi-head complementarity attention mechanism (MHCA) within this complementor can effectively capture global features and local texture information of images, hence achieving full feature fusion. Then, to fully obtain feature information of multi-source remote sensing images, this paper designs a complete binary tree structure, binary feature search tree (BFST), which fuses multi-modal features at different network levels to obtain multiple image features with stronger representation abilities, effectively enhancing the stability and robustness of the network. Finally, several groups of experiments are designed to compare and analyze the proposed BTRF-Net with traditional methods and several advanced deep learning networks using two datasets: Houston and Trento. The results show that the proposed network outperforms other state-of-the-art methods even with small training samples.