Recently, the burgeoning demands for captioning-related applications have inspired great endeavors in the remote sensing community. However, current benchmark datasets are deficient in data volume, category variety, and description richness, which hinders the advancement of new remote sensing image captioning approaches, especially those based on deep learning. To overcome this limitation, we present a larger and more challenging benchmark dataset termed NWPU-Captions is available at https://github.com/HaiyanHuang98/NWPU-Captions . NWPU-Captions contains 157 500 sentences, with all 31 500 images annotated manually by seven experienced volunteers. The superiority of NWPU-Captions over current publicly available benchmark datasets not only lies in its much larger scale but also in its wider coverage of complex scenes and the richness and variety of describing vocabularies. Furthermore, a novel encoder–decoder architecture, multilevel and contextual attention network (MLCA-Net), is proposed. MLCA-Net employs a multilevel attention module to adaptively aggregate image features of specific spatial regions and scales and introduces a contextual attention module to explore the latent context hidden in remote sensing images. MLCA-Net improves the flexibility and diversity of the generated captions while keeping their accuracy and conciseness by exploring the properties of scale variations and semantic ambiguity. Finally, the effectiveness, robustness, and generalization of MLCA-Net are proved through extensive experiments on existing datasets and NWPU-Captions.

Estimation of forest aboveground biomass (AGB) is crucial for various technical and scientific applications, ranging from regional carbon and bioenergy policies to sustainable forest management. However, passive optical remote sensing, which is the most widely used remote sensing data for retrieving vegetation parameters, is constrained by spectral saturation problems and cloud cover. On the other hand, LiDAR data, which have been extensively used to estimate forest structure attributes, cannot provide sufficient spectral information of vegetation canopies. Thus, this study aimed to develop a novel synergistic approach to estimating biomass by integrating LiDAR data with Landsat 8 imagery through a deep learning-based workflow. First the relationships between biomass and spectral vegetation indices (SVIs) and LiDAR metrics were separately investigated. Next, two groups of combined optical and LiDAR indices (i.e., COLI1 and COLI2) were designed and explored to identify their performances in biomass estimation. Finally, five prediction models, including K-nearest Neighbor, Random Forest, Support Vector Regression, the deep learning model, i.e., Stacked Sparse Autoencoder network (SSAE), and multiple stepwise linear regressions, were individually used to estimate biomass with input variables of different scenarios, i.e., (i) all the COLI1 (ACOLI1), (ii) all the COLI2 (ACOLI2), (iii) ACOLI1 and all the optical (AO) and LiDAR variables (AL), and (iv) ACOLI2, AO and AL. Results showed that univariate models with the combined optical and LiDAR indices as explanatory variables presented better modeling performance than those with either optical or LiDAR data alone, regardless of the combination mode. The SSAE model obtained the best performance compared to the other tested prediction algorithms for the forest biomass estimation. The best predictive accuracy was achieved by the SSAE model with inputs of combined optical and LiDAR variables (i.e., ACOLI1, AO and AL) that yielded an R² of 0.935, root mean squared error (RMSE) of 15.67 Mg/ha, and relative root mean squared error (RMSE_r) of 11.407%. It was concluded that the presented combined indices were simple and effective by integrating LiDAR-derived structure information with Landsat 8 spectral data for estimating forest biomass. Overall, the SSAE model with inputs of Landsat 8 and LiDAR integrated information resulted in accurate estimation of forest biomass. The presented modeling workflow will greatly facilitate future forest biomass estimation and carbon stock assessments.

Because of the rapid growth of multimodal data from the internet and social media, a cross-modal retrieval has become an important and valuable task in recent years.The purpose of the cross-modal retrieval is to obtain the result data in one modality (e.g., image), which is semantically similar to the query data in another modality (e.g., text).In the field of remote sensing, despite a great number of existing works on image retrieval, there has only been a small amount of research on the cross-modal image-text retrieval, due to the scarcity of datasets and the complicated characteristics of remote sensing image data. In this article, we introduce a novel cross-modal image-text retrieval network to establish the direct relationship between remote sensing images and their paired text data. Specifically, in our framework, we designed a semantic alignment module to fully explore the latent correspondence between images and text, in which we used the attention and gate mechanisms to filter and optimize data features so that more discriminative feature representations can be obtained. Experimental results on four benchmark remote sensing datasets, including UCMerced-LandUse-Captions, Sydney-Captions, RSICD, and NWPU-RESISC45-Captions, well showed that our proposed method outperformed other baselines and achieved the state-of-the-art performance in remote sensing image-text retrieval tasks.

Along with rapid urbanization, nighttime activities from places, such as restaurants, pubs and bars, and theatres, have created enormous economic and social benefits. The nighttime economy (NTE), as a newly developed social phenomenon, has been used to describe economic activities at night. However, few studies have investigated urban nighttime economy and its relation to urbanization from nighttime light (NTL) data perspective. To fill this gap, this study proposed a nighttime light economy index (NLEI). The correlation analysis was performed between the NLEI and economic indicators at both the city and provincial levels in China from 1992 to 2012 using the DMSP/OLS (Defense Meteorological Satellite Program/Operational Linescan System) time series data. Results revealed that correlations between the NLEI and all kinds of economic indicators were statistically significant. It was observed that both the urbanization and nighttime economy levels increased greatly from 1992 to 2012 in China. Cities and provinces in east China displayed relatively higher annual growth rates of NLEI compared to those in southwest and northwest China. Based on the quadrant map of urbanization and nighttime economy levels, most of the provincial capitals and provinces in east China were in the advanced coordination pattern while those in west China in the low-level coordination pattern.

Brain tumor segmentation is a fundamental task and existing approaches usually rely on multi-modality magnetic resonance imaging (MRI) images for accurate segmentation. However, the common problem of missing/incomplete modalities in clinical practice would severely degrade their segmentation performance, and existing fusion strategies for incomplete multi-modality brain tumor segmentation are far from ideal. In this work, we propose a novel framework named M $^{2}$ FTrans to explore and fuse cross-modality features through modality-masked fusion transformers under various incomplete multi-modality settings. Considering vanilla self-attention is sensitive to missing tokens/inputs, both learnable fusion tokens and masked self-attention are introduced to stably build long-range dependency across modalities while being more flexible to learn from incomplete modalities. In addition, to avoid being biased toward certain dominant modalities, modality-specific features are further re-weighted through spatial weight attention and channel- wise fusion transformers for feature redundancy reduction and modality re-balancing. In this way, the fusion strategy in M $^{2}$ FTrans is more robust to missing modalities. Experimental results on the widely-used BraTS2018, BraTS2020, and BraTS2021 datasets demonstrate the effectiveness of M $^{2}$ FTrans, outperforming the state-of-the-art approaches with large margins under various incomplete modalities for brain tumor segmentation. Code is available at https://github.com/Jun-Jie-Shi/M2FTrans .

Accurate land cover mapping is important for urban planning and management. Remote sensing data have been widely applied for urban land cover mapping. However, obtaining land cover classification via optical remote sensing data alone is difficult due to spectral confusion. To reduce the confusion between dark impervious surface and water, the Sentinel-1A Synthetic Aperture Rader (SAR) data are synergistically combined with the Sentinel-2B Multispectral Instrument (MSI) data. The novel support vector machine with composite kernels (SVM-CK) approach, which can exploit the spatial information, is proposed to process the combination of Sentinel-2B MSI and Sentinel-1A SAR data. The classification based on the fusion of Sentinel-2B and Sentinel-1A data yields an overall accuracy (OA) of 92.12% with a kappa coefficient (KA) of 0.89, superior to the classification results using Sentinel-2B MSI imagery and Sentinel-1A SAR imagery separately. The results indicate that the inclusion of Sentinel-1A SAR data to Sentinel-2B MSI data can improve the classification performance by reducing the confusion between built-up area and water. This study shows that the land cover classification can be improved by fusing Sentinel-2B and Sentinel-1A imagery.