The remarkable achievements of ChatGPT and Generative Pre-trained Transformer 4 (GPT-4) have sparked a wave of interest and research in the field of large language models (LLMs) for artificial general intelligence (AGI). These models provide intelligent solutions that are closer to human thinking, enabling us to use general artificial intelligence (AI) to solve problems in various applications. However, in the field of remote sensing (RS), the scientific literature on the implementation of AGI remains relatively scant. Existing AI-related research in RS focuses primarily on visual-understanding tasks while neglecting the semantic understanding of the objects and their relationships. This is where vision-LMs (VLMs) excel as they enable reasoning about images and their associated textual descriptions, allowing for a deeper understanding of the underlying semantics. VLMs can go beyond visual recognition of RS images and can model semantic relationships as well as generate natural language descriptions of the image. This makes them better suited for tasks that require both visual and textual understanding, such as image captioning and visual question answering (VQA). This article provides a comprehensive review of the research on VLMs in RS, summarizing the latest progress, highlighting current challenges, and identifying potential research opportunities. Specifically, we review the application of VLMs in mainstream RS tasks, including image captioning, text-based image generation, text-based image retrieval (TBIR), VQA, scene classification, semantic segmentation, and object detection. For each task, we analyze representative works and discuss research progress. Finally, we summarize the limitations of existing works and provide possible directions for future development. This review aims to provide a comprehensive overview of the current research progress of VLMs in RS (see Figure 1), and to inspire further research in this exciting and promising field.

The data‐driven approaches for medium‐range weather forecasting are recently shown to be extraordinarily promising for ensemble forecasting due to their fast inference speed compared to the traditional numerical weather prediction models. However, their forecast accuracy can hardly match the state‐of‐the‐art operational ECMWF Integrated Forecasting System (IFS) model. Previous data‐driven approaches perform ensemble forecasting using some simple perturbation methods, like the initial condition perturbation and the Monte Carlo dropout. However, their ensemble performance is often limited arguably by the sub‐optimal ways of applying perturbation. We propose a Swin Transformer‐based Variational Recurrent Neural Network (SwinVRNN), which is a stochastic weather forecasting model combining a SwinRNN predictor with a perturbation module. SwinRNN is designed as a Swin Transformer‐based recurrent neural network, which predicts the future states deterministically. Furthermore, to model the stochasticity in the prediction, we design a perturbation module following the Variational Auto‐Encoder paradigm to learn the multivariate Gaussian distributions of a time‐variant stochastic latent variable from the data. Ensemble forecasting can be easily performed by perturbing the model features leveraging the noise sampled from the learned distribution. We also compare four categories of perturbation methods for ensemble forecasting, that is, fixed distribution perturbation, learned distribution perturbation, MC dropout, and multi model ensemble. Comparisons on the WeatherBench data set show that the learned distribution perturbation method using our SwinVRNN model achieves remarkably improved forecasting accuracy and reasonable ensemble spread due to the joint optimization of the two targets. More notably, SwinVRNN surpasses operational IFS on the surface variables of the 2‐m temperature and the 6‐hourly total precipitation at all lead times up to 5 days (Code is available at https://github.com/tpys/wwprediction).

With the advancement of geo-systems and the increased availability of satellite data, a plethora of Land-Use and Land-Cover (LULC) products have been developed. The existing LULC products primarily relied on time-series imagery to classify land by pixel-based classifiers, allowing for local analysis and accurate boundary detection. However, the advent of deep learning has shifted towards the use of patch-based CNN models for generating land cover maps. In this paper, (1) we create a training dataset for China using a voting strategy based on three off-the-shelf available LULC products, avoiding the labor-intensive manual annotation. (2) We design a novel CNN-based model for LULC task, called Multi-modal Fine-grained Dual Network (dubbed as Dual-Net), which takes dual-date images to generate final maps, and reduces the need for gap-free temporal sequences or separate cloud detection. To leverage the correlation between location, date, and category, we embed multi-modal information (dates and geo-locations) to the model. Further, by incorporating low-level constraints and using pseudo-label refinement, we continually improve the performance and achieve more refined segmentation. (3) Due to the lack of a suitable validation dataset for China, we create a new validation dataset called China Sentinel2 Validation Dataset (CSVD) by manually annotating 733 finely labeled images of 1024 × 1024 pixels of China-specific Sentinel2 data. (4) Extensive experiments demonstrate that our model outperforms existing LULC products and produces more fine-grained segmentation results comparable to other patch-based products. Finally, we release annual LULC maps for China in 2020-2022 and also make our model accessible online for real-time results export.

Deep-learning-based radar echo extrapolation methods have achieved remarkable progress in the precipitation nowcasting field. However, they suffer from a common notorious problem—they tend to produce blurry predictions. Although some efforts have been made in recent years, the blurring problem is still under-addressed. In this work, we propose three effective strategies to assist deep-learning-based radar echo extrapolation methods to achieve more realistic and detailed prediction. Specifically, we propose a spatial generative adversarial network (GAN) and a spectrum GAN to improve image fidelity. The spatial and spectrum GANs aim at penalizing the distribution discrepancy between generated and real images from the spatial domain and spectral domain, respectively. In addition, a masked style loss is devised to further enhance the details by transferring the detailed texture of ground truth radar sequences to extrapolated ones. We apply a foreground mask to prevent the background noise from transferring to the outputs. Moreover, we also design a new metric termed the power spectral density score (PSDS) to quantify the perceptual quality from a frequency perspective. The PSDS metric can be applied as a complement to other visual evaluation metrics (e.g., LPIPS) to achieve a comprehensive measurement of image sharpness. We test our approaches with both ConvLSTM baseline and U-Net baseline, and comprehensive ablation experiments on the SEVIR dataset show that the proposed approaches are able to produce much more realistic radar images than baselines. Most notably, our methods can be readily applied to any deep-learning-based spatiotemporal forecasting models to acquire more detailed results.