In this article, we introduce the task of visual grounding for remote sensing data (RSVG). RSVG aims to localize the referred objects in remote sensing (RS) images with the guidance of natural language. To retrieve rich information from RS imagery using natural language, many research tasks, such as RS image visual question answering, RS image captioning, and RS image–text retrieval, have been investigated a lot. However, the object-level visual grounding on RS images is still underexplored. Thus, in this work, we propose to construct the dataset and explore deep learning models for the RSVG task. Specifically, our contributions can be summarized as follows. First, we build the new large-scale benchmark of RSVG based on detection in optical remote sensing (DIOR) dataset, termed DIOR-RSVG, to fully advance the research of RSVG. This new dataset includes image/expression/box triplets for training and evaluating visual grounding models. Second, we benchmark extensive state-of-the-art (SOTA) natural image visual grounding methods on the constructed DIOR-RSVG dataset, and some insightful analyses are provided based on the results. Third, a novel transformer-based multigranularity visual language fusion (MGVLF) module is proposed. Remotely sensed images are usually with large-scale variations and cluttered backgrounds. To deal with the scale-variation problem, the MGVLF module takes advantage of multiscale visual features and multigranularity textual embeddings to learn more discriminative representations. To cope with the cluttered background problem, MGVLF adaptively filters irrelevant noise and enhances salient features. In this way, our proposed model can incorporate more effective multilevel and multimodal features to boost performance. This work can provide useful insights for developing better RSVG models.

The accurate and robust prediction of vessel traffic flow is gaining importance in maritime intelligent transportation system (ITS), such as vessel traffic services, maritime spatial planning, and traffic safety management, etc. To achieve fine-grained vessel traffic flow prediction, we will first generate the maritime traffic network (which is essentially a graph), and then propose a graph-driven neural network. In particular, to represent various correlations among spatio-temporal vessel traffic flow, we tend to extract the feature points (i.e., starting, way and ending points) by utilizing the knowledge of vessel positioning data. These feature points are essentially related to the geometrical structures of massive vessel trajectories collected from massive automatic identification system (AIS) records, contributing to the generation of maritime traffic network. We then propose a spatio-temporal multi-graph convolutional network (STMGCN)-based vessel traffic flow prediction method by exploiting multiple types of inherent correlations in the generated maritime graph. The proposed STMGCN mainly contains one spatial multi-graph convolutional layer and two temporal gated convolutional layers, beneficial for extracting spatial and temporal traffic flow patterns. The main benefit of our graph-driven prediction method is that it takes full advantage of the maritime graph and multi-graph learning. Comprehensive experiments have been implemented on realistic AIS dataset to compare our method with several state-of-the-art prediction methods. The fine-grained prediction results have demonstrated our superior performance in terms of both accuracy and robustness.

Vision-and-language pre-training (VLP) models have experienced a surge in popularity recently. By fine-tuning them on specific datasets, significant performance improvements have been observed in various tasks. However, full fine-tuning of VLP models not only consumes a significant amount of computational resources but also has a significant environmental impact. Moreover, as remote sensing (RS) data is constantly being updated, full fine-tuning may not be practical for real-world applications. To address this issue, in this work, we investigate the parameter-efficient transfer learning (PETL) method to effectively and efficiently transfer visual-language knowledge from the natural domain to the RS domain on the image-text retrieval task. To this end, we make the following contributions. 1) We construct a novel and sophisticated PETL framework for the RS image-text retrieval (RSITR) task, which includes the pretrained CLIP model, a multimodal remote sensing adapter, and a hybrid multi-modal contrastive (HMMC) learning objective; 2) To deal with the problem of high intra-modal similarity in RS data, we design a simple yet effective HMMC loss; 3) We provide comprehensive empirical studies for PETL-based RS image-text retrieval. Our results demonstrate that the proposed method is promising and of great potential for practical applications. 4) We benchmark extensive state-of-the-art PETL methods on the RSITR task. Our proposed model only contains 0.16M training parameters, which can achieve a parameter reduction of 98.9% compared to full fine-tuning, resulting in substantial savings in training costs. Our retrieval performance exceeds traditional methods by 7-13% and achieves comparable or better performance than full fine-tuning. This work can provide new ideas and useful insights for RS vision-language tasks.

We introduce a novel task of 3D visual grounding in monocular RGB images using language descriptions with both appearance and geometry information. Specifically, we build a large-scale dataset, Mono3DRefer, which contains 3D object targets with their corresponding geometric text descriptions, generated by ChatGPT and refined manually. To foster this task, we propose Mono3DVG-TR, an end-to-end transformer-based network, which takes advantage of both the appearance and geometry information in text embeddings for multi-modal learning and 3D object localization. Depth predictor is designed to explicitly learn geometry features. The dual text-guided adapter is proposed to refine multiscale visual and geometry features of the referred object. Based on depth-text-visual stacking attention, the decoder fuses object-level geometric cues and visual appearance into a learnable query. Comprehensive benchmarks and some insightful analyses are provided for Mono3DVG. Extensive comparisons and ablation studies show that our method significantly outperforms all baselines. The dataset and code will be released.