Remote sensing image analysis has drawn more attentions in the field of computer vision. At present, the methods commonly used in remote sensing image analysis are mainly at a low level semantically. The scene graph is an abstraction of objects and their relationships, which is a high-level image understanding task. To fully comprehend the meanings of remote sensing images, in this article, we propose a novel segmentation-based model to generate remote sensing image scene graphs (SRSG). In the SRSG model, a more complete and accurate scene graph is generated with the segmentation results as inputs, while the shapes of objects are reasonably coded. The morphological features of object pairs are embedded together by different branches of the SRSG model and then mapped to semantic space to predicate their relationships. Furthermore, a new dataset for scene graph generation of remote sensing images, namely, segmentation results to scene graphs (S2SG), is constructed based on pixel-level segmentation results. Experimental results demonstrate that the performance of the SRSG model is far superior to the previous methods in the task of generating remote sensing image scene graphs. The proposed SRSG model opens up new possibilities for remote sensing image analysis at a high level. Moreover, the S2SG dataset further allows for the evaluation of different approaches and is provided for the benefit of the research community.

To semantically understand remote sensing images, it is not only necessary to detect the objects in them but also to recognize the semantic relationships between the instances. Scene graph generation aims to represent the image as a semantic structural graph, where objects and relationships between them are described as nodes and edges, respectively. Some existing methods rely only on visual features to sequentially predict the relationships between objects, ignoring contextual information and making it difficult to generate high-quality scene graphs, especially for remote sensing images. Therefore, we propose a novel model for remote sensing image scene graph generation by fusing contextual information and statistical knowledge, namely RSSGG_CS. To integrate contextual information and calculate attention among all objects, the RSSGG_CS model adopts a filter module (FiM) that is based on adjusted transformer architecture. Moreover, to reduce the blindness of the model when searching semantic space, statistical knowledge of relational predicates between objects from the training dataset and the cleaned Wikipedia text is used as supervision when training the model. Experiments show that fusing contextual information and statistical knowledge allows the model to generate more complete scene graphs of remote sensing images and facilitates the semantic understanding of remote sensing images.

Active object recognition (AOR) aims at collecting additional information to improve recognition performance by purposefully adjusting the viewpoint of an agent. How to determine the next best viewpoint of the agent, i.e., viewpoint planning (VP), is a research focus. Most existing VP methods perform viewpoint exploration in the discrete viewpoint space, which have to sample viewpoint space and may bring in significant quantization error. To address this challenge, a continuous VP approach for AOR based on reinforcement learning is proposed. Specifically, we use two separate neural networks to model the VP policy as a parameterized Gaussian distribution and resort the proximal policy optimization framework to learn the policy. Furthermore, an adaptive entropy regularization based dynamic exploration scheme is presented to automatically adjust the viewpoint exploration ability in the learning process. To the end, experimental results on the public dataset GERMS well demonstrate the superiority of our proposed VP method.

Accurate feature matching plays important role in subsequent robotic vision tasks. Random sample consensus (RANSAC) is a commonly used estimator for removing the mismatches from the matching results based on 2D or 3D features. The classic RANSAC tries to search for the optimal matches with the maximum number of consistent point pairs. However, this way makes it easy to fall into a local maximum when it is used for fitting the inlier matches from 3D LiDAR point clouds containing lots of similar local features. To solve this problem in the 3D matching of LiDAR point cloud, we propose an optimized RANSAC. The optimized RANSAC estimates the best matches by maximizing the global dispersion under the condition of the truncated number of consistent point pairs. In experiments, we evaluate its performance based on the matching results of LinK3D features on KITTI dataset. The evaluation results show that our method can effectively increase the success rate when LinK3D is executed in challenging scenes (e.g., high-way and forest scenes) and improve its robustness to these scenes.

In recent years, the continuous advancement of deep learning has led to significant progress in object-detection technology for remote-sensing images. However, most current detection methods passively perform detection on the input image without considering the relationship between imaging configurations and detection-algorithm performance. Therefore, when factors such as poor lighting conditions, extreme shooting angles, or long acquisition distances degrade image quality, the passive detection framework limits the effectiveness of the current detection algorithm, preventing it from completing the detection task. To address the limitations above, this paper proposes an active object-detection (AOD) method based on deep reinforcement learning, taking adaptive brightness and collection position adjustments as examples. Specifically, we first established an end-to-end network structure to generate attribute control policies. Then, we designed a reward function suitable for remote-sensing images based on the degree of improvement in detection performance. Finally, we propose a new viewpoint-management method in this paper, which is successfully implemented by a training method of long-term Prioritized Experience Replay (LPER), which significantly reduces the accumulation of negative and repetitive samples and improves the success rate of the AOD algorithm for remote-sensing images. The experiments on two public datasets have fully demonstrated the effectiveness and advantages of the algorithm proposed in this paper.