18 September 2023 in IEEE Transactions on Geoscience and Remote Sensing
Remote sensing spatiotemporal prediction aims to infer future trends from historical spatiotemporal data, e.g., videos and time series images, has a broad application prospect in many fields. The foundation model is a promising research direction for spatiotemporal information mining because of its robust feature extraction capability, and has made rapid progress in natural scenes. Nevertheless, due to the spatially multi-scale and temporally multi-scale properties in remote sensing data, these methods still encounter bottlenecks when applied to remote sensing. Therefore, we propose a foundation model for remote sensing spatiotemporal prediction via spatiotemporal evolution decoupling, abbreviated as RingMo-Sense. Considering spatial affinity, temporal continuity, and spatiotemporal interaction, we construct spatial, temporal, and spatiotemporal triple-branch prediction networks. Specifically, we use parameter-sharing and progressive joint training strategies to achieve stable long-range prediction and parameter reduction simultaneously. In addition, we build a remote sensing spatiotemporal dataset by collecting various remote sensing videos and time series images. The experimental results on six downstream spatiotemporal tasks demonstrate that the proposed model yields competitive performance.
1 September 2023 in IEEE Transactions on Geoscience and Remote Sensing
Datasets play a key role in developing superior building detection approaches. However, most of the previous work focuses on accurate building masks and scale expansion, while the categories are always missing, which hinders the further analysis of urban development and cultures. Therefore, we propose a benchmark for building detection and fine-grained classification from very high-resolution (VHR) satellite imagery. An extensive annotation is performed for about 0.5 million building instances with 12 fine-grained roof types and individual polygons. The annotation of building functions of two cities in the previous version (UBCv1) [1] is also integrated. To ensure the building variety, it consists of VHR optical images of 20 unique cities worldwide with various landforms and styles of architecture. Its variety and fine-grained categories pose great challenges and meanwhile provide a foundation for the building extraction and fine-grained classification on a global scale. Besides, 17 cities are provided with finely aligned Synthetic Aperture Radar (SAR) images, which can be employed for the development and evaluation of approaches optionally based on optical, SAR, or multi-modal images. Significantly, the proposed benchmark is used as the base of the 2023 IEEE GRSS Data Fusion Contest [2]. The dataset and codes of the baseline methods are available at: https://github.com/AICyberTeam/UBC-dataset/tree/UBCv2.
3 February 2023 in IEEE Transactions on Geoscience and Remote Sensing
Aircraft detection is a challenging task for remote sensing images. The anchor-based methods are of high complexity and the keypoint-based detectors suffer the grouping difficulty. And some line-based models relying on local features are hindered by the adhesion and dis-integrity problems. Moreover, those detection representations rarely take into account the sword-shaped component geometric semantics ( e.g ., fuselage and the wing) of the aircraft itself, leading to being less robust and unfavorable for downstream tasks, such as ones needing the detailed size and shape of aircraft. Accordingly, we model the sword-shaped component geometry and propose S2CGNet, a more robust appearance-based aircraft detector. The Sword Attenuation Mask (SAM) module is devised to encode a “sword-shaped mask” for each aircraft while exploring more robustness via the geometric surface embedding. The SAM can provide clearer borders to separate different aircraft more precisely. Besides, to address the instance disintegrity problem and further boost the quality of SAM, we propose an Instance Aware Graph (IAG) module to jointly optimize the parameters of the fuselage/wing detection heads. Experimental results show that the performance of S2CGNet can reach the state-of-the-art (SOTA) level. Specifically, it achieves 98.5% in term of AP 50 on the combined dataset of Aircraft-KP and NWPU VHR-10, boosting 3.8% than the baseline. Besides, S2CGNet boosts the quality of detection results greatly, e.g ., it yields a significant improvement of 21.3% on AP 75 compared to the baseline. Furthermore, the generalization comparisons on FAIR1M dataset strongly demonstrate the robustness of our model surpasses other oriented detectors by a large margin.
8 May 2024 in IEEE Transactions on Geoscience and Remote Sensing
As drone technology advances, using unmanned aerial vehicles for aerial surveys has become the dominant trend in modern low-altitude remote sensing. The surge in aerial video data necessitates accurate prediction for future scenarios and motion states of the interested target, particularly in applications like traffic management and disaster response. Existing video prediction methods focus solely on predicting future scenes (video frames), suffering from the neglect of explicitly modeling target’s motion states, which is crucial for aerial video interpretation. To address this issue, we introduce a novel task called Target-Aware Aerial Video Prediction, aiming to simultaneously predict future scenes and motion states of the target. Further, we design a model specifically for this task, named TAFormer, which provides a unified modeling approach for both video and target motion states. Specifically, we introduce Spatiotemporal Attention (STA), which decouples the learning of video dynamics into spatial static attention and temporal dynamic attention, effectively modeling the scene appearance and motion. Additionally, we design an Information Sharing Mechanism (ISM), which elegantly unifies the modeling of video and target motion by facilitating information interaction through two sets of messenger tokens. Moreover, to alleviate the difficulty of distinguishing targets in blurry predictions, we introduce Target-Sensitive Gaussian Loss (TSGL), enhancing the model’s sensitivity to both target’s position and content. Extensive experiments on UAV123VP and VisDroneVP (derived from single-object tracking datasets) demonstrate the exceptional performance of TAFormer in target-aware video prediction, showcasing its adaptability to the additional requirements of aerial video interpretation for target awareness.