9 June 2022 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Extracting distinctive, robust, and general 3D local features is essential to downstream tasks such as point cloud registration. However, existing methods either rely on noise-sensitive handcrafted features, or depend on rotation-variant neural architectures. It remains challenging to learn robust and general local feature descriptors for surface matching. In this paper, we propose a new, simple yet effective neural network, termed SpinNet, to extract local surface descriptors which are rotation-invariant whilst sufficiently distinctive and general. A Spatial Point Transformer is first introduced to embed the input local surface into an elaborate cylindrical representation (SO(2) rotation-equivariant), further enabling end-to-end optimization of the entire framework. A Neural Feature Extractor, composed of point-based and 3D cylindrical convolutional layers, is then presented to learn representative and general geometric patterns. An invariant layer is finally used to generate rotation-invariant feature descriptors. Extensive experiments on both indoor and outdoor datasets demonstrate that SpinNet outperforms existing state-of-the-art techniques by a large margin. More critically, it has the best generalization ability across unseen scenarios with different sensor modalities.
1 April 2020 in Pattern Recognition
12 August 2024 in IEEE Geoscience and Remote Sensing Letters
Three-dimensional place recognition aims to search point cloud in a large database that matches the query. It is an essential task in remote sensing applications, such as smart city management and disaster monitoring. The existing methods commonly leverage global descriptors to perform point cloud retrieval for place recognition. However, these methods rely on spatial aggregation to obtain global descriptors, which are neither discriminative nor general. In this letter, we propose a unified global retrieval and local reranking (namely, GRLoR) framework for 3-D place recognition. Specifically, we first utilize a self-attention mechanism to capture the channel dependencies of local features and design a spatial-fusion pooling (SFP) approach to obtain a discriminative global descriptor for retrieval. We then construct a feature correlation module for local reranking, which uses a cross-attention mechanism to determine whether the point cloud pair matches correctly by predicting the similarity of local regions. Experiments conducted on several public benchmarks validate the superiority performance of our method. For instance, it outperforms the strongest model by an average of about 1% on the public datasets in terms of AR@1.
17 February 2025 in IEEE Transactions on Multimedia
Event-based video reconstruction has emerged as an appealing research direction to break through the limitations of traditional cameras to better record dynamic scenes. Most existing methods reconstruct each frame from its corresponding event subset in chronological order. Since the temporal information contained in the whole event sequence is not fully exploited, these methods suffer inferior reconstruction quality. In this paper, we propose to enhance event-based video reconstruction by leveraging the bidirectional temporal information in event sequences. The proposed model processes event sequences in a bidirectional fashion, allowing for exploiting bidirectional information in the whole sequence. Furthermore, a transformer-based temporal information fusion module is introduced to aggregate long-range information in both temporal and spatial dimensions. Additionally, we propose a new dataset for the event-based video reconstruction task which contains a variety of objects and movement patterns. Extensive experiments demonstrate that the proposed model outperforms existing state-of-the-art event-based video reconstruction methods both quantitatively and qualitatively.
26 March 2026 in IEEE Transactions on Image Processing
3D scene CAD recomposition aims to reconstruct a given scene by retrieving and assembling CAD models from a database, so as to accurately simulate the geometric properties and spatial arrangement of the original environment. Recent methods learn this task through training on limited scan-to-CAD annotation data, which hinders their generalization to diverse real-world scenes. In this paper, we propose POSITION, an open-world 3D scene CAD recomposition method to construct the 3D scene with CADs retrieved from an open-set database. POSITION is designed following a divide-and-conquer strategy. Firstly, we extract open-world multi-modal object representations from a captured 3D scene. Secondly, on top of the representations, we propose a coarse-to-fine retrieval method to retrieve CADs that are visually, geometrically and semantically match real objects. Thirdly, we present a physically plausible pose alignment method to adjust retrieved CAD models to maintain consistent geometry and layout with the observation. By decomposing the problem into well-defined subtasks, our approach achieves generalization across various scene types and scalable CAD databases without retraining or fine-tuning. Our approach demonstrates superior CAD recomposition performance on both the Scan2CAD and diverse real-world 3D scene datasets. Our project page: https://yangrongkun.github.io/position/.