Semantic feature learning on 3D point clouds is quite challenging because of their irregular and unordered data structure. In this paper, we propose a novel structure-aware convolution (SAC) to generalize deep learning on regular grids to irregular 3D point clouds. Similar to the template-matching process of convolution on 2D images, the key of our SAC is to match the point clouds’ neighborhoods with a series of 3D kernels, where each kernel can be regarded as a “geometric template” formed by a set of learnable 3D points. Thus, the interested geometric structures of the input point clouds can be activated by the corresponding kernels. To verify the effectiveness of the proposed SAC, we embedded it into three recently developed point cloud deep learning networks (PointNet, PointNet++, and KCNet) as a lightweight module, and evaluated its performance on both classification and segmentation tasks. Experimental results show that, benefiting from the geometric structure learning capability of our SAC, all these back-end networks achieved better classification and segmentation performance (e.g., +2.77% mean accuracy for classification and +4.99% mean intersection over union (IoU) for segmentation) with few additional parameters. Furthermore, results also demonstrate that the proposed SAC is helpful in improving the robustness of networks with the constraints of geometric structures.

A complete building model reconstruction needs data collected from both air and ground. The former often has sparse coverage on building façades, while the latter usually is unable to observe the building rooftops. Attempting to solve the missing data issues in building reconstruction from single data source, we describe an approach for complete building reconstruction that integrates airborne LiDAR data and ground smartphone imagery. First, by taking advantages of GPS and digital compass information embedded in the image metadata of smartphones, we are able to find airborne LiDAR point clouds for the corresponding buildings in the images. In the next step, Structure-from-Motion and dense multi-view stereo algorithms are applied to generate building point cloud from multiple ground images. The third step extracts building outlines respectively from the LiDAR point cloud and the ground image point cloud. An automated correspondence between these two sets of building outlines allows us to achieve a precise registration and combination of the two point clouds, which ultimately results in a complete and full resolution building model. The developed approach overcomes the problem of sparse points on building façades in airborne LiDAR and the deficiency of rooftops in ground images such that the merits of both datasets are utilized.

Recent open data initiatives allow free access to a vast amount of light detection and ranging (LiDAR) data in many cities. However, most open LiDAR data of cities are acquired by airborne scanning, where points on building façades are sparse or even completely missing due to occlusions in the urban environment, leading to the absence of façade details. This paper presents an approach for improving the LiDAR data coverage on building façades by using point cloud generated from ground images. A coarse-to-fine strategy is proposed to fuse these two-point clouds of different sources with very limited overlaps. First, the façade point cloud generated from ground images is leveled by adjusting the facade normal to perpendicular to the upright direction. Then leveling façade point cloud is geolocated by alignment between images GPS data and their structure from motion (SfM) coordinates. Next, a modified coherent point drift algorithm with (surface) normal consistency is proposed to accurately align the façade point cloud to the LiDAR data. The significance of this work resides in the use of 2D overlapping points on the building outlines instead of the limited 3D overlap between the two-point clouds. This way we can still achieve reliable and precise registration under incomplete coverage and ambiguous correspondence. Experiments show that the proposed approach can significantly improve the façade details in open LiDAR data, and achieve 2 to 10 times higher registration accuracy, when compared to classic registration methods.

Single-image plane segmentation plays an important role in understanding 3D indoor scenes, including applications such as 3D indoor reconstruction. In recent years, PlaneTR, a transformer-based architecture, has achieved remarkable performance in single-image plane instance segmentation. It has garnered significant attention from researchers and remains one of the most advanced algorithms in this field. However, PlaneTR has the following two major limitations: its ineffective utilization of line segment information within images and the high number of parameters. In this study, we propose an improved version of PlaneTR, named Spatial Prompt Learning PlaneTR (SPL-PlaneTR), to address these issues. Our approach effectively balances model complexity and performance. Specifically, to more effectively leverage structural information provided by line segments, we replace the original line segment’s transformer branch with a lightweight line segment prompt module and line segment prompt adapter. Additionally, we introduce spatial queries to replace conventional position queries, enabling the network to accurately localize planes across diverse indoor scenes. The experimental results demonstrate that our model, with fewer parameters, outperforms PlaneTR on both the original and noise-corrupted ScanNet datasets. Furthermore, SPL-PlaneTR achieves superior zero-shot transfer performance on the Matterport3D, ICL-NUIM RGB-D, and 2D-3D-S datasets compared to PlaneTR. Notably, our lightweight SPL-PlaneTR also surpasses several state-of-the-art algorithms in this domain. Our code and model have been publicly available.

Recent advances in open data initiatives allow us to free access to a vast amount of open LiDAR data in many cities. However, most of these open LiDAR data over cities are acquired by airborne scanning, where the points on façades are sparse or even completely missing due to the viewpoint and object occlusions in the urban environment. Integrating other sources of data, such as ground images, to complete the missing parts is an effective and practical solution. This paper presents an approach for improving open LiDAR data coverage on building façades by using point cloud generated from ground images. A coarse-to-fine strategy is proposed to fuse these two different sources of data. Firstly, the façade point cloud generated from terrestrial images is initially geolocated by matching the SFM camera positions to their GPS meta-information. Next, an improved Coherent Point Drift algorithm with normal consistency is proposed to accurately align building façades to open LiDAR data. The significance of the work resides in the use of 2D overlapping points on the outline of buildings instead of limited 3D overlap between the two point clouds and the achievement to a reliable and precise registration under possible incomplete coverage and ambiguous correspondence. Experiments show that the proposed approach can significantly improve the façades details of buildings in open LiDAR data and improving registration accuracy from up to 10 meters to less than half a meter compared to classic registration methods.