Automated damage evaluation is of great importance in the maintenance and preservation of heritage structures. Damage investigation of large cultural buildings is time-consuming and labor-intensive, meaning that many buildings are not repaired in a timely manner. Additionally, some buildings in harsh environments are impossible to reach, increasing the difficulty of damage investigation. Oblique images facilitate damage detection in large buildings, yet quantitative damage information, such as area or volume, is difficult to generate. In this paper, we propose a method for quantitative damage evaluation of large heritage buildings in wild areas with repetitive structures based on drone images. Unlike existing methods that focus on building surfaces, we study the damage of building components and extract hidden linear symmetry information, which is useful for localizing missing parts in architectural restoration. First, we reconstruct a 3D mesh model based on the photogrammetric method using high-resolution oblique images captured by drone. Second, we extract 3D objects by applying advanced deep learning methods to the images and projecting the 2D object segmentation results to 3D mesh models. For accurate 2D object extraction, we propose an edge-enhanced method to improve the segmentation accuracy of object edges. 3D object fragments from multiple views are integrated to build complete individual objects according to the geometric features. Third, the damage condition of objects is estimated in 3D space by calculating the volume reduction. To obtain the damage condition of an entire building, we define the damage degree in three levels: no or slight damage, moderate damage and severe damage, and then collect statistics on the number of damaged objects at each level. Finally, through an analysis of the building structure, we extract the linear symmetry surface from the remaining damaged objects and use the symmetry surface to localize the positions of missing objects. This procedure was tested and validated in a case study (the Jiankou Great Wall in China). The experimental results show that in terms of segmentation accuracy, our method obtains results of 93.23% mAP and 84.21% mIoU on oblique images and 72.45% mIoU on the 3D mesh model. Moreover, the proposed method shows effectiveness in performing damage assessment of objects and missing part localization.

With the expansion of model scale and the improvement of model accuracy, the real-time rendering and displaying of 3D mesh models remain infeasible. To relieve such pressure, mesh simplification methods have been proposed to reduce the structural complexity while preserving the appearance. This work introduces a shape-preserving simplification method for urban building models. Compared to traditional simplification methods that only consider preserving local geometric features, we also consider the overall shapes of building models to avoid collapse with an increased simplification rate. The proposed method works in four steps. First, we filter mesh models to yield planar structures while preserving sharp features. Second, we detect the shapes of planar regions. Third, we simplify the planar and non-planar regions constrained by the overall shapes and local geometric features. Finally, we remap the texture. Experiments show the effectiveness of this method by evaluating it on various building models and comparing its performance with the original QEM algorithms. Furthermore, we maintain better spatial consistency in building models with different levels of detail (LOD) than traditional methods.

With the progress of photogrammetry and computer vision technology, three-dimensional (3D) reconstruction using aerial oblique images has been widely applied in urban modelling and smart city applications. However, state-of-the-art image-based automatic 3D reconstruction methods cannot effectively handle the unavoidable geometric deformation and incorrect texture mapping problems caused by moving cars in a city. This paper proposes a method to address this situation and prevent the influence of moving cars on 3D modelling by recognizing moving cars and combining the recognition results with a photogrammetric 3D modelling procedure. Through car detection using a deep learning method and multiview geometry constraints, we can analyse the state of a car’s movement and apply a proper preprocessing method to the geometrically model generation and texture mapping steps of 3D reconstruction pipelines. First, we apply the traditional Mask R-CNN object detection method to detect cars from oblique images. Then, a detected car and its corresponding image patch calculated by the geometry constraints in the other view images are used to identify the moving state of the car. Finally, the geometry and texture information corresponding to the moving car will be processed according to its moving state. Experiments on three different urban datasets demonstrate that the proposed method is effective in recognizing and removing moving cars and can repair the geometric deformation and error texture mapping problems caused by moving cars. In addition, the methods proposed in this paper can be applied to eliminate other moving objects in 3D modelling applications.

Although great success has been achieved in instance segmentation, accurate segmentation of instances remains difficult, especially at object edges. This problem is more prominent for instance segmentation in remote sensing imagery due to the diverse scales, variable illumination, smaller objects, and complex backgrounds. We find that most current instance segmentation networks do not consider the segmentation difficulty of different instances and different regions within the instance. In this paper, we study this problem and propose an ensemble method to segment instances from remote sensing images, considering the enhancement of hard-to-segment instances and instance edges. First, we apply a pixel-level Dice metric that reliably describes the segmentation quality of each instance to achieve online hard instance learning. Instances with low Dice values are studied with emphasis. Second, we generate a penalty map based on the analysis of boundary shapes to not only enhance the edges of objects but also discriminatively strengthen the edges of different shapes. That is, different areas of an object, such as internal areas, flat edges, and sharp edges, are distinguished and discriminatively weighed. Finally, the hard-to-segment instance learning and the shape-penalty map are integrated for precise instance segmentation. To evaluate the effectiveness and generalization ability of the proposed method, we train with the classic instance segmentation network Mask R-CNN and conduct experiments on two different types of remote sensing datasets: the iSAID-Reduce100 and the JKGW_WHU datasets, which have extremely different feature distributions and spatial resolutions. The comprehensive experimental results show that the proposed method improved the segmentation results by 2.78% and 1.77% in mask AP on the iSAID-Reduce100 and JKGW_WHU datasets, respectively. We also test other state-of-the-art (SOTA) methods that focus on inaccurate edges. Experiments demonstrate that our method outperforms these methods.

Oblique photogrammetry-based three-dimensional (3D) urban models are widely used for smart cities. In 3D urban models, road signs are small but provide valuable information for navigation. However, due to the problems of sliced shape features, blurred texture and high incline angles, road signs cannot be fully reconstructed in oblique photogrammetry, even with state-of-the-art algorithms. The poor reconstruction of road signs commonly leads to less informative guidance and unsatisfactory visual appearance. In this paper, we present a pipeline for embedding road sign models based on deep convolutional neural networks (CNNs). First, we present an end-to-end balanced-learning framework for small object detection that takes advantage of the region-based CNN and a data synthesis strategy. Second, under the geometric constraints placed by the bounding boxes, we use the scale-invariant feature transform (SIFT) to extract the corresponding points on the road signs. Third, we obtain the coarse location of a single road sign by triangulating the corresponding points and refine the location via outlier removal. Least-squares fitting is then applied to the refined point cloud to fit a plane for orientation prediction. Finally, we replace the road signs with computer-aided design models in the 3D urban scene with the predicted location and orientation. The experimental results show that the proposed method achieves a high mAP in road sign detection and produces visually plausible embedded results, which demonstrates its effectiveness for road sign modeling in oblique photogrammetry-based 3D scene reconstruction.