This research investigates vision‐based automated bridge component recognition, which is critical for automating visual inspection of bridges during initial response after earthquakes. Semantic segmentation algorithms with up to 45 convolutional layers are applied to recognize bridge components from images of complex scenes. One of the challenges in such scenarios is to get the recognition results consistent with high‐level scene structure using limited amount of training data. To impose the high‐level scene consistency, this research combines 10‐class scene classification and 5‐class bridge component classification. Three approaches are investigated to combine scene classification results into bridge component classification: (a) naïve configuration, (b) parallel configuration, and (c) sequential configuration of classifiers. The proposed approaches, sequential configuration in particular, are demonstrated to be effective in recognizing bridge components in complex scenes, showing less than 1% of accuracy loss from the naïve/parallel configuration for bridge images, and less than 1% false positives for the nonbridge images.

Manual visual inspection of civil infrastructure is high-risk, subjective, and time-consuming. The success of deep learning and the proliferation of low-cost consumer robots has spurred rapid growth in research and application of autonomous inspections. The major components of autonomous inspection include data acquisition, data processing, and decision making, which are usually studied independently. However, for robust real-world applicability, these three aspects of the overall process need to be addressed concurrently with end-to-end testing, incorporating scenarios such as variations in structure type, color, damage level, camera distance, view angle, lighting, etc. Developing real-world datasets that span all these scenarios is nearly impossible. In this paper, we propose a framework to create a virtual visual inspection testbed using 3D synthetic environments that can enable end-to-end testing of autonomous inspection strategies. To populate the 3D synthetic environment with virtual damaged buildings, we propose the use of a non-linear finite element model to inform the realistic and automated visual rendering of different damage types, the damage state, and the material textures of what are termed herein physics-based graphics models (PBGMs). To demonstrate the benefits of the autonomous inspection testbed, three experiments are conducted with models of earthquake damaged reinforced concrete buildings. First, we implement the proposed framework to generate a new large-scale annotated benchmark dataset for post-earthquake inspections of buildings termed QuakeCity. Second, we demonstrate the improved performance of deep learning models trained using the QuakeCity dataset for inference on real data. Finally, a comparison of deep learning-based damage state estimation for different data acquisition strategies is carried out. The results demonstrate the use of PBGMs as an effective testbed for the development and validation of strategies for autonomous vision-based inspections of civil infrastructure.

This research investigates a framework for the efficient development of vision-based dense three-dimensional displacement measurement algorithms to support reliable structural health monitoring of civil structures. The framework exploits the use of a photo-realistic synthetic model, termed a physics-based graphics model, to simulate the entire process of vision-based measurement. At the same time, the synthetic environment is used to evaluate the performance of different post-processing algorithms quantitatively for a given measurement scenario, such as camera selection and camera placement. The effectiveness of the framework is demonstrated by optimizing the algorithms for the three-dimensional displacement measurement of a 14-bay laboratory truss structure. The vision-based dense three-dimensional displacement estimation algorithms optimized in this study consist of four steps: (1) camera parameter estimation, (2) camera motion estimation and compensation, (3) vision-based two-dimensional tracking, and (4) projection of two-dimensional tracking results to three-dimensional space. The algorithms use the knowledge from the finite element model to facilitate the implementation and maximize the measurement outcome, that is, model-informed approach. To test and evaluate the model-informed approach, synthetic videos are rendered for two measurement scenarios, that is, using a Digital Single Lens Reflex camera mounted on a tripod and using an Unmanned Aerial Vehicle camera. Then, the performance of the model-informed approach is evaluated by comparing the estimated displacement with the ground truth values. Based on the performance evaluation, an algorithm with the highest expected performance is selected for each of the two measurement scenarios. Finally, the selected algorithm is tested in a laboratory experiment. In contrast to the existing literature that investigates fixed individual measurement scenarios, the proposed framework can be used to test different measurement scenarios and estimate the outcome of each scenario before performing actual tests, facilitating the implementation of vision-based measurement for the structural health monitoring of civil structures.

Research has been continually growing toward the development of image-based structural health monitoring tools that can leverage deep learning models to automate damage detection in civil infrastructure. However, these tools are typically based on RGB images, which work well under ideal lighting conditions, but often have degrading performance in poor and low-light scenes. On the other hand, thermal images, while lacking in crispness of details, do not show the same degradation of performance in changing lighting conditions. The potential to enhance automated damage detection by fusing RGB and thermal images together within a deep learning network has yet to be explored. In this paper, RGB and thermal images are fused in a ResNET-based semantic segmentation model for vision-based inspections. A convolutional neural network is then employed to automatically identify damage defects in concrete. The model uses a thermal and RGB encoder to combine the features detected from both spectrums to improve its performance of the model, and a single decoder to predict the classes. The results suggest that this RGB-thermal fusion network outperforms the RGB-only network in the detection of cracks using the Intersection Over Union (IOU) performance metric. The RGB-thermal fusion model not only detected damage at a higher performance rate, but it also performed much better in differentiating the types of damage.