1 January 2021 in IEEE Transactions on Instrumentation and Measurement
This article studies the problem of aero-engine blade surface defect detection in large images. The effective method for aero-engine blade surface inspection for this real application is currently lacking since most defects are relatively small. Therefore, the task of aero-engine blade surface defect detection is mainly implemented by experienced operators, which is subjective and time-consuming. Moreover, it is hard to fit the requirements of higher accuracy and efficiency manually. Therefore, an effective and efficient method for aero-engine blade surface defect detection is demanded. To achieve this, we propose a vision-based framework in this study to detect defects in a coarse-to-fine manner. First, the captured raw images are with a high resolution of 2448×2048 to ensure the accuracy of defect detection. The raw images are then cropped into smaller regions and fed into our deep convolution neural network (DCNN) to learn features with high representation. Next, the coarse classifier module is proposed to filter most background regions out. Finally, the defects are located and classified by a fine detector module in the defective images, which are selected by the coarse classifier module. Instead of directly applying a detector, our coarse-to-fine framework can effectively save computation and improve accuracy. In addition, the coarse-to-fine framework can be trained in an end-to-end manner. Compared with classical methods for object detection, our method also achieves state-of-the-art performance for aero-engine blade surface defect detection in terms of accuracy and efficiency. Furthermore, our framework has been applied for practical application in many aero-engine blade production lines.
16 August 2022 in IEEE Transactions on Instrumentation and Measurement
In the field of aircraft maintenance, regular inspection of fuselage surface during the aircraft life cycle is a vital task to ensure the aircraft quality and flight safety. Currently, the inspection task is generally carried out manually in an indoor hangar, which is with low efficiency and reliability. In this article, a novel system based on the unmanned aerial vehicle (UAV) is presented to achieve automated aircraft surface inspection efficiently. The hardware is established with a lightweight and low-cost flight platform, on which a sensor containing an inertial measurement unit (IMU) and a camera is equipped for UAV localization. A high-resolution camera is equipped to collect images of fuselage for defect detection. Our inspection framework is mainly composed of two modules: the UAV localization module and the defect detection module. The localization module is designed to estimate the relative pose between the UAV and the aircraft, providing the foundation for image positioning on the aircraft surface. The existing visual–inertial odometry (VIO) approach is adopted to implement the pose estimation. To reduce the large drifts caused by the VIO approach, a novel method is proposed to deploy precalibrated ArUco markers around the aircraft, which serve as external constraints for the VIO objective to realize joint optimization of the camera pose. In addition, an adaptive weighting method is proposed, which takes into consideration the recognition effect of markers to balance the external constraints. The defect detection module aims to detect defects on the fuselage surface from images captured by the high-resolution camera, which is implemented based on deep learning. To address the issue of detection on a few training samples, the transfer learning strategy is exploited to first pretrain the model on a public defect dataset and then fine-tune it on our collected aircraft defect dataset. After detecting the defects, the defective region is reflected on the fuselage surface through the UAV pose on the corresponding frame provided by the localization module, realizing the accurate defect localization. Experiments on both the simulation environment and real data demonstrate the superiority of our proposed external localization module and the effectiveness of the crack detection module.
21 December 2022 in IEEE Transactions on Intelligent Transportation Systems
Existing learning-based algorithms have a certain potential in visual odometry. In this work, we propose the solution of the learning-based method, which contains the attention mechanism and pose graph optimization. We set a self-supervised network as our backbone to cope with image data and error-heavy estimation pose for pose correction The pre-processing camera poses involved in the network can provide prior information. Combining the advantages of the abundant feature information and efficient attention mechanism, we design a geometric attention module that is sensitive to geometrical structure from images to accurately regress the rotation matrix. Then we improve the loss function with the weights of the attention module to consider the diversity of the data. Experimental results demonstrate the effectiveness and reliability of our approach on the public datasets KITTI with monocular task and stereo task. In comparison, the proposed method is superior to the existing methods in the translation component. In the self-supervised network, learning an attention mechanism can extract an effective connect relation of feature maps. We conduct ablation experiments under the self-supervised network backbone setting different strategies, and conclude that the proposed attention module is applicable to various sequences, and provide loss function improvements on the visual odometry task.
23 June 2021 in IEEE Transactions on Instrumentation and Measurement
Accurate localization of gangue plays an essential role in the vision-based gangue sorting system. However, the gangue commonly has indistinguishable appearances and is stacked with coals in a cluttered mining environment, posing challenges for efficient gangue identification and volume measurement from images. In this article, we propose a novel inspection framework for fully automated gangue localization and volume measurement based on deep learning and point cloud processing techniques. To efficiently perform gangue inspection, first, a 3-D scanning system is assembled to automatically capture the clear images and 3-D point clouds of gangue. Second, instead of relying on handcrafted features, a data-driven approach streamlined gangue detection network (SGDNet) is proposed to detect gangue from images. In particular, we design an adaptive fusion unit (AFU) to fuse multiple hierarchical features and assign adaptive weights to different scales of feature maps via the attention mechanism in convolutional neural networks, which can include more location details of gangue. Furthermore, we project the gangue detection results into the 3-D scanning space and introduce a simplified 3-D object segmentation algorithm based on a surface curvature filter to extract the points of gangue. The extracted points can then be used to perform volume calculations. Several experiments are performed to demonstrate the priority of the proposed SGDNet and the effectiveness of the presented gangue localization and volume measurement framework.