This article aims to improve the performance of deep learning-based defective fastener inspection method. Due to the defective fasteners are insufficient and far less than the defect-free ones in real railway, it is difficult to train a robust fastener inspection model on such imbalanced dataset. In view of this problem, a novel image generation method called four-discriminator cycle-consistent adversarial network (FD-Cycle-GAN) is proposed to generate the defect fastener images using a large number of defect-free ones. Extensive experiments are conducted on the real fastener images and generated images. Experimental results demonstrate that the defect fastener images generated by our proposed method have better quality and richer diversity than those generated by other state-of-the-art methods. In addition, compared with the CNN-only baseline, the performance of the fastener inspection model trained on the expanded dataset containing the defect fastener images generated by FD-Cycle-GAN is improved significantly. The detection accuracy and relative IMP reach 93.25% and 21.59% respectively.

Sample imbalance is the main factor affecting detection performance of defective fasteners. Most of existing methods focus on leveraging image generation and unsupervised idea to overcome sample imbalance and realize detection of defective fasteners, while learning discriminative features with deep metric learning for imbalanced fastener detection is neglected. Thus, we study variants of deep metric learning losses and propose a novel loss named center-triplet loss for defective fastener detection in the case of imbalanced samples. Specially, the proposed center-triplet loss can learn a center for each fastener state and this center is closer to the same fastener state than to different fastener states. In this way, the learned features are more robust and discriminative and it is beneficial to realize accurate detection of defective fasteners when samples are uneven. Extensive experiments are conducted on the constructed fastener dataset, and the results demonstrate the effectiveness and superiority of our proposed loss for the detection of imbalanced fasteners.

Fast detection of fasteners is important to improve the efficiency of railroad maintenance. However, this task remains challenging due to the limited computing resources of inspection system. To solve the challenge, a new lightweight detection network Op-YOLOv4-tiny is proposed in this paper. The proposed network firstly uses ResBlock-N modules to replace the CSPBlock modules in YOLOv4-tiny to reduce the computation complexity. Then, a large scale feature map ( $52\times 52$ ) is added to obtain more features of fasteners to improve the detection accuracy. Extensive experiments are conducted on the captured railway and subway track images and the results show that Op-YOLOv4-tiny has good performance in terms of detection accuracy and speed. In detail, the detection speed and accuracy reach 408 FPS and 96.8%, respectively. In addition, compared with other detection networks and state-of-the-arts, it achieves the better performance. Thus, our proposed Op-YOLOv4-tiny is with some potential industrial application value for fast detection of fasteners.

Fatigue driving behavior recognition in all-weather real driving environments is a challenging task. Accurate recognition of fatigue driving behavior is helpful to improve traffic safety. The facial landmark detector is crucial to fatigue driving recognition. However, existing facial landmark detectors are mainly aimed at stable front face color images instead of side face gray images, which is difficult to adapt to the fatigue driving behavior recognition in real dynamic scenes. To maximize the driver’s facial feature information and temporal characteristics, a fatigue driving behavior recognition method based on a multi-scale facial landmark detector (MSFLD) is proposed. First, a spatial pyramid pooling and multi-scale feature output (SPP-MSFO) detection model is built to obtain a face region image. The MSFLD is a lightweight facial landmark detector, which is composed of convolution layers, inverted bottleneck blocks, and multi-scale full connection layers to achieve accurate detection of 23 key points on the face. Second, the aspect ratios of the left eye, right eye and mouth are calculated in accordance with the coordinates of the key points to form a fatigue parameter matrix. Finally, the combination of adaptive threshold and statistical threshold is used to avoid misjudgment of fatigue driving recognition. The adaptive threshold is dynamic, which solves the problem of the difference in the aspect ratio of the eyes and mouths of different drivers. The statistical threshold is a supplement to solve the problem of driver’s low eye threshold and high mouth threshold. The proposed methods are evaluated on the Hunan University Fatigue Detection (HNUFDD) dataset. The proposed MSFLD achieves a normalized mean error value of 5.4518%, and the accuracy of the fatigue driving recognition method based on MSFLD achieves 99.1329%, which outperforms that of state-of-the-art methods.

Rail surface defect segmentation can provide a reliable basis for the severity evaluation of rail diseases. The deep learning technology has been widely applied to segment rail surface defects due to the powerful feature representation ability. However, most of the existing deep learning-based methods often produce inaccurate defect region boundaries and unsatisfactory segmentation results since the inadequate integration of contextual information and insufficient edge features. To tackle this problem, a novel region and edge-aware network (REA-Net) is proposed for rail surface defect segmentation in this paper. REA-Net adopts an encoder-decoder framework, where the encoder network incorporates three modules: feature pyramid edge module, multi-task learning module, and cross-level fusion module, to encode richer context information and finer-grained features at each stage. The decoder network aggregates the encoding features from each stage to obtain decoding features, and final these features are used to predict the defect segmentation results. Extensive experiments are conducted on expanded public rail surface defect dataset (RSDD) and the results demonstrate our proposed REA-Net achieves good segmentation performance and outperforms the state-of-the-arts in terms of F ₁ -scores and IoU.

Shadow occlusion problem has been a hot research topic in fringe projection profilometry (FPP). Since the shadow area cannot be covered by fringe patterns and is easy to induce error, it should be identified and removed. The key to shadow extraction is to determine an accurate shadow threshold. In addition, the threshold is expected to be automatically adjusted as the brightness changes. In this article, we put forward a novel shadow extraction method based on 2-D discrete wavelet transform (SE-DWT) to suppress outlier points caused by the shadow area. Meanwhile, the multi-information fusion strategy is presented to improve the anti-interference performance and the accuracy of shadow segmentation. More importantly, unlike the conventional methods that determine the shadow threshold based on the light intensity, we make full of the high-frequency property of shadow on the phase map and convert the problem of shadow segmentation into the problem of frequency analysis. Furthermore, the shadow threshold can be determined by analyzing the discrete shadow area. Finally, we can utilize the shadow threshold to binarize the background image and obtain a continuous shadow mask. The experimental results demonstrate that the proposed method can dynamically adjust the shadow threshold under different brightnesses, contrasts, and positions and can obtain an accurate shadow mask.