Railway intrusion seriously threatens railway safety and can cause enormous loss of life and property. Therefore, railway intrusion detection is crucial for the safety of railway operation. Among the current methods of intrusion detection, machine vision-based methods have been widely used in railways, and have attracted close attention because of their great benefits. This paper proposes a comprehensive review of railway intrusion detection based on machine vision, covering ground monitoring, on-board inspection, and unmanned aerial vehicle (UAV) inspection. First, this paper systematically reviews most of the studies over the past two decades and presents the survey in three parts. Second, by analyzing these studies and the requirements for railway monitoring, we summarize the major challenges that hinder railway intrusion detection based on machine vision. Finally, we propose several promising perspectives for railway intrusion detection based on machine vision by comprehensively considering the development of machine vision, sensors, and pattern recognition together with the needs of railway scenes.

As one of the main pollution sources in China, haze can blur the railway monitoring and threaten railway safety. In this paper, we propose an end-to-end multi-scale residual network (MSRN) which can achieve remarkable dehazing effect on railway monitoring images. The method optimizes the image dehazing algorithm in three aspects: network structure, loss function and hazy dataset. Firstly, inspired by the residual network, the paper presents a method of fusing multi-scale feature information based on the residual network, which can extract more effective information at different scales. Secondly, a combined loss function is designed to achieve better convergent results by balancing training time, training calculations, and precision. Thirdly, the paper synthesizes an outdoor dataset specifically for railway scenarios, which relies on real depth maps and various outdoor images. Extensive experimental results on both full reference image quality assessment and no reference image quality assessment of image restoration demonstrate that the proposed algorithm shows higher performance than the state-of-the-art algorithms. Moreover, the haze of railway monitoring images is removed under hazy weather, and the detection algorithm achieved higher detection accuracy on the images after dehazing. The proposed network structure, loss function, and hazy dataset are discussed and analyzed in detail to verify the effectiveness of the proposed method.

Fasteners are a critical part of the rail and are used to fix the rail, which are important for train operation. Rail vibrations during train operation can cause anomalies in fasteners. At present, the 3D structured light camera is used to detect anomalies on the railway sites, but there is a lack of sufficient mining of three-dimensional data and effective fusion of multi-sensor data. To address this issue, this paper proposes a novel approach for railway fastener anomaly detection via multi-sensor fusion and self-driven loss reweighting. First, a pixel-level attention mechanism multi-sensor fusion method is applied, where the depth map is used as an attention factor to highlight edge contours and enhance abrupt changes in the gray level. Second, a feature fusion-decoupled module is proposed to obtain the dense feature maps and then decouple the detection task to output class, location and confidence, respectively. Finally, given the characteristics of the sample imbalance in the fastener dataset, a dynamic self-driven loss reweighting method is used to improve the detection accuracy of difficult samples. The experimental results show that the proposed method can achieve 86.6% precision and 61.72 FPS detection speed, better than other state-of-the-art algorithms.

Semantic segmentation in railway scenes constitutes a fundamental task in the context of autonomous train driving, serving as a critical source of positional information for other visual tasks. Yet, prior semantic segmentation algorithms have faced challenges in achieving real-time and precise railway panoptic segmentation. Railway scene images often have highly imbalanced content density. Hence, the distribution of different regions in the image is highly non-uniform, which leads to the issue of semantic sample imbalance. To achieve this, we propose a dense multi-scale fusion network (DMFNet), a lightweight and precise semantic segmentation framework tailored for railway scenes. DMFNet employs a network framework that combines dense connections and multi-scale fusion to efficiently extract and utilize features. To enhance the feature extraction capability of the neural network, we design a lightweight residual block with multiple scales, composed of large convolution kernels. Additionally, we introduce a novel method for reassigning loss function weights to balance the samples within railway scenes. Extensive experimentation demonstrates that our DMFNet achieves outstanding results on railway scene datasets. It is designed to provide a lightweight and accurate semantic segmentation solution for railway scenes.

Railway scene understanding is key to autonomous train operation and important in active train perception. However, most railway scene understanding methods focus on track extraction and ignore other components of railway scenes. Although several semantic segmentation algorithms are used to identify railway scenes, they are computationally expensive and slow with limits applications in railways. To solve these problems, we propose efficient dual-stream fusion network (EDFNet), a lightweight semantic segmentation algorithm, for understanding railway scenes. First, a dual-stream backbone network based on mobile inverted residual blocks is proposed to extract and fuse detailed features and semantic features. Next, a bi-directional feature pyramid pooling module is proposed to obtain multi-scale features and deep semantic features. Finally, a multi-task aggregate loss is designed to learn semantic and boundary information, thus improving the accuracy without increasing the computational complexity. Extensive experimental results demonstrate that EDFNet outperforms the lightweight state-of-the-art algorithms with high accuracy and fast speed on two railway datasets.

Point cloud-based 3D object detection technology provides precise information for detecting obstacles in front of trains. In rail transit scenarios, obstacles are usually located at a distance, and the sparsity of point cloud data leads to information loss during feature extraction and spatial transformation, adversely affecting the accuracy of obstacle detection. To tackle this issue, we propose RAE3D, a single-stage end-to-end architecture that incorporates a multiscale voxel feature aggregation module on point queries (MVA-PQ) and a bird's-eye view multilevel auxiliary (BEV-MLA) module for efficient 3D object detection. The MVA-PQ module encodes multiscale voxel features into centroids of the final voxel layer, integrating spatial and scale information. The BEV-MLA module converts shallow 3D features into sparse 2D features using height compression, guiding the network to learn the spatial structure of small objects. We also introduce a 3D centroid offset loss for optimizing bounding boxes. Extensive experiments on the Rail3D and KITTI datasets have demonstrated the superiority of the proposed RAE3D.