This paper provides a review of the literature on vision-based on-board obstacle detection and distance estimation in railways. Environment perception is crucial for autonomous detection of obstacles in a vehicle’s surroundings. The use of on-board sensors for road vehicles for this purpose is well established, and advances in Artificial Intelligence and sensing technologies have motivated significant research and development in obstacle detection in the automotive field. However, research and development on obstacle detection in railways has been less extensive. To the best of our knowledge, this is the first comprehensive review of on-board obstacle detection methods for railway applications. This paper reviews currently used sensors, with particular focus on vision sensors due to their dominant use in the field. It then discusses and categorizes the methods based on vision sensors into methods based on traditional Computer Vision and methods based on Artificial Intelligence.

This paper presents a novel method for generation of synthetic images of obstacles on and near rail tracks over long-range distances. The main goal is to augment the dataset for autonomous obstacle detection (OD) in railways, by inclusion of synthetic images that reflect the specific need for long-range OD in rail transport. The presented method includes a novel deep learning (DL)-based rail track detection that enables context- and scale-aware obstacle-level data augmentation. The augmented dataset is used for retraining of a state-of-the-art CNN for object detection. The evaluation results demonstrate significant improvement of detection of distant objects by augmentation of training dataset with synthetic images.

Rail transport plays a crucial role in promoting sustainability and reducing the environmental impact of transport. Ongoing efforts to improve the sustainability of rail transport through technological advancements and operational improvements are further enhancing its reputation as a sustainable mode of transport. Autonomous obstacle detection in railways is a critical aspect of railway safety and operation. While the widespread deployment of autonomous obstacle detection systems is still under consideration, the ongoing advancements in technology and infrastructure are paving the way for their full implementation. The SMART2 project developed a holistic obstacle detection (OD) system consisting of three sub-systems: long-range on-board, trackside (TS), and Unmanned Aerial Vehicle (UAV)-based OD sub-systems. All three sub-systems are integrated into a holistic OD system via interfaces to a central Decision Support System (DSS) that analyzes the inputs of all three sub-systems and makes decision about locations of possible hazardous obstacles with respect to trains. A holistic approach to autonomous obstacle detection for railways increases the detection area, including areas behind a curve, a slope, tunnels, and other elements blocking the train’s view on the rail tracks, in addition to providing long-range straight rail track OD. This paper presents a demonstration of the SMART2 holistic OD performed during the operational cargo haul with in-service trains. This paper defines the demonstration setup and scenario and shows the performance of the developed holistic OD system in a real environment.