Automatic detection of industrial product damage using machine learning is a promising research area. At the same time, various machine learning methods based on convolutional neural networks have a very important role in the application of visual automatic detection. Therefore, the machine vision-based automatic detection of high-speed railway rail damage has received widespread attention. This paper proposes an efficient detection method for the damage of high-speed railway rails called SCueU-Net. For the first time, the combination of U-Net graph segmentation network and the saliency cues method of damage location is applied to the task of high-speed railway rail damage detection. The experimental results show that our method has a detection accuracy rate of 99.76%, which is 6.74% higher than the recent method in damage identification accuracy, which improves the detection efficiency of rail damage significantly.