In computer vision technology, image segmentation is a significant technological advancement for the current problems of high-speed railroad image scene changes, low segmentation accuracy, and serious information loss. We propose a segmentation algorithm, DFA-UNet, based on an improved U-Net network architecture. The model uses the same encoder–decoder structure as U-Net. To be able to extract image features efficiently and further integrate the weights of each channel feature, we propose to embed the DFA attention module in the encoder part of the model for the adaptive adjustment of feature map weights. We evaluated the performance of the model on the RailSem19 dataset. The results showed that our model showed improvements of 2.48%, 0.22%, 3.31%, 0.97%, and 2.2% in mIoU, F1-score, Accuracy, Precision, and Recall, respectively, compared with U-Net. The model can effectively achieve the segmentation of railroad images.