Road hazards can lead to dangerous accidents and endanger the safety of pedestrians. Frequent and thorough road inspection is required to maintain road safety. This paper proposes an improved U-Net model that combines gray-scale images and depth images and uses a data statistics method based on a road depth map to eliminate hazard-free data automatically and reduce the computational complexity involved in hazard detection. Experiments showed that the proposed model, based on the improved U-Net-based pavement hazard recognition and extraction algorithm, could smoothly and efficiently extract pavement cracks and deformation hazards in complex scenes with noise interference and produce results with strong robustness. Comprehensive indicators such as global recognition accuracy rate A (Accuracy), precision rate P (Precision), recall rate R (Recall), evaluation index F1 (F -Measure), and Mean Intersection over Union (MIoU) were used to assess the effectiveness of the proposed model in comparison with existing hazard detection models; the proposed model greatly outperformed the models in all the indicators. The proposed model can provide a significant reference for subsequent pavement repair work and be used to improve road safety.