Surface defect inspection is a key step to ensure the quality of the hot rolled steel surface. However, current advanced detection (DET) methods have high precision but low detection speed, which hinders the application of the detector in actual production. In this work, a real-time detection network (RDN) focusing on both speed and accuracy is proposed to solve the problem of steel surface defect detection. RDN takes ResNet-dcn, a modular encoding, and decoding network with light weights, as the basic convolutional architecture whose backbone is pretrained on ImageNet. To improve the detection accuracy, a skip layer connection module (SCM) and a pyramid feature fusion module (PFM) are involved into RDN. On the standard dataset NEU-DET, the proposed method can achieve the state-of-the-art recognition speed of 64 frames per second (FPS) and the mean average precision of 80.0% on a single GPU, which fully meets the requirements of the detection accuracy and speed in the actual production line.