With the advancement of industrial intelligence, defect recognition has become an indispensable part of facilitating surface quality in the steel manufacturing process. To assure product quality, most previous studies were typically trained with many defect samples. Nonetheless, a large quantity of defect samples is difficult to obtain, owing to the rare occurrence of defects. In general, deep learning-based methods underperformed as they have inherent limitations due to inadequate information, thereby restraining the application of models. In this study, a two-level Gaussian pyramid is applied to decompose raw data into different resolution levels simultaneously filtering the noises to acquire compact and representative features. Subsequently, a multi-receptive field fusion-based network (MRFFN) is developed to learn the hierarchical features and synthesize the respective prediction scores to form the final recognition result. As a result, the proposed method is capable of exhibiting an outstanding performance of 99.75% when trained using a lightweight dataset. In addition, the experiments conducted using the disturbance defect dataset showed the robustness of the proposed MRFFN against common noises and motion blur.