Deep-learning-based detection methods have been widely applied to industrial defect inspection. However, directly using vanilla detection methods fails to achieve satisfying performance due to the lack of identifiable features. In this paper, a novel attention-based multi-scale feature fusion method (AMFF) is proposed, aiming to enhance defect features and improve defect identification by leveraging attention mechanism in the feature fusion. AMFF includes self-enhanced attention module (SEAM) and cross-enhanced attention module (CEAM). SEAM is performed on a single feature map, which first adopts multiple dilation convolutions to enrich contextual information without compromising resolution and then utilizes the intra-layer attention on the current feature map. CEAM takes both the current feature map and the adjacent feature map as input to perform cross-layer attention. The adjacent feature map is modulated with the guidance of the current feature map, which is then combined with the current feature map and the output of SEAM for final prediction. AMFF is utilized in current feature fusion networks, e.g., FPN and PAFPN, and is further integrated into prevalent detectors to guide them to pay more attention to defects rather than the background. Extensive experiments are conducted on two real industrial datasets released by Tianchi platform, i.e., fabric and aluminum defect datasets. For each dataset, 500 images are randomly selected for test and the rest for training. The proposed AMFF is demonstrated to significantly boost defect detection accuracy with acceptable computational cost, and the real-time performance could fully satisfy practical requirements.