Surface quality assessment of magnetic tile before mounting is extremely significant. At present, this task is mainly accomplished by experienced workers in industry, which exposes the drawbacks of low efficiency and high cost. To overcome these issues, an intelligent system is developed to perform this task, which appears to be an efficient and reliable substitute for human workers. In this paper, deep learning technique is embedded into our system for automatic defect identification. However, conventional convolutional neural network (CNN) is not suitable for this classification task, since the input is a sample rather than a single image. To overcome this problem, an end-to-end CNN architecture is proposed, termed fusion feature CNN (FFCNN). FFCNN consists of three modules: feature extraction module, feature fusion module and decision-making module. The feature extraction module is designed to extract features from different images. The feature fusion module is to fuse the features extracted by feature extraction module. The decision-making module is to predict the label by the fused features. Furthermore, an attention mechanism is introduced to focus on more representative parts and suppress less important information. Experimental results demonstrated that the developed system is effective and efficient for magnetic tile surface defect detection.