Defective shafts need to be classified because some defective shafts can be reworked to avoid replacement costs. Therefore, the detection and classification of shaft surface defects has important engineering application value. However, in the factory, shaft surface defect inspection and classification are done manually, with low efficiency and reliability. In this paper, a deep learning method based on convolutional neural network feature extraction is used to realize the object detection and classification of metal shaft surface defects. Through image segmentation, the system methods setting of a Fast-R-CNN object detection framework and parameter optimization settings are implemented to realize the classification of 16,384 × 4096 large image little objects. The experiment proves that the method can be applied in practical production and can also be extended to other fields of large image micro-fine defects with a high light surface. In addition, this paper proposes a method to increase the proportion of positive samples by multiple settings of IOU values and discusses the limitations of the system for defect detection.