The defects on a leather surface may be caused by the poor material handling process during the production and manufacturing stages. It is essential to eliminate the natural variations and artificial injuries on the leather surfaces, in order to control the quality of the products and achieve customer satisfaction. To date, the visual inspection of the leather defects is performed manually by human operators. Thus, this paper aims to introduce an automatic defect detection technique by employing a deep learning method. Specifically, the proposed method consists of two stages: classification and instance segmentation. The former stage distinguishes whether the piece of the leather sample contains a defective part or not, whereas the latter is to localize the precise defective location. To accomplish the tasks, the dataset is first collected under a proper laboratory environment. Among 250 defective samples and 125 non-defective samples, the proposed method has been demonstrated its feature learning capability by producing promising performance when considering relatively fewer training samples. Particularly, the defect types focused in this study are the black lines and wrinkles. The best performance obtained is ∼95% for the classification task, whereas the segmentation task reaches an Intersection over Union rate of 99.84%