Rail surface defects have negative impacts on riding comfort and track safety, and could even lead to accidents. Based on the safety database (2020) of the Federal Railroad Administration (FRA), rail surface defects have been among the main factors causing derailments. During the past decades, there have been many efforts to detect such rail surface defects. However, the applications of earlier methods are limited by the high requirements of specialized equipment and personnel training. To date, rail surface defect inspection is still a very labor-intensive and time-consuming process, which hardly satisfies the field maintenance expectations. Therefore, a cost-effective and user-friendly automatic system that can inspect the rail surface defects with high accuracy is urgently needed. To address this issue, this study proposes a computer vision-based instance segmentation framework for rail surface defect inspection. A rail surface database including 1,040 images (260 source images and 780 augmented images) has been built. The classic instance segmentation model, Mask R-CNN, has been re-trained and fine-tuned for inspecting rail surface defects with the customized dataset. The influences of different backbones and learning rates are investigated and discussed. Experimental results indicate the ResNet101 backbone reaches better inspection capability. With a learning rate of 0.005, the re-trained Mask R-CNN model can achieve the best performance on the bounding box and mask predictions. Sixteen images are used to test the inspection performance of the fine-tuned model. The results are promising and indicate potential field applications in the future.