In the academic world, ginseng (Panax ginseng C. A. Meyer) has received much attention as the most representative element of Chinese medicine. To address the lack of traditional algorithms in the identification of ginseng appearance quality and further improve the manual identification on ginseng, we propose a grading method of ginseng appearance quality based on deep learning, taking advantage of the benefits of deep learning in the image identification. Firstly, we substituted LeakyReLU for the conventional activation function ReLU to enhance the predictive power of the model. Secondly, we added an ECA module to the residual block, which allowed attention to be focused on the input object to capture more precise and detailed features. Thirdly, we used the focal loss function to solve the problem of an imbalanced dataset. Then, the self-constructed dataset was processed with data enhancement and divided into four different classes of ginseng. The dataset was trained on a model with transfer learning to finally obtain the best model applicable to the identification of ginseng appearance quality. The experiments showed that, compared with the classical convolutional neural network models VGG16, GoogLeNet, ResNet50 and Densenet121, the proposed model reported the best performance, its accuracy in the test set was as high as 97.39%, and the loss value was 0.035. This method can efficiently classify the appearance quality of ginseng, and has a significant value in the field of ginseng appearance quality identification.