In this study, series networks (AlexNet and VGG-19) and directed acyclic graph (DAG) networks (ResNet-18, ResNet-50, and ResNet-101) with transfer learning were employed to identify and classify 13 classes of apples from 7439 images. Two training datasets, model evaluation metrics, and three visualization methods were used to objectively assess, compare, and interpret five Convolutional Neural Network (CNN)-based models. The results show that the dataset configuration had a significant impact on the classification results, as all models achieved over 96.1% accuracy on dataset A (training-to-testing = 2.4:1.0) compared to 89.4–93.9% accuracy on dataset B (training-to-testing = 1.0:3.7). VGG-19 achieved the highest accuracy of 100.0% on dataset A and 93.9% on dataset B. Moreover, for networks of the same framework, the model size, accuracy, and training and testing times increased as the model depth (number of layers) increased. Furthermore, feature visualization, strongest activations, and local interpretable model-agnostic explanations techniques were used to show the understanding of apple images by different trained models, as well as to reveal how and why the models make classification decisions. These results improve the interpretability and credibility of CNN-based models, which provides guidance for future applications of deep learning methods in agriculture.