Drill core lithology is an important indicator reflecting the geological conditions of the drilling area. Traditional lithology identification usually relies on manual visual inspection, which is time-consuming and professionally demanding. In recent years, the rapid development of convolutional neural networks has provided an innovative way for the automatic prediction of drill core images. In this work, a core dataset containing a total of 10 common lithology categories in underground engineering was constructed. ResNeSt-50 we adopted uses a strategy of combining channel-wise attention and multi-path network to achieve cross-channel feature correlations, which significantly improves the model accuracy without high model complexity. Transfer learning was used to initialize the model parameters, to extract the feature of core images more efficiently. The model achieved superior performance on testing images compared with other discussed CNN models, the average value of its Precision, Recall, F_1−score for each category of lithology is 99.62%, 99.62%, and 99.59%, respectively, and the prediction accuracy is 99.60%. The test results show that the proposed method is optimal and effective for automatic lithology classification of borehole cores.