Crop disease has been a severe issue for agriculture, causing economic loss for growers. Thus, disease identification urgently needs to be addressed, especially for precision agriculture. As of today, deep learning has been widely used for crop disease identification combined with optical imaging sensors. In this study, a lightweight convolutional neural network model is designed and validated on two publicly available imaging datasets and one self-built dataset with 28 types of leaf and leaf disease images of 6 crops as the research object. This model is an improvement of the existing convolutional neural network, reducing the floating-point operations by 65%. In addition, dilated depth-wise convolutions were used to increase the network receptive field and improve the model recognition accuracy without affecting the network computational speed. Meanwhile, two attention mechanisms are optimized to reduce attention module computation, improving the capability of the model to select the correct regions of interest. After training, this model achieved an average accuracy of 99.86%, and the image calculation speed was 0.173 s. Comparing with 11 backbone models and 5 latest crop leaf disease identification studies, the proposed model achieved the highest accuracy. Therefore, this model with an advantage of balancing between the calculation speed and recognition accuracy. Furthermore, the proposed model provides a theoretical basis and technical support for the practical application and mobile terminal applications of crop disease recognition in precision agriculture.