In the process of rice production, rice pests are one of the main factors that cause rice yield reduction. To implement prevention and control measures, it is necessary to accurately identify the types of rice pests and diseases. However, the application of image recognition technologies focused on the agricultural field, especially in the field of rice disease and pest identification, is relatively limited. Existing research on rice diseases and pests has problems such as single data types, low data volume, and low recognition accuracy. Therefore, we constructed the rice pest and disease dataset (RPDD), which was expanded through data enhancement methods. Then, based on the ResNet structure and the convolutional attention mechanism module, we proposed a Lightweight Multi-scale Feature Extraction Network (LMN) to extract multi-scale features at a finer granularity. The proposed LMN model achieved an average classification accuracy of 95.38% and an F1-Score of 94.5% on the RPDD. The parameter size of the model is 1.4 M, and the FLOPs is 1.65 G. The results suggest that the LMN model performs rice disease and pest classification tasks more effectively than the baseline ResNet model by significantly reducing the model size and improving accuracy.