Early recognition of fruit body diseases in edible fungi can effectively improve the quality and yield of edible fungi. This study proposes a method based on improved ShuffleNetV2 for edible fungi fruit body disease recognition. First, the ShuffleNetV2+SE model is constructed by deeply integrating the SE module with the ShuffleNetV2 network to make the network pay more attention to the target area and improve the model’s disease classification performance. Second, the network model is optimized and improved. To simplify the convolution operation, the 1 × 1 convolution layer after the 3 × 3 depth convolution layer is removed, and the ShuffleNetV2-Lite+SE model is established. The experimental results indicate that the accuracy, precision, recall, and Macro-F1 value of the ShuffleNetV2-Lite+SE model on the test set are, respectively, 96.19%, 96.43%, 96.07%, and 96.25%, which are 4.85, 4.89, 3.86, and 5.37 percent higher than those before improvement. Meanwhile, the number of model parameters and the average iteration time are 1.6 MB and 41 s, which is 0.2 MB higher and 4 s lower than that before the improvement, respectively. Compared with the common lightweight convolutional neural networks MobileNetV2, MobileNetV3, DenseNet, and EfficientNet, the proposed model achieves higher recognition accuracy, and its number of model parameters is significantly reduced. In addition, the average iteration time is reduced by 37.88%, 31.67%, 33.87%, and 42.25%, respectively. The ShuffleNetV2-Lite+SE model proposed in this paper has a good balance among performance, number of parameters, and real-time performance. It is suitable for deploying on resource-limited devices such as mobile terminals and helps in realization of real-time and accurate recognition of fruit body diseases of edible fungi.

Classification of wild mushrooms based on improved ShuffleNetV2