Combining disease categories and crop species leads to complex intra-class and inter-class differences. Significant intra-class difference and subtle inter-class difference pose a great challenge to high-precision crop disease classification tasks. To this end, we propose a multi-granularity feature aggregation method for accurately identifying disease types and crop species as well as better understanding the disease-affected regions implicitly. Specifically, in order to capture fine-grained discriminating clues to disease categories, we first explored the pixel-level spatial self-attention to model the pair-wise semantic relations. Second, we utilized the block-level channel self-attention to enhance the feature-discriminative ability of different crop species. Finally, we used a spatial reasoning module to model the spatial geometric relationship of the image patches sequentially, such that the feature-discriminative ability of characterizing both diseases and species is further improved. The proposed model was verified on the PDR2018 dataset, the FGVC8 dataset, and the non-lab dataset PlantDoc. Experimental results demonstrated our method reported respective classification accuracies of 88.32%, 89.95%, and 89.75% along with F1-scores of 88.20%, 89.24%, and 89.13% on three datasets. More importantly, the proposed architecture not only improved the classification accuracy but also promised model efficiency with low complexity, which is beneficial for precision agricultural applications.