In the recognition of fruit leaf diseases, image recognition technology based on deep learning has received increasing attention. However, deep learning models often perform poorly in complex backgrounds, and in some cases, they even outperform traditional algorithms. To address this issue, this paper proposes a Frequency-Adaptive Attention (FA-attention) mechanism that leverages the significance of frequency-domain features in fruit leaf disease regions. By enhancing the processing of frequency domain features, the recognition performance in complex backgrounds is improved. Specifically, FA-attention combines Fourier transform with the attention mechanism to extract frequency domain features as key features. Then, this mechanism is integrated with the Unet model to obtain feature maps strongly related to frequency domain features. These feature maps are fused with multi-scale convolutional feature maps and then used for classification. Experiments were conducted on the Plant Village (PV) dataset and the Plant Pathology (PP) dataset with complex backgrounds. The results indicate that the proposed FA-attention mechanism achieves significant effects in learning frequency domain features. Our model achieves a recognition accuracy of 99.91% on the PV dataset and 89.59% on the PP dataset. At the same time, the convergence speed is significantly improved, reaching 94% accuracy with only 20 epochs, demonstrating the effectiveness of this method. Compared with classical models and state-of-the-art (SOTA) models, our model performs better on complex background datasets, demonstrating strong generalization capabilities.