With global climate change, urbanization, and agricultural resource limitations, precision agriculture and crop monitoring are crucial worldwide. Integrating multi-source remote sensing data with deep learning enables accurate crop mapping, but selecting optimal network architectures remains challenging. To improve remote sensing-based fruit planting classification and support orchard management and rural revitalization, this study explored feature selection and network optimization. We proposed an improved CF-EfficientNet model (incorporating FGMF and CGAR modules) for fruit planting classification. Multi-source remote sensing data (Sentinel-1, Sentinel-2, and SRTM) were used to extract spectral, vegetation, polarization, terrain, and texture features, thereby constructing a high-dimensional feature space. Feature selection identified 13 highly discriminative bands, forming an optimal dataset, namely the preferred bands (PBs). At the same time, two classification datasets—multi-spectral bands (MS) and preferred bands (PBs)—were constructed, and five typical deep learning models were introduced to compare performance: (1) EfficientNetB0, (2) AlexNet, (3) VGG16, (4) ResNet18, (5) RepVGG. The experimental results showed that the EfficientNetB0 model based on the preferred band performed best in terms of overall accuracy (87.1%) and Kappa coefficient (0.677). Furthermore, a Fine-Grained Multi-scale Fusion (FGMF) and a Condition-Guided Attention Refinement (CGAR) were incorporated into EfficientNetB0, and the traditional SGD optimizer was replaced with Adam to construct the CF-EfficientNet architecture. The results indicated that the improved CF-EfficientNet model achieved high performance in crop classification, with an overall accuracy of 92.6% and a Kappa coefficient of 0.830. These represent improvements of 5.5 percentage points and 0.153, compared with the baseline model, demonstrating superiority in both classification accuracy and stability.