Litchi (Litchi chinensis Sonn.) is a pillar of the tropical agricultural economy in southern China, yet its production faces increasing instability due to climate change. Traditional agronomic models often fail to capture the complex, non-linear interactions between meteorological drivers and yield formation in perennial fruit trees. To address this challenge, the study constructed a yield prediction framework using an optimized Random Forest (RF) model integrated with interpretable machine learning (SHAP), based on a comprehensive dataset from 17 major production regions in Hainan Province (2000–2022). The model demonstrated robust predictive capability at the provincial scale (R² = 0.564, RMSE = 2.1 t/ha) and high consistency across regions (R² ranging from 0.51 to 0.94). Feature importance analysis revealed that heat accumulation (specifically growing degree days above 20 °C) is the dominant driver, explaining over 85% of yield variability. Crucially, scenario simulations uncovered asymmetric climate risks across phenological stages: while moderate warming generally enhances yield by promoting vegetative growth and ripening, it acts as a stressor during the Fruit Development stage, where temperatures exceeding 26 °C trigger yield decline. Furthermore, the yield penalty for drought during Flowering (−8.09%) far outweighed the marginal benefits of surplus rainfall, identifying this window as critically sensitive to water deficits. These findings underscore the necessity of phenology-aligned adaptation strategies—specifically, securing irrigation during flowering and deploying cooling interventions during fruit development—providing a data-driven basis for climate-smart management in tropical agriculture.