The Soil and Plant Analyzer Development (SPAD) value is a reliable proxy for chlorophyll, yet conventional field measurement remains labor-intensive and spatially limited. Current remote sensing inversion models typically depend on costly multispectral sensors and rarely account for phenological changes, restricting their applicability across orchards and seasons. To overcome these limitations, this study introduces a stage-aware and low-cost SPAD inversion framework for jujube trees, integrating multi-source data fusion and an optimized ensemble model. A two-year experiment (2023–2024) combined UAV multispectral vegetation indices (VI) with RGB-derived color indices (CI) across leaf expansion, flowering, and fruit-setting stages. Rather than using static features, stage-specific predictors were systematically identified through a hybrid selection mechanism combining Random Forest Cumulative Feature Importance (RF-CFI), Recursive Feature Elimination (RFE), and F-tests. Building on these tailored features, XGBoost, decision tree (DT), CatBoost, and an Optimized Integrated Architecture (OIA) were developed, with all hyperparameters globally tuned using a genetic algorithm (GA). The RFI-CFI-OIA-GA model delivered superior accuracy (R² = 0.758–0.828; MSE = 0.214–2.593; MAPE = 0.01–0.045 in 2024) in the training dataset, and robust cross-year transferability (R² = 0.541–0.608; MSE = 0.698–5.139; MAPE = 0.015–0.058 in 2023). These results demonstrate that incorporating phenological perception into multi-source data fusion substantially reduces interference and enhances generalizability, providing a scalable and reusable strategy for precision orchard management and spatiotemporal SPAD mapping.