Urban forest tree species composition and spatial distribution are essential for refined greenspace management, ecosystem-service assessment, and forest-health monitoring. High-resolution imagery captures crown texture and spatial boundaries, multi-temporal NDVI reflects phenological differences, LiDAR provides canopy height and structural information, and hyperspectral imagery offers detailed spectral discrimination. However, differences in spatial resolution and information representation limit the effectiveness of simple feature stacking. Motivated by the challenges associated with heterogeneous stand structure, similarity among broadleaved species, and insufficient multi-source fusion, this study developed HybridFusion for fine-grained tree species classification in the northern section of Beijing Olympic Forest Park. The model uses 0.5 m high-resolution imagery as its spatial basis and integrates NDVI phenological features from March, April, and August, LiDAR-derived canopy structural features, and hyperspectral–semantic priors through branch encoding, dynamic gated fusion, and reliability-aware hyperspectral modulation. The full-modality model achieved optimal performance (OA: 90.44%; Tree OA: 85.06%; mIoU: 77.88%). Poplar achieved an F1 score of 95.31%, while willow, coniferous trees, and other tree species reached 86.27%, 84.90%, and 85.35%, respectively. Ablation studies confirmed the complementary value of the multi-source data: integrating NDVI and LiDAR increased Tree OA from 67.29% to 81.90%, and the addition of hyperspectral–semantic priors further raised it to 85.06%. These results demonstrate the effectiveness of jointly exploiting phenological, structural, and spectral–semantic information for fine-scale urban forest inventory and management.