Moroccan installations of solar photovoltaic panels experience operational difficulties due to shading and vegetation-related soiling, which reduce energy output by 15–30%. Most monitoring systems depend upon a single vegetation index, which can reduce the accuracy of detecting even moderate anomalies. This paper presents a novel integration of multi-criteria hierarchical labeling with dual-branch deep learning for enhanced vegetation anomaly detection. We combined MODIS (2000–2015) and Sentinel-2 (2015–2025) images and NASA POWER weather records to study a 25-year vegetation record using multi-source satellite data in 5 of Morocco’s ecologically diverse zones. We introduced a three-class hierarchical labeling scheme (normal, moderate, severe) for dynamic vegetation models based on combined vegetation indices (NDVI, EVI, NDWI) and meteorological thresholds. The proposed dual-branch architecture uses independent data streams for unfused data, which include temporal multi-scale CNNs (TMSCNN) for spatiotemporal modeling and bidirectional LSTMs for weather-integrated vegetation data. Systematic ablation studies show improvements from using NDVI (68.98%) to multispectral indices (77.74%), meteorological integration (81.02%), and a final accuracy of 82.34% ± 0.88%. The moderate anomaly class exhibits lower precision (65%), demonstrating the challenge of operationalizing severity-based anomaly classification. This work integrates hierarchical, multi-criteria labeling and hybrid deep learning for solar photovoltaic vegetation monitoring.