Photovoltaic (PV) power exhibits high variability and non-stationarity due to irradiance fluctuations, cloud shading, and seasonal changes, which complicate ultra-short-term multi-step forecasting. This study proposes a Solar-Historical Prior Network (SHPNet) for forecasting at 5 min resolution. SHPNet integrates a solar-geometry clear-sky prior-guided temporal convolutional network (SGCP-TCN), a similar historical window (SHW) branch, and horizon-wise adaptive fusion (HA). SGCP-TCN estimates clear-sky power potential from site coordinates and timestamps and reformulates direct power prediction as clear-sky power ratio forecasting. SHW retrieves training windows from the same intra-day time slot that exhibit similar power–irradiance evolution, thereby constructing a non-parametric historical prior, while HA determines horizon-specific fusion weights based on validation errors. Unlike purely data-driven predictors and conventional similar-day methods, SHPNet combines a physically interpretable power scale with input-window-level historical evolution patterns and adaptively balances the two priors across forecasting horizons. Across the two sites, SHPNet reduced the mean MAE and RMSE by 14.07% and 10.26%, respectively, compared with the original TCN, while increasing the mean R² from 0.8279 to 0.8613. Evaluations under different weather conditions and across seasons demonstrate consistent forecasting performance, while convergence analysis confirms stable training behavior.