Fashion trend forecasting is crucial for proactive supply chain management, sustainable production, and personalized marketing. However, accurately predicting fine-grained fashion trends across diverse user groups remains challenging, as conventional numerical models rely solely on historical popularity signals and cannot explicitly exploit the high-level semantic context underlying trend evolution. To address this limitation, we propose a Dual-Stream Driven Fashion Trend Forecasting (DDFTF) framework that integrates numerical forecasting with textual semantic reasoning. The numerical stream employs a multi-scale patch Transformer with metadata-aware feature fusion to capture temporal dynamics, while the semantic stream converts historical trends into structured textual descriptions, extracting interpretable semantic representations of global trends and turning points. A residual fusion module combines both streams by treating semantic signals as complementary corrections to numerical predictions. Extensive experiments on the FIT and GeoStyle benchmarks demonstrate that DDFTF consistently outperforms state-of-the-art methods, achieving up to a 14.6% relative MAE reduction in long-term forecasting. Ablation and qualitative analyses further show that the performance gains arise from meaningful semantic information rather than increased model capacity, while also improving the interpretability of fashion trend prediction. Our code is publicly available.