Accurate and timely crop yield prediction over large spatial regions is critical to national food security and sustainable agricultural development. However, designing a robust model for crop yield prediction over a large spatial region remains challenging due to inadequate surveyed samples and an under-development of deep-learning frameworks. To tackle this issue, we integrated multi-source (remote sensing, weather, and soil properties) data into a dual-stream deep-learning neural network model for winter wheat in China’s major planting regions. The model consists of two branches for robust feature learning: one for sequential data (remote sensing and weather series data) and the other for statical data (soil properties). The extracted features by both branches were aggregated through an adaptive fusion model to forecast the final wheat yield. We trained and tested the model by using official county-level statistics of historical winter wheat yields. The model achieved an average R² of 0.79 and a root-mean-square error of 650.21 kg/ha, superior to the compared methods and outperforming traditional machine-learning methods. The dual-stream deep-learning neural network model provided decent in-season yield prediction, with an error of about 13% compared to official statistics about two months before harvest. By effectively extracting and aggregating features from multi-source datasets, the new approach provides a practical approach to predicting winter wheat yields at the county scale over large spatial regions.