Accurate soybean yield estimation is critically imperative for modern agricultural systems amidst escalating global food security pressures, yet conventional methodologies are constrained for large-scale, high-frequency monitoring. To address this, an innovative deep learning framework, TransBiHGRU-PSO, is proposed for precise large-scale soybean yield estimation via effective fusion of multi-source, multitemporal remote sensing data, emphasizing robust and accurate estimation even with anomalous yield data. This framework synergistically integrates an optimized Bidirectional Hierarchical Gated Recurrent Unit (BiHGRU), a Transformer encoder, and a novel Greenness and Water Content Composite Index (GWCCI), with critical parameters optimized by Particle Swarm Optimization (PSO). County-level yield data from 12 U.S. states were used, supplemented by multi-temporal remote sensing datasets (MODIS surface reflectance, vegetation indices, environmental variables). Empirical analyses showed TransBiHGRU-PSO demonstrated improved estimation capability and generalizability compared to multiple benchmark models. Notably, with anomalous yield data retained, the model achieved solid test set performance (coefficient of determination (R²) of 0.71, root mean square error (RMSE) of 4.2812 bushels/acre). Compared to the best traditional machine learning model (SVR), R² increased by 52.96% and RMSE decreased by 26.05%; and relative to the best deep learning baseline model (LSTM), R² and RMSE improved by 7.04% and 7.04%, respectively. Furthermore, validation of interannual stability (2008–2018, anomalies retained) revealed a mean R² of 0.70 and mean RMSE of 4.4701 bushels/acre, affirming its consistency under complex real-world conditions. This TransBiHGRU-PSO algorithmic framework, combined with multi-source and multitemporal remote sensing data, offers a valuable exploration for large-scale soybean yield estimation.