The immersive experience of holographic displays is fundamentally limited by their space–bandwidth product (SBP), which imposes an inherent trade-off between the field of view (FOV) and eyebox size. This paper proposes a method to extend the SBP by employing cascaded modulation with a dynamic spatial light modulator (SLM) and a passive high-resolution binary random phase mask (BRPM). We find that the key to unlocking this extension of SBP lies in a sophisticated algorithmic optimization, grounded in a physically accurate model of the system. We identify and correct the Nyquist undersampling problem caused by high-frequency scattering in standard diffraction models. Based on this physically accurate model, we employ a gradient descent optimization framework to achieve efficient, end-to-end solving for complex light fields. Simulation and experimental results demonstrate that our method achieves an approximately 16-fold SBP extension (4-fold FOV) while delivering significantly superior reconstructed image quality compared to the traditional Gerchberg–Saxton (GS) algorithm. Furthermore, this study quantitatively reveals the system’s extreme sensitivity to sub-pixel level alignment accuracy, providing critical guidance for the engineering and implementation of our proposed method.