Cloud contamination in optical remote sensing imagery can obscure land-surface information and induce structural blurring and spectral shifts, thereby seriously affecting subsequent land-cover interpretation and quantitative analysis. To address the instability of optical image restoration under complex cloud occlusion and the difficulty of effectively exploiting SAR auxiliary information, this paper proposes LTGC-Diff, a latent diffusion model for SAR-assisted remote sensing image cloud removal. The proposed method formulates cloud-free RGB image restoration as a latent denoising process constrained by multi-source observations. To better incorporate SAR auxiliary information into diffusion reconstruction, a diffusion-oriented conditional representation learning module is designed to transform SAR-derived features into effective conditional representations. In addition, a token-level structural condition adaptation module is designed to transform SAR patch representations into local structural guidance suitable for diffusion-based reconstruction. By jointly exploiting global scene-level guidance and local structure-aware constraints, the proposed model enables more stable and reliable recovery of cloud-obscured regions. Experimental results demonstrate that LTGC-Diff improves the structural consistency and restoration reliability of SAR-assisted optical remote sensing image cloud removal under complex cloud-contaminated scenarios. Specifically, LTGC-Diff achieves a PSNR of 32.87 dB while maintaining improved structural and spectral consistency, demonstrating its effectiveness for SAR-assisted cloud removal.