Speckle noise is a significant challenge in synthetic aperture radar (SAR) images, severely degrading the visual quality and compromising subsequent image interpretation tasks. While existing despeckling methods can reduce noise, they often fail to strike a appropriate balance between noise suppression and the preservation of fine image details. To address this issue, in this paper, we propose a novel SAR image despeckling method that leverages both structural image priors and noise distribution characteristics in an end-to-end framework. Our approach consists of two key components: a dual-branch subnet for coarse despeckling and noise estimation, and a noise-guided Transformer-based subnet for final image refinement. The dual-branch subnet decouples the tasks of noise estimation and despeckling, improving both noise suppression accuracy and structural detail preservation. Furthermore, a combination of grouped pooling attention (GPA) and context-aware fusion (CAF) modules enables effective multi-scale feature fusion by jointly capturing local details and global contextual information. The noise estimation branch generates adaptive priors that guide the Transformer refinement, which incorporates deformable convolutions and a masked self-attention mechanism to selectively focus on relevant image regions. Extensive experiments conducted on both synthetic and real SAR datasets demonstrate that the proposed method consistently outperforms current state-of-the-art methods, achieving superior speckle suppression while preserving fine details more effectively.