Pansharpening plays an important role in remote sensing image processing. Its purpose is to fuse a high-spatial-resolution panchromatic (PAN) image and a low-spatial-resolution multispectral (LRMS) image, thereby reconstructing a high-resolution multispectral (HRMS) image with both high spatial clarity and high spectral fidelity. In recent years, diffusion models have shown great potential in image generation. However, existing diffusion-based pansharpening methods usually adopt a fixed denoising strategy, making it difficult to adapt to the stage-wise changes in the denoising process and complex degradation distributions. Based on this, we propose PanDiM, an efficient generative framework for pansharpening. Specifically, we reformulate pansharpening as a high-frequency residual restoration process constrained by multimodal conditions. To improve the response accuracy of the model in complex regions, we design a Degradation-Posterior Guidance Module (DPGM), which extracts dual-scale physical detail priors from the PAN image, explicitly infers the degradation posterior, and converts it into dynamic control variables to adaptively regulate the state evolution of Mamba. In addition, we propose a time-aware mechanism, which allows temporal information to directly intervene in posterior estimation and state-space modeling, so as to accurately match the modeling requirements of different denoising stages. Considering the characteristics of residual reconstruction, we further propose a frequency-decoupled loss (FDL), which separates low- and high-frequency components in the frequency domain and applies targeted constraints. This significantly enhances the model’s ability to represent textures and achieves more robust spectral fidelity. Extensive experiments on three benchmark datasets, including WorldView-3, GaoFen-2, and QuickBird, show that PanDiM significantly outperforms existing mainstream methods in both reduced-resolution and full-resolution evaluations, providing a new solution for high-fidelity pansharpening in complex scenarios.