Remote sensing imagery is a crucial component of remote sensing data. However, in its application to downstream tasks, cloud cover can hinder effective data utilization, making the removal of cloud occlusion from remote sensing images a persistent and important research direction. Recently, diffusion models have demonstrated powerful performance in conditional image generation. However, their direct application to cloud removal yields suboptimal results, as the interference pattern of random Gaussian noise differs significantly from that of actual cloud occlusion. To address this, we developed the Perlin Noise-Based Cloud Removal Diffusion Model (PCRDiff). Compared to traditional diffusion models, PCRDiff abandons random Gaussian noise and instead utilizes Perlin noise to simulate the interference pattern of cloud occlusion on images. Based on this, we designed a novel training and iterative denoising process, along with a corresponding Perlin noise intensity quantization module. Furthermore, we developed a multi-attention fusion module as the backbone of the model to enhance its performance. Extensive experiments on two commonly used benchmark datasets demonstrate that our method achieves superior performance across multiple metrics.