High-resolution remote sensing satellites have revolutionized remote sensing research, yet accurately segmenting specific targets from complex satellite imagery remains challenging. While the Segment Anything Model (SAM) has emerged as a promising universal segmentation model, its direct application to remote sensing imagery yields suboptimal results. To address these limitations, we propose RSAM-Seg, a novel deep learning model adapted from SAM specifically designed for remote sensing applications. Our model incorporates two key components: Adapter-Scale and Adapter-Feature modules. The Adapter-Scale modules, integrated within Vision Transformer (ViT) blocks, enhance model adaptability through learnable transformations, while the Adapter-Feature modules, positioned between ViT blocks, generate image-informed prompts by incorporating task-specific information. Extensive experiments across four binary and two multi-class segmentation scenarios demonstrate the superior performance of RSAM-Seg, achieving an $F1$ score of 0.815 in cloud detection, 0.834 in building segmentation, and 0.755 in road extraction, consistently outperforming established architectures like U-Net, DeepLabV3+, and Segformer. Moreover, RSAM-Seg shows significant improvements of up to 56.5% in $F1$ score compared to the original SAM. In addition, RSAM-Seg maintains robust performance in few-shot learning scenarios, achieving an $F1$ score of 0.656 with only 1% of the training data and increasing to 0.815 with full data availability. Furthermore, RSAM-Seg exhibits the capability to detect missing areas within the ground truth of certain datasets, highlighting its capability for completion.