Semantic segmentation is an important task for the interpretation of remote sensing images. Remote sensing images are large in size, contain substantial spatial semantic information, and generally exhibit strong symmetry, resulting in images exhibiting large intraclass variance and small interclass variance, thus leading to class imbalance and poor small-object segmentation. In this paper, we propose a new remote sensing image semantic segmentation network, called CAS-Net, which includes coordinate attention (CA) and SPD-Conv. In the model, we replace stepwise convolution with SPD-Conv convolution in the feature extraction network and add a pooling layer into the network to avoid the loss of detailed information, effectively improving the segmentation of small objects. The CA is introduced into the atrous spatial pyramid pooling (ASPP) module, thus improving the recognizability of classified objects and target localization accuracy in remote sensing images. Finally, the Dice coefficient was introduced into the cross-entropy loss function to maximize the gradient optimization of the model and solve the classification imbalance problem in the image. The proposed model is compared with several state-of-the-art models on the ISPRS Vaihingen dataset. The experimental results demonstrate that the proposed model significantly optimizes the segmentation effect of small objects in remote sensing images, effectively solves the problem of class imbalance in the dataset, and improves segmentation accuracy.