Remote sensing imagery (RSI) segmentation plays a crucial role in environmental monitoring and geospatial analysis. However, in real-world practical applications, the domain shift problem between the source domain and target domain often leads to severe degradation of model performance. Most existing unsupervised domain adaptation methods focus on aligning global-local domain features or category features, neglecting the variations of ground object categories within local scenes. To capture these variations, we propose the scene covariance alignment (SCA) approach to guide the learning of scene-level features in the domain. Specifically, we propose a scene covariance alignment model to address the domain adaptation challenge in RSI segmentation. Unlike traditional global feature alignment methods, SCA incorporates a scene feature pooling (SFP) module and a covariance regularization (CR) mechanism to extract and align scene-level features effectively and focuses on aligning local regions with different scene characteristics between source and target domains. Experiments on both the LoveDA and Yanqing land cover datasets demonstrate that SCA exhibits excellent performance in cross-domain RSI segmentation tasks, particularly outperforming state-of-the-art baselines across various scenarios, including different noise levels, spatial resolutions, and environmental conditions.