The domain of remote sensing image processing has witnessed remarkable advancements in recent years, with deep convolutional neural networks (CNNs) establishing themselves as a prominent approach for building segmentation. Despite the progress, traditional CNNs, which rely on convolution and pooling for feature extraction during the encoding phase, often fail to precisely delineate global pixel interactions, potentially leading to the loss of vital semantic details. Moreover, conventional CNN-based segmentation models frequently neglect the nuanced semantic differences between shallow and deep features during the decoding phase, which can result in subpar feature integration through rudimentary addition or concatenation techniques. Additionally, the unique boundary characteristics of buildings in remote sensing images, which offer a rich vein of prior information, have not been fully harnessed by traditional CNNs. This paper introduces an innovative approach to building segmentation in remote sensing images through a prior-guided dual branch multi-feature fusion network (PDBMFN). The network is composed of a prior-guided branch network (PBN) in the encoding process, a parallel dilated convolution module (PDCM) designed to incorporate prior information, and a multi-feature aggregation module (MAM) in the decoding process. The PBN leverages prior region and edge information derived from superpixels and edge maps to enhance edge detection accuracy during the encoding phase. The PDCM integrates features from both branches and applies dilated convolution across various scales to expand the receptive field and capture a more comprehensive semantic context. During the decoding phase, the MAM utilizes deep semantic information to direct the fusion of features, thereby optimizing segmentation efficacy. Through a sequence of aggregations, the MAM gradually merges deep and shallow semantic information, culminating in a more enriched and holistic feature representation. Extensive experiments are conducted across diverse datasets, such as WHU, Inria Aerial, and Massachusetts, revealing that PDBMFN outperforms other sophisticated methods in terms of segmentation accuracy. In the key segmentation metrics, including mIoU, precision, recall, and F1 score, PDBMFN shows a marked superiority over contemporary techniques. The ablation studies further substantiate the performance improvements conferred by the PBN’s prior information guidance and the efficacy of the PDCM and MAM modules.

The attention mechanism has been widely used and achieved good results in many visual tasks. But the calculations of attention mechanism in vision tasks consume huge spaces and times, which is the obvious disadvantage of this method. In order to alleviate this problem, we use the DWT(Discrete Wavelet Transform) method to reduce the complexity of attention calculation. DWT can transform an N-dimensional vector into two vectors, one is the low-frequency component of N/2 dimension and the other is high-frequency component of N/2 dimension too. We only use the low-frequency to calculate the attention matrixes, which can reduce the complexity of matrix multiplication, then the time and space consumption of the network is reduced significantly. We also find that the building segmentation in the remote sensing image is different from the other scene segmentation, that the sizes and numbers of different classes of the targets in the general scene images are obvious. Despite all this, our method is still applicable for the targets with large numbers and sizes in general scene images, but not for the targets with small sizes and numbers, and this view is also verified by the subsequent experiments on different datasets. We apply our method on three typical networks (Danet, Swin and Segmenter), and carry out comprehensive experiments on the Cityscape dataset and three building segmentation datasets (Inria Aerial Dataset, Massachusetts Buildings Dataset and Chinese Style Architecture Dataset). The experiments show that, our method is more suitable for building segmentation and can reduce the complexity of the model calculation in building segmentation, and the Mean IoU of segmentation results is not reduced clearly, some even improved.