Building extraction is a basic task in the field of remote sensing, and it has also been a popular research topic in the past decade. However, the shape of the semantic polygon generated by semantic segmentation is irregular and does not match the actual building boundary. The boundary of buildings generated by semantic edge detection has difficulty ensuring continuity and integrity. Due to the aforementioned problems, we cannot directly apply the results in many drawing tasks and engineering applications. In this paper, we propose a novel convolutional neural network (CNN) model based on multitask learning, Dense D-LinkNet (DDLNet), which adopts full-scale skip connections and edge guidance module to ensure the effective combination of low-level information and high-level information. DDLNet has good adaptability to both semantic segmentation tasks and edge detection tasks. Moreover, we propose a universal postprocessing method that integrates semantic edges and semantic polygons. It can solve the aforementioned problems and more accurately locate buildings, especially building boundaries. The experimental results show that DDLNet achieves great improvements compared with other edge detection and semantic segmentation networks. Our postprocessing method is effective and universal.

Deep convolutional neural network (DCNN)-based methods have shown great improvements in building extraction from high spatial resolution remote sensing images. In this paper, we propose a postprocessing method based on DCNNs for building extraction. Specifically, building regions and boundaries are learned simultaneously or separately by DCNNs. The predicted building regions and boundaries are then combined by the postprocessing method to produce the final building regions. In addition, we introduce a manually labeled dataset based on high spatial resolution images for building detection, the XIHU building dataset. This dataset is then used in the experiments to evaluate our methods. Besides the WHU building dataset, East Asia (WHUEA) is also included. Results demonstrate that our method that combines the results of DeepLab and BDCN shows the best performance on the XIHU building dataset, which achieves 0.78% and 23.30% F1 scores, and 1.13% and 28.45% intersection-over-union (IoU) improvements compared with DeepLab and BDCN, respectively. Additionally, our method that combines the results of Mask R-CNN and DexiNed performs best on the WHUEA dataset. Moreover, our methods outperform the state-of-the-art multitask learning network, PMNet, on both XIHU and WHUEA datasets, which indicates that the overall performance can be improved although building regions and boundaries are learned in the training stage.