The precise extraction of building footprints using remote sensing technology is increasingly critical for urban planning and development amid growing urbanization. However, considering the complexity of building backgrounds, diverse scales, and varied appearances, accurately and efficiently extracting building footprints from various remote sensing images remains a significant challenge. In this paper, we propose a novel network architecture called ME-FCN, specifically designed to perceive and optimize multi-scale features to effectively address the challenge of extracting building footprints from complex remote sensing images. We introduce a Squeeze-and-Excitation U-Block (SEUB), which cascades multi-scale semantic information exploration in shallow feature maps and incorporates channel attention to optimize features. In the network’s deeper layers, we implement an Adaptive Multi-scale feature Enhancement Block (AMEB), which captures large receptive field information through concatenated atrous convolutions. Additionally, we develop a novel Dual Multi-scale Attention (DMSA) mechanism to further enhance the accuracy of cascaded features. DMSA captures multi-scale semantic features across both channel and spatial dimensions, suppresses redundant information, and realizes multi-scale feature interaction and fusion, thereby improving the overall accuracy and efficiency. Comprehensive experiments on three datasets demonstrate that ME-FCN outperforms mainstream segmentation methods.