Automatic building extraction plays an important role in various remote sensing applications, such as seismic disaster investigation, seismic hazard risk assessment, urban planning, and photogrammetry. Despite the substantial progress, state-of-the-art building extraction methods are still limited by two issues: (i) existing approaches leverage convolutional layers or non-local self-attention to encode the position-aware dependencies, while they cannot flexibly adapt to the complex background contexts and varied structure patterns of buildings; and (ii) the local details cannot be well preserved by existing hierarchical decoders due to the imperfect feature aggregation, yielding unsatisfactory segmentation outputs in the local adjacent region. To address these issues, we propose Relative Position Aggregated Transformer (RPAFormer), which is capable of modeling the relative position dependencies of buildings and producing accurate local details using a dual attention transformer network. Specifically, we propose a Relative Position-aware Self-attention (RPSA) framework to learn the token dependencies within the local window. A transformer decoder network consisting of multiple Cross Masked Attention (CMA) blocks is also introduced to fuse the multi-scale features. Extensive experiments demonstrate the superior performance of the proposed method and its great promise for real-world engineering deployment.