The automatic extraction of buildings from remote sensing images is crucial for various applications such as urban planning and management, emergency response, and map making and updating. In recent years, deep learning (DL) methods have made significant progress in this field. However, due to the complex and diverse structures of buildings and their interconnections, the accuracy of extracted buildings remains insufficient for high-precision applications such as maps and navigation. To address the issue of enhancing building boundary extraction, we propose a modified instance segmentation model, AP-PointRend (Adaptive Parameter-PointRend), to improve the performance of building instance extraction. Specifically, the model can adaptively select the number of iterations and points based on the size of buildings to improve the segmentation accuracy of large buildings. By introducing regularization constraints, discrete small patches are removed, preserving boundaries better during the segmentation process. We also designed an image merging method to eliminate seams, ensure the recall rate, and improve the extraction accuracy. The Vaihingen and WHU benchmark datasets were used to evaluate the performance of the AP-PointRend method. The experimental results showed that the proposed AP-PointRend approach generated better building extraction results compared with other state-of-the-art methods.