The existing methods of building change detection (CD) using remote sensing images are still deficient in handling scale variation and class imbalance problems, indicating a decrease in the robustness of small-object detection and pseudo-change information. Thus, a novel building CD framework called the multiscale context aggregation network (MSCANet) is proposed. The high resolution network is integrated into the feature extracting stage to maintain high resolution representations through the whole process. Then, multiscale context information is aggregated using a scale-aware feature pyramid module. Recognition performance can be improved from discriminant feature representation learning by using a channel-spatial attention module. Furthermore, a class-balanced loss is proposed to reduce the impact of class imbalance in long-tail datasets. Experimental results from using the LEVIR-CD and SZTAKI AirChange benchmark datasets prove the superiority of the MSCANet over the other baseline methods, with improved maximum F1 scores of 5.28 and 8.47, respectively.