Remote sensing target detection faces significant challenges in handling multi-scale targets, with the high similarity in color and shape between targets and backgrounds in complex scenes further complicating the detection task. To address this challenge, we propose a multi-Scale and complex Background Aware network for remote sensing target detection, named YOLO-SBA. Our proposed YOLO-SBA first processes the input through the Multi-Branch Attention Feature Fusion Module (MBAFF) to extract global contextual dependencies and local detail features. It then integrates these features using the Bilateral Attention Feature Mixer (BAFM) for efficient fusion, enhancing the saliency of multi-scale target features to tackle target scale variations. Next, we utilize the Gated Multi-scale Attention Pyramid (GMAP) to perform channel–spatial dual reconstruction and gating fusion encoding on multi-scale feature maps. This enhances target features while finely suppressing spectral redundancy. Additionally, to prevent the loss of effective information extracted by key modules during inference, we improve the downsampling method using Asymmetric Dynamic Downsampling (ADDown), maximizing the retention of image detail information. We achieve the best performance on the DIOR, DOTA, and RSOD datasets. On the DIOR dataset, YOLO-SBA improves mAP by 16.6% and single-category detection AP by 0.8–23.8% compared to the existing state-of-the-art algorithm.