This paper introduces the ABMS-YOLO algorithm to tackle challenges in small-object detection within UAV aerial photography, such as significant background interference, diverse object sizes, and limited edge computing capacity. Using YOLO11n as a foundation, the algorithm integrates background suppression, dynamic multi-scale feature enhancement, and lightweight end-to-end collaboration. The ABMS-YOLO algorithm consists of three main modules: the Background Suppression Module (BSM), which reduces background noise while maintaining small-object details through channel reorganization and spatial pyramid pooling; the Multi-Scale Feature Module (MSFM), which employs scale-aware dynamic dilated convolutions and adaptive receptive field adjustments to enhance multi-scale feature representation; and the Streamlined Lightweight Collaboration Module (SLCM), which achieves efficient processing via parameter-free spatial rearrangement and parameter-based channel filtering. Experimental results on datasets such as VisDrone2019, TinyPerson, and DOTAv1.0 reveal that ABMS-YOLO, with only 3.04 million parameters—a 17.8% increase over the baseline—improves mAP50 by 12.9%, 6.6%, and 5.8% to 45.1%, 19%, and 44.3%, respectively, while maintaining an FPS of 231.38. The algorithm’s overall detection performance surpasses both contemporary mainstream and representative algorithms, fulfilling the demands for real-time edge detection.