Visual detection of unmanned aerial vehicles (UAVs) is an essential perception component of urban low-altitude security systems. However, UAV targets in complex urban scenes appear small and exhibit weak visual features. Existing convolutional structures inadequately extract their contours and fine-grained details, and background clutter such as building textures and trees attenuates target feature responses. To address these problems, this paper proposes a lightweight detection model, SA-YOLO (Small-target Adaptive YOLO). A bilateral detail refinement convolution (BDRConv) strengthens target details while preserving primary features through dual-branch collaboration. An adaptive SimAM (ASimAM) attention mechanism modulates attention response strength via a learnable scaling factor. DySample dynamic upsampling achieves sub-pixel-level reconstruction of cross-scale features through content-adaptive sampling offsets. A weighted joint loss based on the normalized Wasserstein distance improves bounding-box regression for small targets. On the DUT-Anti-UAV and the self-constructed C-UAV datasets, SA-YOLO achieves mAP50-95 values of 52.1% and 43.2%, respectively, with only 2.41 M parameters at 171 FPS. Across five training seeds, the mAP50-95 improvement over the baseline is statistically significant on DUT-Anti-UAV, whereas on C-UAV the gain is directionally consistent but not statistically significant. Compared with the accuracy-leading detectors, SA-YOLO delivers comparable accuracy in a more compact model, achieving a favorable accuracy–model-size tradeoff at a real-time inference speed.