The widespread adoption of small unmanned aerial vehicles poses increasing challenges to public safety. Compared with visible-light sensors, infrared imaging offers excellent nighttime observation capabilities and strong robustness against interference, enabling all-weather UAV surveillance. However, detecting small UAVs in infrared imagery remains challenging due to low target contrast and weak texture features. To address these challenges, we propose IUAV-YOLO, a context-aware detection framework built upon YOLOv10. Specifically, inspired by the receptive field mechanism in human vision, the backbone network is re-designed with a multi-branch structure to improve sensitivity to small targets. Additionally, a Pyramid Global Attention Module is incorporated to strengthen target–background associations, while a Spatial Context-Aware Module is developed to integrate spatial contextual cues and enhance target-background discrimination. Extensive experiments demonstrate that, compared with the baseline model, IUAV-YOLO achieves performance gains of 4.3% in AP0.5 and 2.6% in AP0.5–0.95 on the self-built IRSUAV dataset, with a reduction of 0.7M parameters. On the public SIRST-UAVB dataset, IUAV-YOLO attains improvements of 29.7% in AP0.5 and 16.3% in AP0.5–0.95. Compared with other advanced object detection algorithms, IUAV-YOLO demonstrates a superior accuracy-efficiency trade-off, highlighting its potential for practical infrared UAV surveillance applications.