Addressing the formidable challenges in spatial infrared dim target detection, this paper introduces an advanced detection approach based on the refinement of the YOLOv8 algorithm. In contrast to the conventional YOLOv8, our method achieves remarkable improvements in detection accuracy through several novel strategies. Notably, by incorporating a deformable convolutional module into the YOLOv8 backbone network, our method effectively captures more intricate image features, laying a solid foundation for subsequent feature fusion and detection head predictions. Furthermore, a dedicated small target detection layer, built upon the original model, significantly enhances the model’s capability in recognizing infrared small targets, thereby boosting overall detection performance. Additionally, we utilize the WIoU-v3 as the localization regression loss function, effectively reducing sensitivity to positional errors and leveraging the advantages of multi-attention mechanisms. To enrich the quantity and quality of the spatial infrared dim target dataset, we employ image enhancement techniques to augment the original dataset. Extensive experiments demonstrate the exceptional performance of our method. Specifically, our approach achieves a precision of 95.6%, a recall rate of 94.7%, and a mean average precision (mAP) exceeding 97.4%, representing substantial improvements over the traditional YOLOv8 algorithm. Moreover, our detection speed reaches 59 frames/s, satisfying the requirements for real-time detection. This achievement not only validates the efficacy and superiority of our algorithm in spatial infrared dim target detection, but also offers novel insights and methodologies for research and applications in related fields, holding immense potential for future applications.