Wild animals are an essential component of natural ecosystems, and the accurate identification of wildlife targets plays a critical role in ecological conservation and species monitoring. However, the effectiveness of conventional object detection algorithms is often limited by the challenges posed by complex outdoor environments, small target sizes, and group occlusions. To address these issues, this study constructs a dataset comprising over 8000 images of 10 protected wildlife species and investigates effective detection methods for wildlife in natural habitats. We propose a novel deep learning-based detection framework, YOLO-WildASM, which incorporates three key improvements to the YOLOv8 architecture: a P2 detection layer for small objects, a multi-head self-attention (MHSA) mechanism, and a bidirectional feature pyramid network (BiFPN). Experimental results demonstrate that YOLO-WildASM significantly outperforms YOLOv8 and other state-of-the-art models on the custom wildlife dataset, achieving a mAP50 of 94.1%, which is 2.8% higher than the baseline model and superior to the latest YOLOv12 model (92.2%). Furthermore, ablation and generalization experiments validate the model’s enhanced performance and adaptability in multi-scale wildlife detection tasks. The proposed deep learning-based detection framework provides an efficient and robust solution for wildlife monitoring and ecological conservation in complex natural ecosystems.