In the contemporary context, pest detection is progressively moving toward automation and intelligence. However, current pest detection algorithms still face challenges, such as lower accuracy and slower operation speed in detecting small objects. To address this issue, this study presents a crop pest target detection algorithm, GLU-YOLOv8, designed for complex scenes based on an enhanced version of You Only Look Once version 8 (YOLOv8). The algorithm introduces the SCYLLA-IOU (SIOU) loss function, which enhances the model generalization to various pest sizes and shapes by ensuring smoothness and reducing oscillations during training. Additionally, the algorithm incorporates the Convolutional Block Attention Module (CBAM) and Locality Sensitive Kernel (LSK) attention mechanisms to boost the pest target features. A novel Gated Linear Unit CONV (GLU-CONV) is also introduced to enhance the model’s perceptual and generalization capabilities while maintaining performance. Furthermore, GLU-YOLOv8 includes a small-object detection layer with a feature map size of 160 × 160 to extract more features of small-target pests, thereby improving detection accuracy and enabling more precise localization and identification of small-target pests. The study conducted a comparative analysis between the GLU-YOLOv8 model and other models, such as YOLOv8, Faster RCNN, and RetinaNet, to evaluate detection accuracy and precision. In the Scolytidae forestry pest dataset, GLU-YOLOv8 demonstrated an improvement of 8.2% in mAP@0.50 for small-target detection compared to the YOLOv8 model, with a resulting mAP@0.50 score of 97.4%. Specifically, on the IP102 dataset, GLU-YOLOv8 outperforms the YOLOv8 model with a 7.1% increase in mAP@0.50 and a 5% increase in mAP@0.50:0.95, reaching 58.7% for mAP@0.50. These findings highlight the significant enhancement in the accuracy and recognition rate of small-target detection achieved by GLU-YOLOv8, along with its efficient operational performance. This research provides valuable insights for optimizing small-target detection models for various pests and diseases.