Conveyor belt deviation is one of the most common failures in belt conveyors. To address issues such as the high computational complexity, large number of parameters, long inference time, and difficulty in feature extraction of existing conveyor belt deviation detection models, we propose a GES-YOLO algorithm for detecting deviation in mining belt conveyors, based on an improved YOLOv8s model. The core of this algorithm is to enhance the model’s ability to extract features in complex scenarios, thereby improving the detection efficiency. Specifically, to improve real-time detection capabilities, we introduce the Groupwise Separable Convolution (GSConv) module. Additionally, by analyzing scene features, we remove the large object detection layer, which enhances the detection speed while maintaining the feature extraction capability. Furthermore, to strengthen feature perception under low-light conditions, we introduce the Efficient Multi-Scale Attention Mechanism (EMA), allowing the model to obtain more robust features. Finally, to improve the detection capability for small objects such as conveyor rollers, we introduce the Scaled Intersection over Union (SIoU) loss function, enabling the algorithm to sensitively detect rollers and provide a precise localization for deviation detection. The experimental results show that the GES-YOLO significantly improves the detection performance in complex environments such as high-noise and low-illumination conditions in coal mines. Compared to the baseline YOLOv8s model, GES-YOLO’s mAP@0.5 and mAP@0.5:0.95 increase by 1.5% and 2.3%, respectively, while the model’s parameter count and computational complexity decrease by 38.2% and 10.5%, respectively. The Frames Per Second (FPS) of the average detection speed reaches 63.62. This demonstrates that GES-YOLO achieves a good balance between detection accuracy and inference speed, with excellent accuracy, robustness, and industrial application potential.