This study introduces a novel high-accuracy fruit fly detection model based on the Transformer structure, specifically aimed at addressing the unique challenges in fruit fly detection such as identification of small targets and accurate localization against complex backgrounds. By integrating a step attention mechanism and a cross-loss function, this model significantly enhances the recognition and localization of fruit flies within complex backgrounds, particularly improving the model’s effectiveness in handling small-sized targets and its adaptability under varying environmental conditions. Experimental results demonstrate that the model achieves a precision of 0.96, a recall rate of 0.95, an accuracy of 0.95, and an F1-score of 0.95 on the fruit fly detection task, significantly outperforming leading object detection models such as YOLOv8 and DETR. Specifically, this research delves into and optimizes for challenges faced in fruit fly detection, such as recognition issues under significant light variation, small target size, and complex backgrounds. Through ablation experiments comparing different data augmentation techniques and model configurations, the critical contributions of the step attention mechanism and cross-loss function to enhancing model performance under these complex conditions are further validated. These achievements not only highlight the innovativeness and effectiveness of the proposed method, but also provide robust technical support for solving practical fruit fly detection problems in real-world applications, paving new paths for future research in object detection technology.