Using deep learning-based tools in the field of agriculture for the automatic detection of plant leaf diseases has been in place for many years. However, optimizing their use in the specific background of the agriculture field, in the presence of other leaves and the soil, is still an open challenge. This work presents a deep learning model based on YOLOv6s that incorporates (1) Gaussian error linear unit in the backbone, (2) efficient channel attention in the basic RepBlock, and (3) SCYLLA-Intersection Over Union (SIOU) loss function to improve the detection accuracy of the base model in real-field background conditions. Experiments were carried out on a self-collected dataset containing 3305 real-field images of cotton, wheat, and mango (healthy and diseased) leaves. The results show that the proposed model outperformed many state-of-the-art and recent models, including the base YOLOv6s, in terms of detection accuracy. It was also found that this improvement was achieved without any significant increase in the computational cost. Hence, the proposed model stood out as an effective technique to detect plant leaf diseases in real-field conditions without any increased computational burden.