Mandarin fruit detection provides crucial technical support for yield prediction and the precise identification and harvesting of mandarin fruits. However, challenges such as occlusion from leaves or branches, the presence of small or partially visible fruits, and limitations in model efficiency pose significant obstacles in a complex orchard environment. To tackle these issues, we propose ELD-YOLO, a lightweight detection framework designed to enhance edge detail preservation and improve the detection of small and occluded fruits. Our method incorporates edge-aware processing to strengthen feature representation, introduces a streamlined detection head that balances accuracy with computational cost, and employs an adaptive upsampling strategy to minimize information loss during feature scaling. Experiments on a mandarin fruit dataset show that ELD-YOLO achieves a precision of 89.7%, a recall of 83.7%, an mAP@50 of 92.1%, and an mAP@50:95 of 68.6% while reducing the parameter count by 15.4% compared with the baseline. These results demonstrate that ELD-YOLO provides an effective and efficient solution for fruit detection in complex orchard scenarios.