Apple orchards, as an important center of economic activity in forestry special crops, can achieve yield prediction and automated harvesting by detecting and locating apples. Small apples, occlusion, dim lighting at night, blurriness, cluttered backgrounds, and other complex scenes significantly affect the automatic harvesting and yield estimation of apples. To address these issues, this study proposes an apple detection algorithm, “YOLOv5-ACS (Apple in Complex Scenes)”, based on YOLOv5s. Firstly, the space-to-depth-conv module is introduced to avoid information loss, and a squeeze-and-excitation block is added in C3 to learn more important information. Secondly, the context augmentation module is incorporated to enrich the context information of the feature pyramid network. By combining the shallow features of the backbone P2, the low-level features of the object are retained. Finally, the addition of the context aggregation block and CoordConv aggregates the spatial context pixel by pixel, perceives the spatial information of the feature map, and enhances the semantic information and global perceptual ability of the object. We conducted comparative tests in various complex scenarios and validated the robustness of YOLOv5-ACS. The method achieved 98.3% and 74.3% for mAP@0.5 and mAP@0.5:0.95, respectively, demonstrating excellent detection capabilities. This paper creates a complex scene dataset of apples on trees and designs an improved model, which can provide accurate recognition and positioning for automatic harvesting robots to improve production efficiency.