Robotic harvesting has become an urgent need for the development of the apple industry, due to the sharp decline in agricultural labor. At present, harvesting apples using robots in unstructured orchard environments remains a significant challenge. This paper focuses on addressing the challenges of perception, localization, and dual-arm coordination in harvesting robots and presents a dual-arm apple harvesting robot system. First, the paper introduces the integration of the robot’s hardware and software systems, as well as the control system architecture, and describes the robot’s workflow. Secondly, combining a dual-vision perception system, the paper adopts a fruit recognition method based on a multi-task network model and a frustum-based fruit localization approach to identify and localize fruits. Finally, to improve collaboration efficiency, a multi-arm task planning method based on a genetic algorithm is used to optimize the target harvesting sequence for each arm. Field experiments were conducted in an orchard to evaluate the overall performance of the robot system. The field trials demonstrated that the robot system achieved an overall harvest success rate of 76.97%, with an average fruit picking time of 7.29 s per fruit and a fruit damage rate of only 5.56%.

This study addresses the problem of detecting occluded apples in complex unstructured environments in orchards and proposes an apple detection and segmentation model based on improved YOLOv8n-SGW-YOLOv8n. The model improves apple detection and segmentation by combining the SPD-Conv convolution module, the GAM global attention mechanism, and the Wise-IoU loss function, which enhances the accuracy and robustness. The SPD-Conv module preserves fine-grained features in the image by converting spatial information into channel information, which is particularly suitable for small target detection. The GAM global attention mechanism enhances the recognition of occluded targets by strengthening the feature representation of channel and spatial dimensions. The Wise-IoU loss function further optimises the regression accuracy of the target frame. Finally, the pre-prepared dataset is used for model training and validation. The results show that the SGW-YOLOv8n model significantly improves relative to the original YOLOv8n in target detection and instance segmentation tasks, especially in occlusion scenes. The model improves the detection mAP to 75.9% and the segmentation mAP to 75.7% and maintains a processing speed of 44.37 FPS, which can meet the real-time requirements, providing effective technical support for the detection and segmentation of fruits in complex unstructured environments for fruit harvesting robots.

Accurate fruit target identification is crucial for autonomous harvesting robots in complex orchards, where image segmentation using deep learning networks plays a key role. To address the trade-off between segmentation accuracy and inference efficiency, this study proposes LGVM-YOLOv8n, a lightweight instance segmentation model based on YOLOv8n-seg. LGVM is an acronym for lightweight, GSConv, VoVGSCSP, and $MPDIoU$, highlighting the key improvements incorporated into the model. The proposed model integrates three key improvements: (1) the GSConv module, which enhances feature interaction and reduces computational cost; (2) the VoVGSCSP module, which optimizes multi-scale feature representation for small objects; and (3) the $MPDIoU$ loss function, which improves target localization accuracy, particularly for occluded fruits. Experimental results show that LGVM-YOLOv8n reduces computational cost by 9.17%, decreases model weight by 7.89%, and improves inference speed by 16.9% compared to the original YOLOv8n-seg. Additionally, segmentation accuracy under challenging conditions (front-light, back-light, and occlusion) improves by 3.28% to 4.31%. Deployment tests on an edge computing platform demonstrate real-time performance, with inference speed accelerated to 0.084 s per image and frame rate increased to 28.73 FPS. These results validated the model’s robustness and adaptability, providing a practical solution for apple-picking robots in complex orchard environments.