This research presents a soft gripper for apple harvesting to provide constant-pressure clamping and avoid fruit damage during slippage, to reduce the potential danger of damage to the apple pericarp during robotic harvesting. First, a three-finger gripper based on the Fin Ray structure is developed, and the influence of varied structure parameters during gripping is discussed accordingly. Second, we develop a mechanical model of the suggested servo-driven soft gripper based on the mappings of gripping force, pulling force, and servo torque. Third, a real-time control strategy for the servo is proposed, to monitor the relative position relationship between the gripper and the fruit by an ultrasonic sensor to avoid damage from the slip between the fruit and fingers. The experimental results show that the proposed soft gripper can non-destructively grasp and separate apples. In outdoor orchard experiments, the damage rate for the grasping experiments of the gripper with the force feedback system turned on was 0%; while the force feedback system was turned off, the damage rate was 20%, averaged for slight and severe damage. The three cases of rigid fingers and soft fingers with or without slip detection under the gripper structure of this study were tested by picking 25 apple samples for each set of experiments. The picking success rate for the rigid fingers was 100% but with a damage rate of 16%; the picking success rate for soft fingers with slip detection was 80%, with no fruit skin damage; in contrast, the picking success rate for soft fingers with slip detection off increased to 96%, and the damage rate was up to 8%. The experimental results demonstrated the effectiveness of the proposed control method.

Precise localization of occluded fruits is crucial and challenging for robotic harvesting in orchards. Occlusions from leaves, branches, and other fruits make the point cloud acquired from Red Green Blue Depth (RGBD) cameras incomplete. Moreover, an insufficient filling rate and noise on depth images of RGBD cameras usually happen in the shade from occlusions, leading to the distortion and fragmentation of the point cloud. These challenges bring difficulties to position locating and size estimation of fruit for robotic harvesting. In this paper, a novel 3D fruit localization method is proposed based on a deep learning segmentation network and a new frustum-based point-cloud-processing method. A one-stage deep learning segmentation network is presented to locate apple fruits on RGB images. With the outputs of masks and 2D bounding boxes, a 3D viewing frustum was constructed to estimate the depth of the fruit center. By the estimation of centroid coordinates, a position and size estimation approach is proposed for partially occluded fruits to determine the approaching pose for robotic grippers. Experiments in orchards were performed, and the results demonstrated the effectiveness of the proposed method. According to 300 testing samples, with the proposed method, the median error and mean error of fruits’ locations can be reduced by 59% and 43%, compared to the conventional method. Furthermore, the approaching direction vectors can be correctly estimated.

Global fruit production costs are increasing amid intensified labor shortages, driving heightened interest in robotic harvesting technologies. Although multi-arm coordination in harvesting robots is considered a highly promising solution to this issue, it introduces technical challenges in achieving effective coordination. These challenges include mutual interference among multi-arm mechanical structures, task allocation across multiple arms, and dynamic operating conditions. This imposes higher demands on task coordination for multi-arm harvesting robots, requiring collision-free collaboration, optimization of task sequences, and dynamic re-planning. In this work, we propose a framework that models the task planning problem of multi-arm operation as a Markov game. First, considering multi-arm cooperative movement and picking sequence optimization, we employ a two-agent Markov game framework to model the multi-arm harvesting robot task planning problem. Second, we introduce a self-attention mechanism and a centralized training and execution strategy in the design and training of our deep reinforcement learning (DRL) model, thereby enhancing the model’s adaptability in dynamic and uncertain environments and improving decision accuracy. Finally, we conduct extensive numerical simulations in static environments; when the harvesting targets are set to 25 and 50, the execution time is reduced by 10.7% and 3.1%, respectively, compared to traditional methods. Additionally, in dynamic environments, both operational efficiency and robustness are superior to traditional approaches. The results underscore the potential of our approach to revolutionize multi-arm harvesting robotics by providing a more adaptive and efficient task planning solution. We will research improving the positioning accuracy of fruits in the future, which will make it possible to apply this framework to real robots.

In view of the continuous increase in labor costs for complex picking tasks, there is an urgent demand for intelligent harvesting robots in the global fresh fruit cultivation industry. Fruit visual information is essential to guide robotic harvesting. However, obtaining accurate visual information about the target is critical in complex agricultural environments. The main challenges include the image color distortion under changeable natural light, occlusions from the interlaced plant organs (stems, leaves, and fruits), and the picking point location on fruits with variable shapes and poses. On top of summarizing the current status of typical fresh fruit harvesting robots, this paper outlined the state-of-the-art advance of visual information acquisition technology, including image acquisition in the natural environment, fruit recognition from the complex backgrounds, target stereo locating and measurement, and fruit search among the plants. It then analyzed existing problems and raised future potential research trends from two aspects, multiple images fusion and self-improving algorithm model.

The accuracy of apple fruit recognition in orchard environments is significantly affected by factors such as occlusion and lighting variations, leading to issues such as missed and false detections. To address these challenges, particularly related to occluded apples, this study proposes an improved apple-detection model, MSOAR-YOLOv10, based on YOLOv10. Firstly, a multi-scale feature fusion network is enhanced by adding a 160 × 160 feature scale layer to the backbone network, which increases the model’s sensitivity to small local features, particularly for occluded fruits. Secondly, the Squeeze-and-Excitation (SE) attention mechanism is integrated into the C2fCIB convolution module of the backbone network to improve the network’s focus on the regions of interest in the input images. Additionally, a Diverse Branch Block (DBB) module is introduced to enhance the performance of the convolutional neural network. Furthermore, a Normalized Wasserstein Distance (NWD) loss function is proposed to effectively reduce missed detections of densely packed and overlapping targets. Experimental results in orchards indicate that the proposed improved YOLOv10 model achieves precision, recall, and mean average precision rates of 89.3%, 89.8%, and 92.8%, respectively, representing increases of 3.1%, 2.2%, and 3.0% compared to the original YOLOv10 model. These results validate that the proposed network significantly enhances apple recognition accuracy in complex orchard environments, particularly improving the operational precision of harvesting robots in real-world conditions.