Litchi is a highly favored fruit with high economic value. Mechanical automation of litchi picking is a key link for improving the quality and efficiency of litchi harvesting. Our research team has been conducting experiments to develop a visual-based litchi picking robot. However, in the early physical prototype experiments, we found that, although picking points were successfully located, litchi picking failed due to random obstructions of the picking points. In this study, the physical prototype of the litchi picking robot previously developed by our research team was upgraded by integrating a visual system for actively removing obstructions. A framework for an artificial intelligence algorithm was proposed for a robot vision system to locate picking points and to identify obstruction situations at picking points. An intelligent control algorithm was developed to control the obstruction removal device to implement obstruction removal operations by combining with the obstruction situation at the picking point. Based on the spatial redundancy of a picking point and the obstruction, the feeding posture of the robot was determined. The experiment showed that the precision of segmenting litchi fruits and branches was 88.1%, the recognition success rate of picking point recognition was 88%, the average error of picking point localization was 2.8511 mm, and an overall success rate of end-effector feeding was 81.3%. These results showed that the developed litchi picking robot could effectively implement obstruction removal.

Due to the short fruit axis, many leaves, and complex background of grapes, most grape cluster axes are blocked from view, which increases robot positioning difficulty in harvesting. This study discussed the location method for picking points in the case of partial occlusion and proposed a grape cluster-detection algorithm “You Only Look Once v5-GAP” based on “You Only Look Once v5”. First, the Conv layer of the first layer of the YOLOv5 algorithm Backbone was changed to the Focus layer, then a convolution attention operation was performed on the first three C3 structures, the C3 structure layer was changed, and the Transformer in the Bottleneck module of the last layer of the C3 structure was used to reduce the computational amount and execute a better extraction of global feature information. Second, on the basis of bidirectional feature fusion, jump links were added and variable weights were used to strengthen the fusion of feature information for different resolutions. Then, the adaptive activation function was used to learn and decide whether neurons needed to be activated, such that the dynamic control of the network nonlinear degree was realized. Finally, the combination of a digital image processing algorithm and mathematical geometry was used to segment grape bunches identified by YOLOv5-GAP, and picking points were determined after finding centroid coordinates. Experimental results showed that the average precision of YOLOv5-GAP was 95.13%, which was 16.13%, 4.34%, and 2.35% higher than YOLOv4, YOLOv5, and YOLOv7 algorithms, respectively. The average positioning pixel error of the point was 6.3 pixels, which verified that the algorithm effectively detected grapes quickly and accurately.

At present, fruit picking mainly relies on manual operation. Taking the litchi (litchi chinensis Sonn.)-picking robot as an example, visual perception is often affected by illumination variations, low recognition accuracy, complex maturity judgment, and occlusion, which lead to inaccurate fruit localization. This study aims to establish an embodied perception mechanism based on “perception-reasoning-execution” to enhance the visual perception and decision-making capability of the robot in complex orchard environments. First, a Y-LitchiC instance segmentation method is proposed to achieve high-precision segmentation of litchi clusters. Second, a generative artificial intelligence model is introduced to intelligently assess fruit maturity and occlusion, providing auxiliary support for automatic picking. Based on the auxiliary judgments provided by the generative AI model, two types of dynamic harvesting decisions are formulated for subsequent operations. For unoccluded main fruit-bearing branches, a skeleton thinning algorithm is applied within the segmented region to extract the skeleton line, and the midpoint of the skeleton is used to perform the first type of localization and harvesting decision. In contrast, for main fruit-bearing branches occluded by leaves, threshold-based segmentation combined with maximum connected component extraction is employed to obtain the target region, followed by skeleton thinning, thereby completing the second type of dynamic picking decision. Experimental results show that the Y-LitchiC model improves the mean average precision (mAP) by 1.6% compared with the YOLOv11s-seg model, achieving higher accuracy in litchi cluster segmentation and recognition. The generative artificial intelligence model provides higher-level reasoning and decision-making capabilities for automatic picking. Overall, the proposed embodied perception mechanism and dynamic picking strategies effectively enhance the autonomous perception and decision-making of the picking robot in complex orchard environments, providing a reliable theoretical basis and technical support for accurate fruit localization and precision picking.