Accurate recognition of fruits in the orchard is an important step for robot picking in the natural environment, since many CNN models have a low recognition rate when dealing with irregularly shaped and very dense fruits, such as a grape bunch. It is a new trend to use a transformer structure and apply it to a computer vision domain for image processing. This paper provides Swin Transformer and DETR models to achieve grape bunch detection. Additionally, they are compared with traditional CNN models, such as Faster-RCNN, SSD, and YOLO. In addition, the optimal number of stages for a Swin Transformer through experiments is selected. Furthermore, the latest YOLOX model is also used to make a comparison with the Swin Transformer, and the experimental results show that YOLOX has higher accuracy and better detection effect. The above models are trained under red grape datasets collected under natural light. In addition, the dataset is expanded through image data augmentation to achieve a better training effect. After 200 epochs of training, SwinGD obtained an exciting mAP value of 94% when IoU = 0.5. In case of overexposure, overdarkness, and occlusion, SwinGD can recognize more accurately and robustly compared with other models. At the same time, SwinGD still has a better effect when dealing with dense grape bunches. Furthermore, 100 pictures of grapes containing 655 grape bunches are downloaded from Baidu pictures to detect the effect. The Swin Transformer has an accuracy of 91.5%. In order to verify the universality of SwinGD, we conducted a test under green grape images. The experimental results show that SwinGD has a good effect in practical application. The success of SwinGD provides a new solution for precision harvesting in agriculture.

Reliable and robust fruit-detection algorithms in nonstructural environments are essential for the efficient use of harvesting robots. The pose of fruits is crucial to guide robots to approach target fruits for collision-free picking. To achieve accurate picking, this study investigates an approach to detect fruit and estimate its pose. First, the state-of-the-art mask region convolutional neural network (Mask R-CNN) is deployed to segment binocular images to output the mask image of the target fruit. Next, a grape point cloud extracted from the images was filtered and denoised to obtain an accurate grape point cloud. Finally, the accurate grape point cloud was used with the RANSAC algorithm for grape cylinder model fitting, and the axis of the cylinder model was used to estimate the pose of the grape. A dataset was acquired in a vineyard to evaluate the performance of the proposed approach in a nonstructural environment. The fruit detection results of 210 test images show that the average precision, recall, and intersection over union (IOU) are 89.53, 95.33, and 82.00%, respectively. The detection and point cloud segmentation for each grape took approximately 1.7 s. The demonstrated performance of the developed method indicates that it can be applied to grape-harvesting robots.

Collision‐free autonomous path planning under a dynamic and uncertainty vineyard environment is the most important issue which needs to be resolved firstly in the process of improving robotic harvesting manipulator intelligence. We present and apply energy optimal and artificial potential field to develop a path planning method for six degree of freedom (DOF) serial harvesting robot under dynamic uncertain environment. Firstly, the kinematical model of Six‐DOF serial manipulator was constructed by using the Denavit‐Hartenberg (D‐H) method. The model of obstacles was defined by axis‐aligned bounding box, and then the configuration space of harvesting robot was described by combining the obstacles and arm space of robot. Secondly, the harvesting sequence in path planning was computed by energy optimal method, and the anticollision path points were automatically generated based on the artificial potential field and sampling searching method. Finally, to verify and test the proposed path planning algorithm, a virtual test system based on virtual reality was developed. After obtaining the space coordinates of grape picking point and anticollision bounding volume, the path points were drew out by the proposed method. 10 times picking tests for grape anticollision path planning were implemented on the developed simulation system, and the success rate was up to 90%. The results showed that the proposed path planning method can be used to the harvesting robot.

Counting grape berries and measuring their size can provide accurate data for robot picking behavior decision-making, yield estimation, and quality evaluation. When grapes are picked, there is a strong uncertainty in the external environment and the shape of the grapes. Counting grape berries and measuring berry size are challenging tasks. Computer vision has made a huge breakthrough in this field. Although the detection method of grape berries based on 3D point cloud information relies on scanning equipment to estimate the number and yield of grape berries, the detection method is difficult to generalize. Grape berry detection based on 2D images is an effective method to solve this problem. However, it is difficult for traditional algorithms to accurately measure the berry size and other parameters, and there is still the problem of the low robustness of berry counting. In response to the above problems, we propose a grape berry detection method based on edge image processing and geometric morphology. The edge contour search and the corner detection algorithm are introduced to detect the concave point position of the berry edge contour extracted by the Canny algorithm to obtain the best contour segment. To correctly obtain the edge contour information of each berry and reduce the error grouping of contour segments, this paper proposes an algorithm for combining contour segments based on clustering search strategy and rotation direction determination, which realizes the correct reorganization of the segmented contour segments, to achieve an accurate calculation of the number of berries and an accurate measurement of their size. The experimental results prove that our proposed method has an average accuracy of 87.76% for the detection of the concave points of the edge contours of different types of grapes, which can achieve a good edge contour segmentation. The average accuracy of the detection of the number of grapes berries in this paper is 91.42%, which is 4.75% higher than that of the Hough transform. The average error between the measured berry size and the actual berry size is 2.30 mm, and the maximum error is 5.62 mm, which is within a reasonable range. The results prove that the method proposed in this paper is robust enough to detect different types of grape berries.

Replacing humans with robots for fruit harvesting is the trend of agricultural automation in the future. However, for grape harvesting robots, locating the picking point becomes a significant challenge in highly occluded environments due to the small fruit stem, which can be entirely obscured by fruit leaves when the observation angle is poor. In the letter, a view planner based on an active vision strategy is proposed to address the occlusion problem. It aims to find the picking point by altering the observation perspective of the harvesting robot. The view planning process is achieved through multiple iterations. Each iteration consists of three key steps: randomly generating candidate views, predicting the ideal perspective using a score function, and guiding the robotic arm to change the viewpoint. To evaluate the degree of occlusion, a novel concept of Spatial Coverage Rate Metric (SC) is introduced. Based on this, the score function is improved by incorporating SC and motion cost. Finally, to validate the effectiveness of the planner, we conducted comparative experiments with other advanced view planners on a real grape harvesting robot. The experimental results demonstrate that our method achieves a higher picking success rate with lower computation time.