The utilization of machine vision and its associated algorithms improves the efficiency, functionality, intelligence, and remote interactivity of harvesting robots in complex agricultural environments. Machine vision and its associated emerging technology promise huge potential in advanced agricultural applications. However, machine vision and its precise positioning still have many technical difficulties, making it difficult for most harvesting robots to achieve true commercial applications. This article reports the application and research progress of harvesting robots and vision technology in fruit picking. The potential applications of vision and quantitative methods of localization, target recognition, 3D reconstruction, and fault tolerance of complex agricultural environment are focused, and fault-tolerant technology designed for utilization with machine vision and robotic systems are also explored. The two main methods used in fruit recognition and localization are reviewed, including digital image processing technology and deep learning-based algorithms. The future challenges brought about by recognition and localization success rates are identified: target recognition in the presence of illumination changes and occlusion environments; target tracking in dynamic interference-laden environments, 3D target reconstruction, and fault tolerance of the vision system for agricultural robots. In the end, several open research problems specific to recognition and localization applications for fruit harvesting robots are mentioned, and the latest development and future development trends of machine vision are described.

To accurately recognize plant diseases under complex natural conditions, an improved plant disease-recognition model based on the original YOLOv5 network model was established. First, a new InvolutionBottleneck module was used to reduce the numbers of parameters and calculations, and to capture long-distance information in the space. Second, an SE module was added to improve the sensitivity of the model to channel features. Finally, the loss function ‘Generalized Intersection over Union’ was changed to ‘Efficient Intersection over Union’ to address the former’s degeneration into ‘Intersection over Union’. These proposed methods were used to improve the target recognition effect of the network model. In the experimental phase, to verify the effectiveness of the model, sample images were randomly selected from the constructed rubber tree disease database to form training and test sets. The test results showed that the mean average precision of the improved YOLOv5 network reached 70%, which is 5.4% higher than that of the original YOLOv5 network. The precision values of this model for powdery mildew and anthracnose detection were 86.5% and 86.8%, respectively. The overall detection performance of the improved YOLOv5 network was significantly better compared with those of the original YOLOv5 and the YOLOX_nano network models. The improved model accurately identified plant diseases under natural conditions, and it provides a technical reference for the prevention and control of plant diseases.

Fruit detection in real outdoor conditions is necessary for automatic guava harvesting, and the branch-dependent pose of fruits is also crucial to guide a robot to approach and detach the target fruit without colliding with its mother branch. To conduct automatic, collision-free picking, this study investigates a fruit detection and pose estimation method by using a low-cost red–green–blue–depth (RGB-D) sensor. A state-of-the-art fully convolutional network is first deployed to segment the RGB image to output a fruit and branch binary map. Based on the fruit binary map and RGB-D depth image, Euclidean clustering is then applied to group the point cloud into a set of individual fruits. Next, a multiple three-dimensional (3D) line-segments detection method is developed to reconstruct the segmented branches. Finally, the 3D pose of the fruit is estimated using its center position and nearest branch information. A dataset was acquired in an outdoor orchard to evaluate the performance of the proposed method. Quantitative experiments showed that the precision and recall of guava fruit detection were 0.983 and 0.948, respectively; the 3D pose error was 23.43° ± 14.18°; and the execution time per fruit was 0.565 s. The results demonstrate that the developed method can be applied to a guava-harvesting robot.