Recognition and localization of strawberries from 3D binocular cameras for a strawberry picking robot using coupled YOLO/Mask R-CNN

Canopy coverage and plant height are the main crop canopy parameters, which can obviously reflect the growth status of crops on the field. The ability to identify canopy coverage and plant height quickly is critical for farmers or breeders to arrange their working schedule. In precision agriculture, choosing the opportunity and amount of farm inputs is the critical part, which will improve the yield and decrease the cost. The potato canopy coverage and plant height were quickly extracted, which could be used to estimate the spraying volume using the evaluation model obtained by indoor tests. The vegetation index approach was used to extract potato canopy coverage, and the color point cloud data method at different height rates was formed to estimate the plant height of potato at different growth stages. The original data were collected using a low-cost UAV, which was mounted on a high-resolution RGB camera. Then, the Structure from Motion (SFM) algorithm was used to extract the 3D point cloud from ordered images that could form a digital orthophoto model (DOM) and sparse point cloud. The results show that the vegetation index-based method could accurately estimate canopy coverage. Among EXG, EXR, RGBVI, GLI, and CIVE, EXG achieved the best adaptability in different test plots. Point cloud data could be used to estimate plant height, but when the potato coverage rate was low, potato canopy point cloud data underwent rarefaction; in the vigorous growth period, the estimated value was substantially connected with the measured value (R² = 0.94). The relationship between the coverage area of spraying on potato canopy and canopy coverage was measured indoors to form the model. The results revealed that the model could estimate the dose accurately (R² = 0.878). Therefore, combining agronomic factors with data extracted from the UAV RGB image had the ability to predict the field spraying volume.

To improve the accuracy and reliability of orchard spraying robots, an integrated navigation system was developed, consisting of a real-time kinematic positioning-Beidou satellite navigation system (RTK-BDS) receiver, an inertial measurement unit (IMU), a navigation controller, and servo motors. Using the loose coupling combination method, an error Kalman filter algorithm based on the measurement of position and heading angle is implemented to correct the error of the inertial measurement unit in real time. Combining the kinematics model and the pure pursuit model of the spraying robot, a path-tracking control algorithm is proposed. Path planning was conducted according to the terrain characteristics of orchards. Field experiments were carried out on a spraying robot to evaluate the proposed auto-navigation system. The results showed that when the spraying robot was static, the positioning performances of BDS alone and that of the BDS/IMU combined system were similar, the positioning error was less than 1.5 cm, and the heading angle errors were within 0.3°; when the spraying robot moving alone to a straight line at the speed of 0.4 m/s, the position error of the navigation system only using BDS was less than 5.29 cm, the heading angle error was within 3°, while the position error of BDS/IMU integrated navigation system was less than 2.49 cm, and the heading angle error was within 2°. The accuracy of BDS/IMU integrated navigation system is significantly improved. When the orchard spraying robot was moving at the speed of 0.4 m/s, the maximum offset error was lower than 10.77 cm, the average offset error was not higher than 3.55 cm, and the root mean square error (RMSE) of the lateral deviation was 1.19 cm. The results showed that the proposed auto-navigation system could make the spraying robot track the pre-set path smoothly and stably.

In order to detect kiwifruit quickly and accurately in orchard environments for the picking robot, this paper proposed a detection method based on a lightweight YOLOv4-GhostNet network. The implementations of the method are as follows: The original CSP-Darknet53 backbone network model was replaced by GhostNet, a feature layer facilitating small object detection was introduced in the feature fusion layer, and part of the ordinary convolution was replaced by a combination of 1 × 1 convolution and depth-separable convolution to reduce the computational pressure caused by the fused feature layer. The parameters of the new network are reduced, and the generalization ability of the model is improved by loading pre-training weights and freezing some layers. The trained model was tested, and the results showed that the detection performances were better than that of the original YOLOv4 network. The F1 value, map, and precision were improved on the test set, which were 92%, 93.07%, and 90.62%, respectively. The size of weight parameters was reduced to 1/6 of the original YOLOv4 network, and the detection speed reached 53 FPS. Therefore, the method proposed in this study shows the features of fast recognition, lightweight parameters, and high recognition accuracy, which can provide technical support for vision systems of kiwifruit picking robots.

Here, the improved multi‐scale YOLO algorithm (Improved‐YOLOv3) is presented, which was proposed to realize fast and accurate recognition of citrus fruit in a field environment. With the modification of the YOLO‐styled network model, a darknet‐53 backbone network with residual modules was designed. A multi‐scale detection module was to construct a network model for rapid recognition of citrus fruit in complex environments. Using the improved model to detect and identify citrus fruit targets, the network model can extract more feature information. The improved YOLOv3 model was tested with citrus data, and the detection performance of the improved network and the influence of the number of backbone network layers on the feature extraction capability were compared. The results showed a good detection ability (detection rate, accuracy, map, detection speed) for the target fruit, and the improved YOLOv3 network showed higher accuracy. Moreover, the performance of different training models were compared: the Improved‐YOLOv3 has stronger robustness, higher detection accuracy, and shorter training time, and can recognize citrus in complex field environments. The experiment showed that the precision of Improved‐YOLOv3 was 90.5% and the accuracy reached 94.3%, the recall rate was 90.3%, the detection time was 9.89 ms per frame, which could provide technical support for this visual recognition system of citrus picking robot.

Design and performance test of a novel UAV air-assisted electrostatic centrifugal spraying system