With the rapid rising of global population, the demand for improving breeding techniques to greatly increase the worldwide crop production has become more and more urgent. Most researchers believe that the key to new breeding techniques lies in genetic improvement of crops, which leads to a large quantity of phenotyping spots. Unfortunately, current phenotyping solutions are not powerful enough to handle so many spots with satisfying speed and accuracy. As a result, high-throughput phenotyping is drawing more and more attention. In this paper, we propose a new field-based sensing solution to high-throughput phenotyping. We mount a LiDAR (Velodyne HDL64-S3) on a mobile robot, making the robot a “phenomobile.” We develop software for data collection and analysis under Robotic Operating System using open source components and algorithm libraries. Different from conducting phenotyping observations with an in-row and one-by-one manner, our new solution allows the robot to move around the parcel to collect data. Thus, the 3D and 360° view laser scanner can collect phenotyping data for a large plant group at the same time, instead of one by one. Furthermore, no touching interference from the robot would be imposed onto the crops. We conduct experiments for maize plant on two parcels. We implement point cloud merging with landmarks and Iterative Closest Points to cut down the time consumption. We then recognize and compute the morphological phenotyping parameters (row spacing and plant height) of maize plant using depth-band histograms and horizontal point density. We analyze the cloud registration and merging performances, the row spacing detection accuracy, and the single plant height computation accuracy. Experimental results verify the feasibility of the proposed solution.

Precise localization of occluded fruits is crucial and challenging for robotic harvesting in orchards. Occlusions from leaves, branches, and other fruits make the point cloud acquired from Red Green Blue Depth (RGBD) cameras incomplete. Moreover, an insufficient filling rate and noise on depth images of RGBD cameras usually happen in the shade from occlusions, leading to the distortion and fragmentation of the point cloud. These challenges bring difficulties to position locating and size estimation of fruit for robotic harvesting. In this paper, a novel 3D fruit localization method is proposed based on a deep learning segmentation network and a new frustum-based point-cloud-processing method. A one-stage deep learning segmentation network is presented to locate apple fruits on RGB images. With the outputs of masks and 2D bounding boxes, a 3D viewing frustum was constructed to estimate the depth of the fruit center. By the estimation of centroid coordinates, a position and size estimation approach is proposed for partially occluded fruits to determine the approaching pose for robotic grippers. Experiments in orchards were performed, and the results demonstrated the effectiveness of the proposed method. According to 300 testing samples, with the proposed method, the median error and mean error of fruits’ locations can be reduced by 59% and 43%, compared to the conventional method. Furthermore, the approaching direction vectors can be correctly estimated.

Orchard machinery autonomous navigation is helpful for improving the efficiency of fruit production and reducing labor costs. Path planning is one of the core technologies of autonomous navigation for orchard machinery. As normally planted in straight and parallel rows, fruit trees are natural landmarks that can provide suitable cues for orchard intelligent machinery. This paper presents a novel method to realize path planning based on computer vision technologies. We combine deep learning and the least-square (DL-LS) algorithm to carry out a new navigation line extraction algorithm for orchard scenarios. First, a large number of actual orchard images are collected and processed for training the YOLO V3 model. After the training, the mean average precision (MAP) of the model for trunk and tree detection can reach 92.11%. Secondly, the reference point coordinates of the fruit trees are calculated with the coordinates of the bounding box of trunks. Thirdly, the reference lines of fruit trees growing on both sides are fitted by the least-square method and the navigation line for the orchard machinery is determined by the two reference lines. Experimental results show that the trained YOLO V3 network can identify the tree trunk and the fruit tree accurately and that the new navigation line of fruit tree rows can be extracted effectively. The accuracy of orchard centerline extraction is 90.00%.

Visual servoing (VS) control has seen wide adoption in harvesting robots. However, parameter calibration is cumbersome, which makes the use of VS robotic systems inconvenient. Besides, dynamic fruits usually lead to a degeneration of control while tracking. To overcome the drawbacks, we present a new image-based uncalibrated visual servoing (IBUVS) control approach, consisting of a hybrid visual configuration and an adaptive tracking controller, referred to as hybrid-IBUVS. Specifically, our hybrid-IBUVS employs an eye-in-hand camera and a fixed red–green–blue-depth camera to construct a hybrid VS system, basing on multiobject detection and edge-computing technologies. Meanwhile, we also propose adaptive laws to online estimate the uncalibrated parameters of the cameras and robot dynamics. Furthermore, our hybrid-IBUVS uses an adaptive tracking controller to guarantee the harvesting robot to track a predefined trajectory to approach a fruit target. By Lyapunov stability theory, asymptotic convergence of the proposed control scheme is rigorously proven. Experimental results demonstrate the effectiveness of the proposed scheme. All shown results supported the research claims.

Robotic High-Throughput Phenotyping (HTP) technology has been a powerful tool for selecting high-quality crop varieties among large quantities of traits. Due to the advantages of multi-view observation and high accuracy, ground HTP robots have been widely studied in recent years. In this paper, we study an ultra-narrow wheeled robot equipped with RGB-D cameras for inter-row maize HTP. The challenges of the narrow operating space, intensive light changes, and messy cross-leaf interference in rows of maize crops are considered. An in situ and inter-row stem diameter measurement method for HTP robots is proposed. To this end, we first introduce the stem diameter measurement pipeline, in which a convolutional neural network is employed to detect stems, and the point cloud is analyzed to estimate the stem diameters. Second, we present a clustering strategy based on DBSCAN for extracting stem point clouds under the condition that the stem is shaded by dense leaves. Third, we present a point cloud filling strategy to fill the stem region with missing depth values due to the occlusion by other organs. Finally, we employ convex hull and plane projection of the point cloud to estimate the stem diameters. The results show that the R² and RMSE of stem diameter measurement are up to 0.72 and 2.95 mm, demonstrating its effectiveness.