Intelligent agriculture imposes higher requirements on the recognition and localization of fruit and vegetable picking robots. Due to its unique visual information and relatively low hardware cost, machine vision is widely applied in the recognition and localization of fruit and vegetable picking robots. This article provides an overview of the application of machine vision in the recognition and localization of fruit and vegetable picking robots. Firstly, the advantages, disadvantages, and the roles of different visual sensors and machine vision algorithms in the recognition and localization of fruit and vegetable picking robots are introduced, including monocular cameras, stereo cameras, structured light cameras, multispectral cameras, image segmentation algorithms, object detection algorithms, and 3D reconstruction algorithms. Then, the current status and challenges faced by machine vision in the recognition and localization of fruit and vegetable picking robots are summarized. These challenges include the stability of fast recognition under complex background interference, stability of recognition under different lighting environments for the same crop, the reliance of recognition and localization on prior information in the presence of fruit overlap and occlusions caused by leaves and branches, and the uncertainty of picking caused by complex working environments. In current research on algorithms dealing with complex background interference and various occlusion disturbances, good results have been achieved. Different lighting environments have a significant impact on the recognition and positioning of fruits and vegetables, with a minimum accuracy of 59.2%. Finally, this article outlines future research directions to address these challenges.

In a complex off-road environment, due to the low bearing capacity of the soil and the uneven features of the terrain, generating a safe and effective global route for unmanned ground vehicles (UGVs) is critical for the success of their motion and mission. Most traditional global path planning methods simply take the shortest path length as the optimization objective, which makes it difficult to plan a feasible and safe route in complex off-road environments. To address this problem, this research proposes a global path planning method, which considers the influence of terrain factors and soil mechanics on UGV mobility. First, we established a high-resolution 3D terrain model with remote sensing elevation terrain data, land use and soil type distribution data, based on a geostatistical method. Second, we analyzed the vehicle mobility by the terramechanical method (i.e., vehicle cone index and Bakker’s theory), and then calculated the mobility cost based on a fuzzy inference method. Finally, based on the calculated mobility cost, the probabilistic roadmap method was used to establish the connected matrix and the multi-dimensional traffic cost evaluation matrix among the sampling nodes, and then an improved A* algorithm was proposed to generate the global route.

Maize is one of the major food crops in China. Traditionally, field operations are done by manual labor, where the farmers are threatened by the harsh environment and pesticides. On the other hand, it is difficult for large machinery to maneuver in the field due to limited space, particularly in the middle and late growth stage of maize. Unmanned, compact agricultural machines, therefore, are ideal for such field work. This paper describes a method of monocular visual recognition to navigate small vehicles between narrow crop rows. Edge detection and noise elimination were used for image segmentation to extract the stalks in the image. The stalk coordinates define passable boundaries, and a simplified radial basis function (RBF)-based algorithm was adapted for path planning to improve the fault tolerance of stalk coordinate extraction. The average image processing time, including network latency, is 220 ms. The average time consumption for path planning is 30 ms. The fast processing ensures a top speed of 2 m/s for our prototype vehicle. When operating at the normal speed (0.7 m/s), the rate of collision with stalks is under 6.4%. Additional simulations and field tests further proved the feasibility and fault tolerance of our method.

The typical occlusion of cherry tomatoes in the natural environment is one of the most critical factors affecting the accurate picking of cherry tomato picking robots. To recognize occluded cherry tomatoes accurately and efficiently using deep convolutional neural networks, a new occluded cherry tomato recognition model DSP-YOLOv7-CA is proposed. Firstly, images of cherry tomatoes with different degrees of occlusion are acquired, four occlusion areas and four occlusion methods are defined, and a cherry tomato dataset (TOSL) is constructed. Then, based on YOLOv7, the convolution module of the original residual edges was replaced with null residual edges, depth-separable convolutional layers were added, and jump connections were added to reuse feature information. Then, a depth-separable convolutional layer is added to the SPPF module with fewer parameters to replace the original SPPCSPC module to solve the problem of loss of small target information by different pooled residual layers. Finally, a coordinate attention mechanism (CA) layer is introduced at the critical position of the enhanced feature extraction network to strengthen the attention to the occluded cherry tomato. The experimental results show that the DSP-YOLOv7-CA model outperforms other target detection models, with an average detection accuracy (mAP) of 98.86%, and the number of model parameters is reduced from 37.62MB to 33.71MB, which is better on the actual detection of cherry tomatoes with less than 95% occlusion. Relatively average results were obtained on detecting cherry tomatoes with a shade level higher than 95%, but such cherry tomatoes were not targeted for picking. The DSP-YOLOv7-CA model can accurately recognize the occluded cherry tomatoes in the natural environment, providing an effective solution for accurately picking cherry tomato picking robots.

Currently, high-fidelity simulation techniques, such as discrete element methods, are widely employed for predicting the off-road mobility of unmanned ground vehicles, ensuring the successful completion of field missions. However, the traversal simulation process based on discrete element terrain models is highly time-consuming. To address this challenge, we propose a double neural networks enhanced global mobility prediction model. Firstly, acceleration of the vehicle-terrain contact simulation is achieved by developing an artificial neural network surrogate model. This model predicts stress and strain relationships based on a multiscale representative volume element terrain model, effectively replacing complex mechanical calculations. Secondly, the generation of the global mobility map is expedited by introducing a boundary data enhanced artificial neural network model. This model generates a training dataset by simulating Gaussian sampled terrain data and additional enhanced terrain data at the mobility classification boundary. The trained artificial neural network is subsequently utilized to predict the mobility map, thus circumventing the time-consuming process of traversing simulations. The simulation tests validate that our proposed method not only significantly reduces computation time but also ensures high prediction accuracy.

To avoid unmanned ground vehicles being obstructed by deformed terrain in off-road, effective vehicle mobility analysis is required. However, the computational complexity of existing mobility analysis methods, such as discrete element analysis, poses significant challenges when applied to large-scale terrains. To address this problem, we propose an efficient and high-fidelity vehicle mobiliy prediction method for a large-scale terrain. Initially, precise terrain models are constructed employing Gaussian sampling, thereby serving as optimal inputs for the mobility simulation. Subsequently, we introduce a co-simulation method based on a multi-body dynamics model and discrete element analysis to obtain high-fidelity vehicle mobility data on sampled terrains. Following that, the mobility data is utilized to train a PSO-kriging neural network, enabling accurate predictions of the global mobility map. Through rigorous simulation experiments, the proposed method demonstrates its remarkable effectiveness.