The apple target recognition algorithm is one of the core technologies of the apple picking robot. However, most of the existing apple detection algorithms cannot distinguish between the apples that are occluded by tree branches and occluded by other apples. The apples, grasping end-effector and mechanical picking arm of the robot are very likely to be damaged if the algorithm is directly applied to the picking robot. Based on this practical problem, in order to automatically recognize the graspable and ungraspable apples in an apple tree image, a light-weight apple targets detection method was proposed for picking robot using improved YOLOv5s. Firstly, BottleneckCSP module was improved designed to BottleneckCSP-2 module which was used to replace the BottleneckCSP module in backbone architecture of original YOLOv5s network. Secondly, SE module, which belonged to the visual attention mechanism network, was inserted to the proposed improved backbone network. Thirdly, the bonding fusion mode of feature maps, which were inputs to the target detection layer of medium size in the original YOLOv5s network, were improved. Finally, the initial anchor box size of the original network was improved. The experimental results indicated that the graspable apples, which were unoccluded or only occluded by tree leaves, and the ungraspable apples, which were occluded by tree branches or occluded by other fruits, could be identified effectively using the proposed improved network model in this study. Specifically, the recognition recall, precision, mAP and F1 were 91.48%, 83.83%, 86.75% and 87.49%, respectively. The average recognition time was 0.015 s per image. Contrasted with original YOLOv5s, YOLOv3, YOLOv4 and EfficientDet-D0 model, the mAP of the proposed improved YOLOv5s model increased by 5.05%, 14.95%, 4.74% and 6.75% respectively, the size of the model compressed by 9.29%, 94.6%, 94.8% and 15.3% respectively. The average recognition speeds per image of the proposed improved YOLOv5s model were 2.53, 1.13 and 3.53 times of EfficientDet-D0, YOLOv4 and YOLOv3 and model, respectively. The proposed method can provide technical support for the real-time accurate detection of multiple fruit targets for the apple picking robot.

In recent years, many agriculture-related problems have been evaluated with the integration of artificial intelligence techniques and remote sensing systems. The rapid and accurate identification of apple targets in an illuminated and unstructured natural orchard is still a key challenge for the picking robot’s vision system. In this paper, by combining local image features and color information, we propose a pixel patch segmentation method based on gray-centered red–green–blue (RGB) color space to address this issue. Different from the existing methods, this method presents a novel color feature selection method that accounts for the influence of illumination and shadow in apple images. By exploring both color features and local variation in apple images, the proposed method could effectively distinguish the apple fruit pixels from other pixels. Compared with the classical segmentation methods and conventional clustering algorithms as well as the popular deep-learning segmentation algorithms, the proposed method can segment apple images more accurately and effectively. The proposed method was tested on 180 apple images. It offered an average accuracy rate of 99.26%, recall rate of 98.69%, false positive rate of 0.06%, and false negative rate of 1.44%. Experimental results demonstrate the outstanding performance of the proposed method.

The leaf area index (LAI) is a key parameter for describing the canopy structure of apple trees. This index is also employed in evaluating the amount of pesticide sprayed per unit volume of apple trees. Hence, numerous manual and automatic methods have been explored for LAI estimation. In this work, the leaf area indices for different types of apple trees are obtained in terms of multispectral remote-sensing data collected with an unmanned aerial vehicle (UAV), along with simultaneous measurements of apple orchards. The proposed approach was tested on apple trees of the “Fuji”, “Golden Delicious”, and “Ruixue” types, which were planted in the Apple Experimental Station of the Northwest Agriculture and Forestry University in Baishui County, Shaanxi Province, China. Five vegetation indices of strong correlation with the apple leaf area index were selected and used to train models of support vector regression (SVR) and gradient-boosting decision trees (GBDT) for predicting the leaf area index of apple trees. The best model was selected based on the metrics of the coefficient of determination (R²) and the root-mean-square error (RMSE). The experimental results showed that the gradient-boosting decision tree model achieved the best performance with an R² of 0.846, an RMSE of 0.356, and a spatial efficiency (SPAEF) of 0.57. This demonstrates the feasibility of our approach for fast and accurate remote-sensing-based estimation of the leaf area index of apple trees.

In the vision system of apple-picking robots, the main challenge is to rapidly and accurately identify the apple targets with varying halation and shadows on their surfaces. To solve this problem, this study proposes a novel, multi-feature, patch-based apple image segmentation technique using the gray-centered red-green-blue (RGB) color space. The developed method presents a multi-feature selection process, which eliminates the effect of halation and shadows in apple images. By exploring all the features of the image, including halation and shadows, in the gray-centered RGB color space, the proposed algorithm, which is a generalization of K-means clustering algorithm, provides an efficient target segmentation result. The proposed method is tested on 240 apple images. It offered an average accuracy rate of 98.79%, a recall rate of 99.91%, an F1 measure of 99.35%, a false positive rate of 0.04%, and a false negative rate of 1.18%. Compared with the classical segmentation methods and conventional clustering algorithms, as well as the popular deep-learning segmentation algorithms, the proposed method can perform with high efficiency and accuracy to guide robotic harvesting.

For the purpose of minimizing the damage to apples during the operation of an apple-picking robot, improving the stability of an apple-harvesting mechanical hand and reducing the use and operation cost of the robot, a novel biomimetic apple mechanical hand based on negative pressure air suction is purposely designed based on the physical characteristics of apples by imitating octopus predation. The mechanical hand has four suction cups evenly distributed in the hand, which produce concave deformation under negative vacuum pressure to fit the apple surface and adsorb the apple; the fruit stalk is separated by wrist rotation and dragging in a compound way to complete the picking operation. This paper firstly determines the key parameters of the mechanical hand based on the characteristics of silicone material and the three growth postures of apples, tries to make a physical prototype of the mechanical hand and conducts relevant performance tests on the picking platform built in the laboratory to explore the best combination of parameters and further optimize the manipulator. The results show that the success rate of the biomimetic nondestructive apple picker with the best combination of parameters is 100%, 76% and 68% for three different apple-growth postures and the operation is smooth, reliable, safe and non-invasive.

The rapid development of artificial intelligence and remote sensing technologies is indispensable for modern agriculture. In orchard environments, challenges such as varying light conditions and shading complicate the tasks of intelligent picking robots. To enhance the recognition accuracy and efficiency of apple-picking robots, this study aimed to achieve high detection accuracy in complex orchard environments while reducing model computation and time consumption. This study utilized the CenterNet neural network as the detection framework, introducing gray-centered RGB color space vertical decomposition maps and employing grouped convolutions and depth-separable convolutions to design a lightweight feature extraction network, Light-Weight Net, comprising eight bottleneck structures. Based on the recognition results, the 3D coordinates of the picking point were determined within the camera coordinate system by using the transformation relationship between the image’s physical coordinate system and the camera coordinate system, along with depth map distance information of the depth map. Experimental results obtained using a testbed with an orchard-picking robot indicated that the proposed model achieved an average precision (AP) of 96.80% on the test set, with real-time performance of 18.91 frames per second (FPS) and a model size of only 17.56 MB. In addition, the root-mean-square error of positioning accuracy in the orchard test was 4.405 mm, satisfying the high-precision positioning requirements of the picking robot vision system in complex orchard environments.