Litchi clusters in fruit groves are randomly scattered and occur irregularly, so it is difficult to detect and locate the fruit-bearing branches of multiple litchi clusters at one time. This is a highly challenging task related to continuous operation in the natural environment for visual-based harvesting robots to carry out. In this study, a reliable algorithm based on RGB-depth (RGB-D) cameras in the fields was developed to accurately and automatically detect and locate the fruit-bearing branches of multiple litchi clusters simultaneously in large environments. A semantics segmentation method, Deeplabv3, was employed to segment the RGB images into three categories: background, fruit and twig. A pre-processing step is proposed to align the segmented RGB images and remove the twigs that did not bear fruits. Subsequently, the twig binary map image was processed via skeleton extraction and pruning operations, which left behind only the main branches of twigs. A method for non-parametric density-based spatial clustering of application with noise was used to cluster the pixels in the three-dimensional space of the skeleton map of the branches; thus, the fruit-bearing branches belonging to the same litchi clusters were determined. Finally, a three-dimensional straight line was fitted to each cluster via principal component analysis, and the linear information corresponded to the location of the fruit-bearing branches. In the experiments, 452 pairs of RGB-D images under different illumination were collected to test the proposed algorithm. The results show that the detection accuracy of a litchi fruit-bearing branch is 83.33%, positioning accuracy is 17.29°±24.57°, and execution time for the determination of a single litchi fruit-bearing branch is 0.464s. Field experiments show that this method can effectively guide the robot to complete continuous picking tasks.

For the automated robotic picking of bunch-type fruit, the strategy is to roughly determine the location of the bunches, plan the picking route from a remote location, and then locate the picking point precisely at a more appropriate, closer location. The latter can reduce the amount of information to be processed and obtain more precise and detailed features, thus improving the accuracy of the vision system. In this study, a long-close distance coordination control strategy for a litchi picking robot was proposed based on an Intel Realsense D435i camera combined with a point cloud map collected by the camera. The YOLOv5 object detection network and DBSCAN point cloud clustering method were used to determine the location of bunch fruits at a long distance to then deduce the sequence of picking. After reaching the close-distance position, the Mask RCNN instance segmentation method was used to segment the more distinctive bifurcate stems in the field of view. By processing segmentation masks, a dual reference model of “Point + Line” was proposed, which guided picking by the robotic arm. Compared with existing studies, this strategy took into account the advantages and disadvantages of depth cameras. By experimenting with the complete process, the density-clustering approach in long distance was able to classify different bunches at a closer distance, while a success rate of 88.46% was achieved during fruit-bearing branch locating. This was an exploratory work that provided a theoretical and technical reference for future research on fruit-picking robots.

Due to the presence of numerous surface defects, the inadequate contrast between defective and non-defective regions, and the resemblance between noise and subtle defects, edge detection poses a significant challenge in dimensional error detection, leading to increased dimensional measurement inaccuracies. These issues serve as major bottlenecks in the domain of automatic detection of high-precision metal parts. To address these challenges, this research proposes a combined approach involving the utilization of the YOLOv6 deep learning network in conjunction with metal lock body parts for the rapid and accurate detection of surface flaws in metal workpieces. Additionally, an enhanced Canny–Devernay sub-pixel edge detection algorithm is employed to determine the size of the lock core bead hole. The methodology is as follows: The data set for surface defect detection is acquired using the labeling software lableImg and subsequently utilized for training the YOLOv6 model to obtain the model weights. For size measurement, the region of interest (ROI) corresponding to the lock cylinder bead hole is first extracted. Subsequently, Gaussian filtering is applied to the ROI, followed by a sub-pixel edge detection using the improved Canny–Devernay algorithm. Finally, the edges are fitted using the least squares method to determine the radius of the fitted circle. The measured value is obtained through size conversion. Experimental detection involves employing the YOLOv6 method to identify surface defects in the lock body workpiece, resulting in an achieved mean Average Precision ($mAP$) value of 0.911. Furthermore, the size of the lock core bead hole is measured using an upgraded technique based on the Canny–Devernay sub-pixel edge detection, yielding an average inaccuracy of less than 0.03 mm. The findings of this research showcase the successful development of a practical method for applying machine vision in the realm of the automatic detection of metal parts. This achievement is accomplished through the exploration of identification methods and size-measuring techniques for common defects found in metal parts. Consequently, the study establishes a valuable framework for effectively utilizing machine vision in the field of metal parts inspection and defect detection.

This study designed a sub-pixel precision edge detecting algorithm to enhance contour smoothness. First, the coordinate value of RGB pixel is projected on the space line of R=G=B to obtain gray image. Then, pixel edges are located using a Canny detector. Next, the edge width is thinned to a single pixel using a morphological thinning operation. Finally, sub-pixel-level smooth contours are extracted by interpolation. In this sub-pixel level contour extraction process, a Single-Pixel-Multi-Point interpolation method was developed to enhance edge smoothness and obtain high precision in edge estimation. This method divides edges in a $3\times 3$ pixels block into nine arrangement modes. According to the arrangement of the eight neighborhoods of a centered edge pixel, different locations of interpolated sub-pixel points are calculated by interpolation with Bezier curves. For symmetrically arranged linear edge pixels, this method can be used to determine the exact contour. Experimental results showed that the proposed algorithm can improve the smoothness of image edge contour. As the curvature of the edge increases, the maximum systematic error will increase. For the edge pixel centered in the $3\times 3$ pixels block and two pixels located at the corner of one side, the max systematic error is 0.5 pixel. For two edge pixels aligned in a single row or column with one located at a corner, the max systematic error is 0.25 pixel.

Accurately identifying cherry-tomato picking points and obtaining their coordinate locations is critical to the success of cherry-tomato picking robots. However, previous methods for semantic segmentation alone or combining object detection with traditional image processing have struggled to accurately determine the cherry-tomato picking point due to challenges such as leaves as well as targets that are too small. In this study, we propose a YOLOv8n-DDA-SAM model that adds a semantic segmentation branch to target detection to achieve the desired detection and compute the picking point. To be specific, YOLOv8n is used as the initial model, and a dynamic snake convolutional layer (DySnakeConv) that is more suitable for the detection of the stems of cherry-tomato is used in neck of the model. In addition, the dynamic large convolutional kernel attention mechanism adopted in backbone and the use of ADown convolution resulted in a better fusion of the stem features with the neck features and a certain decrease in the number of model parameters without loss of accuracy. Combined with semantic branch SAM, the mask of picking points is effectively obtained and then the accurate picking point is obtained by simple shape-centering calculation. As suggested by the experimental results, the proposed YOLOv8n-DDA-SAM model is significantly improved from previous models not only in detecting stems but also in obtaining stem’s masks. In the mAP@0.5 and F1-score, the YOLOv8n-DDA-SAM achieved 85.90% and 86.13% respectively. Compared with the original YOLOv8n, YOLOv7, RT-DETR-l and YOLOv9c, the mAP@0.5 has improved by 24.7%, 21.85%, 19.76%, 15.99% respectively. F1-score has increased by 16.34%, 12.11%, 10.09%, 8.07% respectively, and the number of parameters is only 6.37M. In the semantic segmentation branch, not only does it not need to produce relevant datasets, but also improved its mIOU by 11.43%, 6.94%, 5.53%, 4.22% and mAP@0.5 by 12.33%, 7.49%, 6.4%, 5.99% compared to Deeplabv3+, Mask2former, DDRNet and SAN respectively. In summary, the model can well satisfy the requirements of high-precision detection and provides a strategy for the detection system of the cherry-tomato.

To address the difficulty of obtaining the optimal driving strategy under the condition of a complex environment and changeable tasks of vehicle autonomous driving, this paper proposes an end-to-end autonomous driving strategy learning method based on deep reinforcement learning. The ideas of target attraction and obstacle rejection of the artificial potential field method are introduced into the distributed proximal policy optimization algorithm, and the APF-DPPO learning model is established. To solve the range repulsion problem of the artificial potential field method, which affects the optimal driving strategy, this paper proposes a directional penalty function method that combines collision penalty and yaw penalty to convert the range penalty of obstacles into a single directional penalty, and establishes the vehicle motion collision model. Finally, the APF-DPPO learning model is selected to train the driving strategy for the virtual vehicle, and the transfer learning method is selected to verify the comparison experiment. The simulation results show that the completion rate of the virtual vehicle in the obstacle environment that generates penalty feedback is as high as 96.3%, which is 3.8% higher than the completion rate in the environment that does not generate penalty feedback. Under different reward functions, the method in this paper obtains the highest cumulative reward value within 500 s, which improves 69 points compared with the reward function method based on the artificial potential field method, and has higher adaptability and robustness in different environments. The experimental results show that this method can effectively improve the efficiency of autonomous driving strategy learning and control the virtual vehicle for autonomous driving behavior decisions, and provide reliable theoretical and technical support for real vehicles in autonomous driving decision-making.