With advances in precision agriculture, autonomous agricultural machines can reduce human labor, optimize workflow, and increase productivity. Accurate and reliable obstacle-detection and avoidance systems are essential for ensuring the safety of automated agricultural machines. Existing LiDAR-based obstacle detection methods for the farmland environment process the point clouds via manually designed features, which is time-consuming, labor-intensive, and weak in terms of generalization. In contrast, deep learning has a powerful ability to learn features autonomously. In this study, we attempted to apply deep learning in LiDAR-based 3D obstacle detection for the farmland environment. In terms of perception hardware, we established a data acquisition platform including LiDAR, a camera, and a GNSS/INS on the agricultural machine. In terms of perception method, considering the different agricultural conditions, we used our datasets to train an effective 3D obstacle detector, known as Focal Voxel R-CNN. We used focal sparse convolution to replace the original 3D sparse convolution because of its adaptable ability to extract effective features from sparse point cloud data. Specifically, a branch of submanifold sparse convolution was added to the upstream of the backbone convolution network; this adds weight to the foreground point and retains more valuable information. In comparison with Voxel R-CNN, the proposed Focal Voxel R-CNN significantly improves the detection performance for small objects, and the AP in the pedestrian class increased from 89.04% to 92.89%. The results show that our model obtains an mAP of 91.43%, which is 3.36% higher than the base model. The detection speed is 28.57 FPS, which is 4.18 FPS faster than the base model. The experiments show the effectiveness of our model, which can provide a more reliable obstacle detection model for autonomous agricultural machines.

In large-scale arable farming, multiple sequential operations involving multiple machines must be carried out simultaneously due to restrictions of short time windows. However, the coordination and planning of multiple sequential operations is a nontrivial task for farmers, since each operation may have its own set of operational features, e.g., operating width and turning radius. Taking the two sequential operations—hoeing cultivation and seeding—as an example, the seeder has double the width of the hoeing cultivator, and the seeder must remain idle while waiting for the hoeing cultivator to finish two rows before it can commence its seeding operation. A flow-shop working mode can coordinate multiple machines in multiple operations within a field when different operations have different implement widths. To this end, an auto-steering-based collaborative operating system for fleet management (FMCOS) was developed to realize an in-field flow-shop working mode, which is often adopted by the scaled agricultural machinery cooperatives. This paper proposes the structure and composition of the FMCOS, the method of operating strip segmenting, and a new algorithm for strip state updating between successive field operations under an optimal strategy for waiting time conditioning between sequential operations. A simulation model was developed to verify the state-updating algorithm. Then, the prototype system of FMCOS was combined with auto-steering systems on tractors, and the collaborative operating system for the server was integrated. Three field experiments of one operation, two operations, and three operations were carried out to verify the functionality and performance of FMCOS. The results of the experiment showed that the FMCOS could coordinate in-field fleet operations while improving both the job quality and the efficiency of fleet management by adopting the flow-shop working mode.

Digital twins in smart farming: An autoware-based simulator for autonomous agricultural vehicles

The application of autonomous agricultural vehicles is gaining popularity as a way to increase production efficiency and lower operational costs. To achieve high performance, perception tasks (such as obstacle detection, road extraction, and drivable area extraction) are of great importance. Compared with structured roads, field roads between farmlands, including unstructured roads and semi-structured roads, are unfavorable for autonomous agricultural vehicle driving due to their bumpiness and unstructured nature. This study proposed an extraction method for the straight field roads between farmlands. The proposed method was based on the point cloud data acquired by LiDAR (Velodyne VLP-16) mounted on a John Deere 1204 6B-1204 tractor. The proposed method has three aspects: Euclidean Clustering-based extraction, boundary-based extraction, and road point cloud curve segment modification. Firstly, Euclidean Clustering with K-Dimensional (KD)-Tree data structure was adopted to extract the road curve segments close to the LiDAR composed of road points. Secondly, the boundary lines constraint was constructed to extract the distant road curve segments. Thirdly, the local distance ratio was used to modify the extracted road curve segments. The average extraction accuracy for both semi-structured and unstructured roads exceeded 98%, and the false positive rate (FPR) was less than 0.5%. These experimental findings demonstrated that the proposed road extraction method was precise and effective. The proposed method of this study can be applied to enhance the perception ability of autonomous agricultural vehicles thereby increasing the efficiency and safety of field road driving. Keywords: road extraction, straight field road, autonomous agricultural vehicle, LiDAR, farmland DOI: 10.25165/j.ijabe.20221505.6933 Citation: Yang L L, Xu Y Y, Liang Y J, Qin J, Li Y B, Wang X X, et al. Extraction of straight field roads between farmlands based on agricultural vehicle-mounted LiDAR. Int J Agric & Biol Eng, 2022; 15(5): 155–162.

In order to qualitatively recognize the behaviors and investigate the relationship between fuel consumption and machinery driving modes of the tractor in a low-cost approach, this study proposed a method for behavior recognition and fuel consumption prediction of tractor sowing operations using a smartphone. First, three driving modes were developed for maize sowing scenarios: manual driving assisted driving and unmanned driving. While sowing, smartphone software and CAN (Controller Area Network) storage devices collected both positional data and engine operating conditions. Second, the tractor trajectory points were divided into kinematic sequences, with six driving cycle indicators built in each series based on the time window. Based on the semantic information of the kinematic sequences, the three operations of sowing, seeds filling, and turning round were well recognized. Last, a model for maize sowing fuel consumption forecast was advanced using the principal component analyses and random forest algorithm, regarding three factors: driving cycles, operating behaviors, and driving patterns. When compared to the traditional K-means algorithm, the results demonstrated that the harmonic mean of the precision and recall (F1 score) of sowing behavior recognition, seeds filling behavior recognition, and turning behavior recognition were enhanced by 2.06%, 8.99%, and 21.79%, respectively. In terms of the impacts of driving modes and operating behaviors on fuel consumption, assisted driving mode had the lowest fuel usage for both sowing and turning behavior. Therefore, assisted driving is the most fuel-efficient mode for maize sowing. Combining the three driving modes, the relative error of the fuel consumption prediction model was 0.11 L/h, with the manual driving mode having the lowest relative error at 0.09 L/h. This research method lays the foundation for the optimization of tractor operation behavior, the selection of tractor driving mode, and the fine management of tractor fuel consumption. Keywords: smartphone, kinematic sequence, operating behavior, fuel consumption forecast, tractor DOI: 10.25165/j.ijabe.20221504.7454 Citation: Yang L L, Tian W Z, Zhai W X, Wang X X, Chen Z B, Wen L, et al. Behavior recognition and fuel consumption prediction of tractor sowing operations using a smartphone. Int J Agric & Biol Eng, 2022; 15(4): 154–162.