Shape is one of the most important traits of agricultural products due to its relationships with the quality, quantity, and value of the products. For strawberries, the nine types of fruit shape were defined and classified by humans based on the sampler patterns of the nine types. In this study, we tested the classification of strawberry shapes by machine learning in order to increase the accuracy of the classification, and we introduce the concept of computerization into this field. Four types of descriptors were extracted from the digital images of strawberries: (1) the Measured Values (MVs) including the length of the contour line, the area, the fruit length and width, and the fruit width/length ratio; (2) the Ellipse Similarity Index (ESI); (3) Elliptic Fourier Descriptors (EFDs), and (4) Chain Code Subtraction (CCS). We used these descriptors for the classification test along with the random forest approach, and eight of the nine shape types were classified with combinations of MVs + CCS + EFDs. CCS is a descriptor that adds human knowledge to the chain codes, and it showed higher robustness in classification than the other descriptors. Our results suggest machine learning's high ability to classify fruit shapes accurately. We will attempt to increase the classification accuracy and apply the machine learning methods to other plant species.
5 May 2021 in International Journal of Automation Technology
Digital image phenotyping has become popular in plant research. Plants are complex in shape, and occlusion can often occur. Three-dimensional (3D) data are expected to measure the morphological traits of plants with higher accuracy. Plants have organs with flat and/or narrow shapes and similar component structures are repeated. Therefore, it is difficult to construct an accurate 3D model by applying methods developed for industrial materials and architecture. Here, we review noncontact and all-around 3D modeling and configuration of camera systems to measure the morphological traits of plants in terms of system composition, accuracy, cost, and usability. Typical noncontact 3D measurement methods can be roughly classified into active and passive methods. We describe their advantages and disadvantages. Structure-from-motion/multi-view stereo (SfM/MVS), a passive method, is the most frequently used measurement method for plants. It is described in terms of “forward intersection” and “backward resection.” We recently developed a novel SfM/MVS approach by mixing the forward and backward methods, and we provide a brief overview of our approach in this paper. While various fields are adopting 3D model construction, nonexpert users struggle to use them and end up selecting inadequate methods, which lead to model failure. We hope that this review will help users who are considering starting to construct and measure 3D models.
In this study, we developed an all-around 3D plant modeling system that operates using images and is capable of measuring plants non-destructively without any contact. During the fabrication of this device, we selected a method capable of performing 3D model reconstruction from multiple images. We then developed an improved SfM-MVS (Structure from Motion / Multi-View-Stereo) method that enables 3D reconstruction by simply capturing images with a camera. The resulting image-based method offers a high degree of freedom because the hardware and software can comprise commercially available products, and it permits the use of one or more cameras according to the shape and size of the plant. The advantages of the image-based method are that 3D reconstruction can be conducted at any time as long as the images are already taken, and that the desired locations can be observed, measured, and analyzed from 2D images and a 3D point cloud. The device we developed is capable of 3D measurements and modeling of plants from a few millimeters to 2.4 m of height using this method. This article explains this device, the principles of its composition, and the accuracy of the models obtained from it.
Three-dimensional measurement is a high-throughput method that can record a large amount of information. Three-dimensional modelling of plants has the possibility to not only automate dimensional measurement, but to also enable visual assessment to be quantified, eliminating ambiguity in human judgment. In this study, we have developed new methods that could be used for the morphological analysis of plants from the information contained in 3D data. Specifically, we investigated characteristics that can be measured by scale (dimension) and/or visual assessment by humans. The latter is particularly novel in this paper. The characteristics that can be measured on a scale-related dimension were tested based on the bounding box, convex hull, column solid, and voxel. Furthermore, for characteristics that can be evaluated by visual assessment, we propose a new method using normal vectors and local curvature (LC) data. For these examinations, we used our highly accurate all-around 3D plant modelling system. The coefficient of determination between manual measurements and the scale-related methods were all above 0.9. Furthermore, the differences in LC calculated from the normal vector data allowed us to visualise and quantify the concavity and convexity of leaves. This technique revealed that there were differences in the time point at which leaf blistering began to develop among the varieties. The precise 3D model made it possible to perform quantitative measurements of lettuce size and morphological characteristics. In addition, the newly proposed LC-based analysis method made it possible to quantify the characteristics that rely on visual assessment. This research paper was able to demonstrate the following possibilities as outcomes: (1) the automation of conventional manual measurements, and (2) the elimination of variability caused by human subjectivity, thereby rendering evaluations by skilled experts unnecessary.
This research aims to develop a novel technique to acquire a large amount of high-density, high-precision 3D point cloud data for plant phenotyping using photogrammetry technology. The complexity of plant structures, characterized by overlapping thin parts such as leaves and stems, makes it difficult to reconstruct accurate 3D point clouds. One challenge in this regard is occlusion, where points in the 3D point cloud cannot be obtained due to overlapping parts, preventing accurate point capture. Another is the generation of erroneous points in non-existent locations due to image-matching errors along object outlines. To overcome these challenges, we propose a 3D point cloud reconstruction method named closed-loop coarse-to-fine method with multi-masked matching (CLCFM3). This method repeatedly executes a process that generates point clouds locally to suppress occlusion (multi-matching) and a process that removes noise points using a mask image (masked matching). Furthermore, we propose the closed-loop coarse-to-fine method (CLCFM) to improve the accuracy of structure from motion, which is essential for implementing the proposed point cloud reconstruction method. CLCFM solves loop closure by performing coarse-to-fine camera position estimation. By facilitating the acquisition of high-density, high-precision 3D data on a large number of plant bodies, as is necessary for research activities, this approach is expected to enable comparative analysis of visible phenotypes in the growth process of a wide range of plant species based on 3D information.
3D measurement is a high-throughput method that can record a large amount of information. In this study, we have developed new methods that could be used for morphological analysis of plants from the information contained in 3D data. Specifically, we investigated characteristics that can be measured by scale (dimension) and/or visual assessment by humans. The characteristics that can be measured on a scale-related dimension were tested based on the bounding box, convex hull, column solid, and voxel. Furthermore, for characteristics that can be evaluated by visual assessment, we propose a new method using normal vectors and local curvature (LC) data. For these examinations, we used our highly accurate all-around 3D plant modelling system. The correlation coefficients between manual measurements and the scale-related methods were all above 0.9. In particular, the differences in LC calculated from the normal vector data allowed us to visualize and quantify the concavity and convexity of leaves. Furthermore, we also found a difference in the time point at which leaf blistering began to develop among the cultivars. The precise 3D model made it possible to perform quantitative measurements of lettuce size and morphological characteristics. In addition, the newly proposed LC-based analysis method made it possible to quantify the characteristics that rely on visual assessment.