Cosegmentation is a newly emerging computer vision technique used to segment an object from the background by processing multiple images at the same time. Traditional plant phenotyping analysis uses thresholding segmentation methods which result in high segmentation accuracy. Although there are proposed machine learning and deep learning algorithms for plant segmentation, predictions rely on the specific features being present in the training set. The need for a multi-featured dataset and analytics for cosegmentation becomes critical to better understand and predict plants’ responses to the environment. High-throughput phenotyping produces an abundance of data that can be leveraged to improve segmentation accuracy and plant phenotyping. This paper introduces four datasets consisting of two plant species, Buckwheat and Sunflower, each split into control and drought conditions. Each dataset has three modalities (Fluorescence, Infrared, and Visible) with 7 to 14 temporal images that are collected in a high-throughput facility at the University of Nebraska-Lincoln. The four datasets (which will be collected under theCosegPPdata repository in this paper) are evaluated using three cosegmentation algorithms: Markov random fields-based, Clustering-based, and Deep learning-based cosegmentation, and one commonly used segmentation approach in plant phenotyping. The integration ofCosegPPwith advanced cosegmentation methods will be the latest benchmark in comparing segmentation accuracy and finding areas of improvement for cosegmentation methodology.
Object segmentation algorithms have heavily relied on deep learning techniques to estimate the count of grapes which is a strong indicator for the yield success of grapes. The issue with using object segmentation algorithms for grape analytics is that they are limited to counting only the visible grapes, thus omitting hidden grapes, which affect the true estimate of grape yield. Many grapes are occluded because of either the compactness of the grape bunch cluster or due to canopy interference. This introduces the need for models to be able to estimate the unseen berries to give a more accurate estimate of the grape yield by improving grape cluster segmentation. We propose the Grape Counting Network (GCNet), a novel framework for grape cluster segmentation, integrating deep learning techniques with correction factors to address challenges in indoor yield estimation. GCNet incorporates occlusion adjustments, enhancing segmentation accuracy even under conditions of foliage and cluster compactness, and setting new standards in agricultural indoor imaging analysis. This approach improves yield estimation accuracy, achieving a R² of 0.96 and reducing mean absolute error (MAE) by 10% compared to previous methods. We also propose a new dataset called GrapeSet which contains visible imagery of grape clusters imaged indoors, along with their ground truth mask, total grape count, and weight in grams. The proposed framework aims to encourage future research in determining which features of grapes can be leveraged to estimate the correct grape yield count, equip grape harvesters with the knowledge of early yield estimation, and produce accurate results in object segmentation algorithms for grape analytics.
Cosegmentation and coattention are extensions of traditional segmentation methods aimed at detecting a common object (or objects) in a group of images. Current cosegmentation and coattention methods are ineffective for objects, such as plants, that change their morphological state while being captured in different modalities and views. The Object State Change using Coattention-Cosegmentation (OSC-CO2) is an end-to-end unsupervised deep-learning framework that enhances traditional segmentation techniques, processing, analyzing, selecting, and combining suitable segmentation results that may contain most of our target object’s pixels, and then displaying a final segmented image. The framework leverages coattention-based convolutional neural networks (CNNs) and cosegmentation-based dense Conditional Random Fields (CRFs) to address segmentation accuracy in high-dimensional plant imagery with evolving plant objects. The efficacy of OSC-CO2 is demonstrated using plant growth sequences imaged with infrared, visible, and fluorescence cameras in multiple views using a remote sensing, high-throughput phenotyping platform, and is evaluated using Jaccard index and precision measures. We also introduce CosegPP+, a dataset that is structured and can provide quantitative information on the efficacy of our framework. Results show that OSC-CO2 out performed state-of-the art segmentation and cosegmentation methods by improving segementation accuracy by 3% to 45%.
A high-throughput plant phenotyping system usually consists of a variety of cameras in different modalities, that are used to image the plants for computing various phenotypes. Most of them, e.g., visible light, fluorescent, and near-infrared, capture the reflected light at wide bands. For example, red-green-blue (RGB) cameras consider the visible part of the spectrum (380 –700 nm) in only three bands (red, green, and blue). Whereas this is adequate for computing many structural and appearance-based phenotypes, this coarse spectral resolution is not sufficient for more subtle phenotypes that may be accessible only at narrow wavelength ranges. Hyperspectral imaging (HSI) addresses this problem by capturing a broad range of wavelengths at very narrow intervals. Hyperspectral cameras can typically capture a scene in hundreds of bands where each band has a resolution of a few nanometers. Figure 9.1 shows 176the wavelengths captured by cameras in different modalities for image-processing applications. It is evident from Figure 9.1 that hyperspectral cameras have the highest coverage of the electromagnetic spectrum (Lowe et al. 2017). A multispectral camera also captures images in narrow wavelength ranges. However, it has only a small number of bands (typically < 20), in contrast to those in hyperspectral images. Figure 9.1 Overview of the different wavelengths captured with different types of cameras. (Reproduced from Lowe et al. 2017 under Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/)).
Cosegmentation is a recent and rapidly emerging and rapidly growing extension of segmentation, which aims to detect the common object(s) in a group of images. Current cosegmentation methods are ideal and effective only for certain dataset types with limited features that still produce errors making it difficult to obtain detailed metrics of object parts. We propose to build a unified, trainable framework that incorporates multiple features of a high-throughput dataset’s segmented images from multiple algorithms using cosegmentation. Specifically, we propose a novel Cosegmentation for Plant Phenotyping Network (CoPPNet) that utilizes a Fully Convolutional Neural Network with a K-Means Clustering feedback loop for optimal temporal loss. The results from this study will set the benchmark for a novel advancement in computer vision segmentation accuracy and plant phenomics to better understand a plant’s environmental interactions for maximal resilience and yield.
Manual survey data entry remains a bottleneck in large-scale research, marketing, and public policy, where survey sheets are still widely used due to accessibility and high response rates. Despite the progress in Optical Character Recognition (OCR) and Optical Mark Recognition (OMR), existing systems treat these tasks separately and are typically tailored to clean, standardized forms, making them unreliable for real-world survey sheets with diverse markings and handwritten inputs. These limitations hinder automation and introduce significant error rates in data transcription. To address this, we propose SurveyNet, a unified deep learning framework that combines OCR and OMR capabilities to automatically digitize complex survey responses within a single model. SurveyNet processes both handwritten digits and a wide variety of mark types including ticks, circles, and crosses across multiple question formats. We also introduce SurveySet, a novel dataset comprising 135 real-world survey forms annotated across four key response types. Experimental results demonstrate that SurveyNet achieves between 50% and 97% classification accuracy across tasks, with strong performance even on small and imbalanced datasets. This framework offers a scalable solution for streamlining survey digitization workflows, reducing manual errors, and enabling timely analysis in domains ranging from consumer research to public health and education.