Maui: modular analytics of UAS imagery for specialty crop research

Abstract

Background

Imaging sensors (e.g., multispectral cameras) mounted on unmanned aerial systems (UAS) have emerged as a powerful tool for deriving insights about agricultural fields, from plant morphology phenotyping to plant disease monitoring. Advances in computer vision-based image analysis have enabled researchers to rapidly and accurately isolate crop spectra in UAS images. Specialty crops often employ unique production styles, such as trellising or inter-cropping. This presents a barrier to using existing image processing methodologies developed for broad-acre, row cropped systems (i.e. corn, wheat, soybean). Here, we present MAUI, a customizable image processing workflow built for specialty crops. Using a pathology research vineyard and hemp breeding trial as test cases, MAUI streamlines the generation of multispectral orthomosaic time-series, the segmentation of crops at the unit of research interest, and the extraction of crop spectra for downstream analysis.

Results

We successfully used MAUI to collect and analyze UAS data at two field sites over two growing seasons. Of the five canopy segmentation methods we tested, a supervised deep convolutional neural network (DeepLabv3) and a vision foundation model (SAM) produced the most accurate crop masks for the vineyard and hemp images, with mean intersection over union (mIoU) values of 0.85 and 0.95, respectively. Segmentation accuracy decreased when we applied each method to the other dataset, highlighting the importance of modular, flexible segmentation workflows for UAS imaging analysis in specialty crops.

Conclusion

We present a modular framework to efficiently extract spectral data for specialty crops from UAS imagery. We highlight two kinds of segmentation applied to trellised and row cropping systems to demonstrate the modularity and versatility of the proposed methodology. MAUI improved spectral discrimination between individual plants and treatment groups for hemp and grapevine, respectively. With the containerized deployment package and open-source codebase, MAUI can be widely adopted by specialty crop researchers to facilitate the integration of UAS imagery analysis into routine research.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13007-025-01376-7.

Background

Ease of data collection and image processing has dramatically increased the use of uncrewed aerial systems (UAS) in agriculture. In 2020 alone, 1275 studies were published using UAS to monitor agricultural areas compared to fewer than 200 publications in 2015 [1]. The lighter weight, lower cost, and increased operational flexibility (i.e., the user chooses the flight time and path) offered by UAS, compared to airplane or satellite-based remote sensing, are among the reasons that adoption has increased in recent years [2–4]. UAS fly at lower altitudes (typically 5–100 m above ground level), yielding high resolution images (e.g. mm to cm$·$pixel^-1) that allow for individual plant monitoring. By way of comparison, satellite Earth observation platforms such as Sentinel-2 and Landsat-9, though useful for field-scale monitoring, offer spatial resolutions of 10 m and 30 m$·$pixel^-1, respectively. However, there are challenges to UAS remote sensing. Lower flight altitudes mean that wind, birds, and other physical obstacles can adversely impact UAS data quality. Additionally, the number of images acquired (i.e. data volume) over a given area is much greater with a UAS than with satellite or airplane-based imaging systems. Although the enhanced spatial resolution of UAS imagery can provide more detailed information about crop status, accurate separation of target crops from soil, shadow, and non-target vegetation (e.g. weeds, cover crops, etc.) remains a challenge [2]. High accuracy segmentation is critical to ensure that reflectance differences can be attributed to the crop rather than off-target factors [5].

Lack of standardization in image acquisition and processing protocols to ensure high-quality raw data and scalable analyses is a major challenge to agricultural applications of UAS imagery [3]. This is particularly true for specialty crop studies. Unlike broad-acre crops such as corn or wheat, specialty crops are characterized by highly heterogeneous fields, mixed cultivar plantings, wider row intervals, and complex backgrounds (cover crops, weeds, variable soil types, etc.). Due to this heterogeneity, standard distance- or area-based image segmentation methods do not provide satisfactory segmentation accuracy. As a result, segmentation workflows proposed for specialty crops often incorporate multiple algorithms and/or multiple image-derived inputs and depend on a combination of software packages. For example, Gennaro and Matese [6] segmented vine canopies from surrounding soil by generating a digital elevation model (DEM) from UAS images and used Otsu’s thresholding to select an elevation value to differentiate vine from soil. To segment crop pixels in multispectral images of potato fields, Yang et al. [7] first split the UAS images into smaller patches and then employed a support vector machine classifier and Gaussian mixture model to select a threshold for segregating potato plants from soil. While threshold-based segmentation is simple and computationally fast, this approach is prone to error caused by changing solar illumination and/or inappropriate selection of threshold values [8]. Additionally, different crops necessitate different segmentation approaches. For trellised systems like grapevine, plant height can be employed to distinguish between crops and non-crop background, while for potato, which grows low to the ground, other features are required for accurate segmentation. Diverse crop varieties and cultivation methods have resulted in a multitude of crop-specific image segmentation methods, but we still lack adaptable workflows to efficiently analyze spectra of many different crops with the same tool suite.

Machine learning, particularly deep learning, has demonstrated substantial improvements for image segmentation in diverse agricultural contexts, especially when the original datasets are sparse [9, 10]. In a review of studies evaluating the performance of convolutional neural networks (CNNs) for agricultural image analysis, Kamilaris and Prenafeta-Boldú [10] found that CNNs performed favorably in most cases compared to other techniques. In grapevine, a SegNet model trained on visible and infrared UAS images accurately identified soil, shadow, healthy vines, and diseased vines [11]. A CNN was also used to segment citrus canopies by first learning the row orientation and plant spacing from training images, and then leveraging the normalized vegetation index (NDVI) to separate the canopy from bare soil [12]. Accurate segmentation allowed for tree size and biomass estimation, facilitating rapid comparison of phenotypes across a large breeding trial [12]. While many new deep learning models have been and are continuously developed [13], DeepLabv3 is a commonly used baseline architecture that employs atrous convolution to capture fine-resolution feature maps at multiple spatial scales [14]. A major advantage of DeepLabv3 for remotely-sensed images is the ability to incorporate contextual information at various scales, which is particularly important for accurate pixel classification at object boundaries [15]. This is critical for segmenting images of specialty crops, where different types of plant tissue must be distinguished from each other. Of six CNNs tested, DeepLabv3 with an Xception backbone was among the best performers in segmenting infected tissue of coffee, soybean, and wheat plants [16]. DeepLabv3 was also used to accurately detect in-field cucumber downy and powdery mildew lesions [17]. Though useful when plant/plant parts of interest share features with other image components, DeepLabv3 models require longer run times compared to other deep learning architectures [14, 16, 18].

Recently, vision foundation models have gained popularity in computer vision tasks, including image segmentation. These models deliver promising performance while drastically reducing or even eliminating the need for extensive data annotation, making them highly efficient and accessible for a variety of applications [19]. Among available models, the Segment Anything Model (SAM) is a widely-used vision foundation model pre-trained on a global dataset of 11 million images [20]. The ability to perform segmentation on new datasets offers an advantage over methods like DeepLabv3, which require significant training and computational time. SAM has also emerged as the most popular choice for researchers in need of “interactive” segmentation tools [21]. Unlike supervised CNN, SAM can run automatically or with user-defined prompts. Prompts can be a single point in the image, a text description, or a bounding box specifying the area of interest [20]. Fully automatic SAM struggles to isolate vegetation and vegetation feature instances (i.e., diseased leaf areas) from agricultural backgrounds, though this can be overcome by adding adapters or using prompts [22, 23]. Generally, SAM performs well when plants are regularly spaced and easily distinguishable from the background (e.g. bare soil), or when the model is prompted with bounding boxes or points [21].

Identifying the most appropriate segmentation models can be challenging for specialty crop researchers seeking to use UAS imagery. There are few open-source protocols that can be readily adapted to segment diverse specialty crops arranged in different planting schemes. Notable exceptions are Bhandari et al. [24] (integrating UAS data in wheat breeding programs) and Lam et al. [25] (open-source workflow for UAS-based weed mapping and management), although both use a single crop species as a representative case study. Domain plant scientists lack standardized tool suites and guidance on which segmentation technique(s) are appropriate for non-traditional agronomic systems and specialized research applications. For example, isolating and comparing the spectra of individual plants or treatment units requires a different approach than evaluating the aggregate reflectance of large fields.

To fill this gap, we present two segmentation methods within a customizable workflow, allowing users to evaluate both segmentation models and select the most suitable option based on their crop characteristics. The customizable pipeline allows users to modify, add, or remove individual modules according to their needs and is designed to facilitate phenotypic trait extraction in specialty crops. We present a vineyard and a hemp breeding trial as our two case studies to demonstrate the versatility of the proposed methodology. This adaptability differentiates MAUI from existing single-crop pipelines and enables researchers to efficiently conduct phenotypic studies in a diverse range of crops using one UAS system and software framework.

MAUI: modular analytics of UAS imagery

The Modular Analytics of UAS imagery (MAUI) framework consists of four modules: image acquisition, image pre-processing, plant canopy segmentation, and trait extraction and export (Fig. 1). All MAUI modules are implemented using Python. To facilitate adoption by plant scientists, the MAUI segmentation module is containerized in Docker for easy deployment. The MAUI codebase is also publicly available for customization based on specific applications. The Docker containers, codebase, sample datasets, and documentation are available in the MAUI Github repository: https://github.com/cu-cairlab/MAUI/tree/main.

Fig. 1.

Diagram of the software architecture of the MAUI framework. Yellow boxes are main MAUI modules. Green boxes indicate intermediate and final data outputs. Blue boxes are user inputs required to generate intermediate results. The format for all input and output files is indicated in italics. The user inputs are not hard coded in the main modules to offer customization flexibility to future users.*Docker images and bash scripts for canopy segmentation models can be accessed via the Box link on the MAUI GitHub repository at: https://github.com/cu-cairlab/MAUI/blob/main/Plant%20Canopy%20Segmentation/README_segmentation.md. **For detailed instructions on how to run scripts in each module, refer to the README documents in the MAUI GitHub repository: https://github.com/cu-cairlab/MAUI/tree/main

Image acquisition

MAUI is a platform-independent system that supports RGB and multispectral imagery collected by consumer-grade UAS. To ensure optimal processing accuracy, MAUI requires georeferenced images along with other standard quality criteria, such as sufficient overlap between consecutive images [26] and ground control points (GCPs) with surveyed GPS coordinates. Guidelines for deploying GCPs are available in a previous study [27]. Georeferenced images and surveyed GCPs enable high-quality orthomosaic generation in the preprocessing module.

Image preprocessing

The image preprocessing module reconstructs and co-registers orthomosaic images using georeferenced images and surveyed GCPs in two steps. The initial step involves generating orthomosaic images for all flights conducted during a research project. Rather than implementing its own mosaicking functionality, MAUI accepts GEOTIFF orthomosaic files generated by the users’ method of choice. This has three key advantages. First, users can choose their preferred orthomosaicking tools, including commercial software (e.g., Pix4Dmapper, Pix4D S.A., Prilly, Switzerland) and open-source libraries (e.g., Colmap [28, 29] and Open Drone Map [30]). Second, achieving high quality orthomosaics often requires manual tuning, which is best left to the user’s discretion. Finally, by defining only the input image format, MAUI provides maximum flexibility for users to select the mosiacking tools and workflows that best fit their needs.

Once the orthomosaics are generated, the preprocessing module provides an optional co-registration step using the Automated and Robust Open-Source Image Co-registration Software (AROSICS) [31]. MAUI employs the AROSICS global co-registration method, with the first acquired image serving as the reference and subsequent images as targets. MAUI uses AROSICS’ default parameters, which are effective in most cases. However, if the maximum shift detected between reference and target images exceeds the default five-pixel threshold, the max_shift parameter can be adjusted to the nearest integer greater than the detected shift. For a comprehensive description of AROSICS, we refer readers to the original publication [31].

Plant canopy segmentation

The plant canopy segmentation module facilitates accurate and automated processing of time series UAS imagery. This often presents a major challenge for plant scientists, limiting their ability to efficiently analyze large volumes of UAS data for routine research projects. To address this, MAUI provides a versatile segmentation module and a set of representative image segmentation methods. These methods include thresholding-based techniques (e.g., Otsu’s thresholding [32] and ExG thresholding [33]), a supervised deep learning model (e.g., DeepLabv3 [14]), and a vision foundation model (e.g., SAM [20]), offering flexibility to accommodate diverse research needs. Figure S4 outlines a decision tree to guide users in selecting the most appropriate model according to their field characteristics and computational resources.

Color-space adapted Otsu’s method

We adapted the threshold-based red-green-near infrared (NIR) color space segmentation described by Cinat et al. [8]. In their comparison of the most common conventional computer vision techniques for image segmentation, the authors found that this method performed most stably across multiple commercial vineyards in two growing seasons. Briefly, the segmentation was accomplished by first converting a multispectral UAS orthomosaic image to a three-band (i.e., red-green-NIR, RGN) image. Next, the RGN image was transformed to a hue-saturation-value (HSV) color space and Otsu’s thresholding was applied sequentially to different bands to separate soil, shadow, and vine pixels.

Supervised deep learning-based method

We fine-tuned a DeepLabv3 model [14] for plant canopy segmentation in UAS images. To fine-tune the model, we first generated red-green-blue (RGB) orthomosaics from the multispectral orthomosaics by using bands 1–3 of the MicaSense RedEdge MX - Blue camera (Table S1). Using RGB images enabled deploying the original DeepLabv3 architecture without any layer modifications or weight dropping and allowed us to improve the segmentation performance using a small set of manually annotated images. The RGB orthomosaics were split into image patches, and a set of patches was randomly selected for human annotation. This annotated image patch set was used to refine a DeepLabv3 model pretrained using the train2017 dataset in Common Objects in Context (COCO). The patch size depends on the crop unit of interest, i.e. smaller patches for segmenting individual plants (hemp) and larger patches for segmenting multi-plant experimental units (grapevine). Patch size is a customizable parameter in the MAUI codebase.

For a given crop of interest, we first converted the original multispectral orthomosaic to RGB and then split the full image into patches for annotation and training. The patches were used to fine-tune the DeepLabv3 model to achieve desired performance. The fine-tuned model was saved and applied to all subsequent orthomosaics of the target crop. To segment a new orthomosaic, the full image was first converted to RGB and subsequently segregated into patches that were fed into the fine-tuned model for segmentation. The resulting patch segmentation masks were reassembled as the complete plant canopy segmentation mask for the full image (Fig. S2).

Vision foundation model-based method

We developed a hybrid method for plant canopy segmentation by integrating the Excess Green (ExG) index with the Segment Anything Model (SAM). This approach consisted of four key steps: ExG calculation, density-based clustering, plant bounding box generation, and refined segmentation using SAM.

For a given orthomosaic image, ExG was first calculated for all pixels to obtain an ExG index map [33]. The ExG index map was then binarized by fixing a user-defined threshold to create a coarse vegetation mask. This mask separated vegetation (pixels with ExG values higher than the threshold) from non-vegetation pixels. Second, we leveraged the spatial layout of specialty crop systems and employed the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm [34] to identify clusters of dense vegetation for plant detection. In general, plants are regularly spaced in specialty crop systems, resulting in distinctive vegetation density patterns across the field. Higher vegetation density (namely more green pixels) indicates crop plant presence. The stationary nature of plants in the field enables re-using a single detection map throughout the growing season when the time series of images is precisely co-registered. Although DBSCAN successfully identified clusters of dense vegetation, it did not produce precise boundaries for individual plants over time. To overcome this limitation, we used the centroid coordinates of each vegetation cluster to generate bounding boxes that delineated plant boundaries more accurately. The size of these bounding boxes was determined based on the inter- and intra-row spacing at the study site (for example, in the hemp trial boxes are 160 $\times$ 160-pixels). Bounding box size is one of the pipeline’s customizable parameters and can be set by the user according to their field characteristics. The bounding boxes were then used to clip the full orthomosaic image into smaller patches, which were fed into SAM to generate refined plant masks. The final plant mask was obtained by intersecting the masks derived from the ExG method and SAM, combining the strengths of both approaches for accurate canopy segmentation.

Trait extraction and export

The trait extraction and export module extracts and exports reflectance traits at the scale of research interest, which can vary from individual plants to blocks or panels. This process utilizes co-registered orthomosaic image sets, corresponding plant canopy masks, and field metadata to ensure precise and reliable outputs. Field metadata is organized in GEOJSON format, where each geo-polygon (typically a bounding box) represents the chosen scale of research. For crops like hemp the scale may correspond to individual plants, while for crops like grape it might represent a group of consecutive vines. Each geo-polygon is associated with metadata such as identifier numbers, treatments, and other relevant attributes.

The metadata file can be prepared using either field RTK-GPS measurement or image-based methods. When RTK-GPS is available, the GPS coordinates of the start and end points of each polygon (panel-based studies), or the centroid of each plant (single-plant studies), are recorded. Using the known in-row and between-row spacing, polygon coordinates are generated automatically. Additional information (i.e. genotypes, heights, disease ratings) can then be added to complete the field metadata file. When RTK-GPS is not available, GPS coordinates can be extracted from the co-registered orthomosaic images using the DBSCAN methodology described above. Typically, the first orthomosaic image in a set is used to define the plant polygons. Scripts for generating GEOJSON metadata files via an RTK-GPS file and via the DBSCAN method are included in the MAUI GitHub repository.

Once the metadata is prepared, each orthomosaic image is processed using its corresponding plant canopy mask. The resulting masked images show only plant pixels. The metadata polygons are applied to the masked images to identify and extract the canopy reflectance data for each research unit. The extracted reflectance traits, along with the corresponding metadata, are exported as CSV files. The CSV file is indexed by unique identifiers assigned to each research unit. This indexing ensures that any additional data collected for these units can be easily appended to the same file, supporting downstream phenotypic analyses.

Case studies using MAUI in grapevine and hemp research

Data acquisition

We conducted weekly UAS flights over two research sites in Geneva, NY, USA (Fig. 2), from June to August in 2022 and 2023: a high-cannabinoid hemp (Cannabis sativa) breeding trial and a grapevine (Vitis vinifera) fungicide efficacy trial. Field locations, site descriptions, and UAS flight parameters are shown in Fig. 2 and detailed in Table 1. To ensure precise orthomosaic generation and co-registration across multiple dates, we installed 0.3 $\times$ 0.3 m acrylic ground control points (GCPs) along the perimeter of each field. The coordinates of each GCP were recorded using a SwiftNav Duro real time kinematic (RTK) GNSS receiver (Swift Navigation, California, USA), achieving sub-centimeter geolocation accuracy.

Fig. 2.

a Locations of the UAS experiments conducted at Cornell AgriTech with field sites highlighted in yellow. b and c show orthomosaics acquired in 2022 at the hemp field and vineyard, respectively. Orthomosaics are overlaid on a Google satellite image of the corresponding field

Table 1.

Study site descriptions and UAS flight parameters for imagery acquired in 2022 and 2023 over vineyard and hemp field sites

Crop	Location	Field size	Inter-plant spacing	Inter-row spacing	Flight altitude	Flight velocity	Ground sample distance (GSD)$_{}^1$
Grapevine Vitis vinifera ‘Chardonnay’	42.878 N, -77.015 E	0.6 hectares	1.83 m	2.74 m	30 m	2 m/s	2.2 cm/ pixel
Hemp Cannabis sativa ‘Carolina Dream’ and ‘BaOx’	42.895 N, -77.007 E	0.4 hectares	2.29 m	2.29 m	20 m	3 m/s	1.4 cm/ pixel

$_{}^1$ GSD values were averaged for all images in each crop dataset

We acquired all imagery with the MicaSense RedEdge-MX^TM Dual Camera System (AgEagle, Kansas, USA) mounted on a DJI Matrice 600 Pro quadcopter (DJI, Shenzhen, China). The system consists of two multispectral cameras, each capturing reflectance in five spectral bands in the visible-to-near infrared range (Table S1). Both cameras have 13 mm focal length and were set to implement automatic gain and exposure control with an image overlap of 20%. Images were 1280 $\times$ 960 pixels in size. All flights occurred between 11:00 AM and 1:00 PM Eastern Daylight Time to standardize solar illumination angles across dates and to mitigate shadow pixels.

Flight paths were created using the Universal Ground Control Software (UgCS) Area Scan tool (UgCS Client v. 4.7.6, SPH Engineering, Riga, Latvia), which generates waypoint grids based on field boundaries input by the user. We set the flight paths to align with the plant rows in either East–West or West–East directions. In 2022, we flew at 30 m altitude and 2 m$·$s^-1 for both hemp and grapevine sites. In 2023 we reduced the flight altitude for hemp to 20 m with a velocity of 3 m$·$s^-1 to improve spatial resolution. These flight parameters resulted in ground sample distances (GSD) of 2.2 cm and 1.4 cm$·$pixel^-1 in 2022 and 2023, respectively (Table 1). Subsequently, we used the MAUI framework to create data processing pipelines for analyzing the UAS imagery collected in each case study (Fig. 3).

Fig. 3.

UAS image acquisition, processing, plant segmentation, and trait extraction workflow for vineyard panels and hemp plants. Steps that differ between the two crops are indicated in bold. Names of hardware and software used in each step are italicized

Data preprocessing

We adapted the Agisoft Metashape guide for processing MicaSense RedEdge-MX^TM imagery to batch-process multiple flightlines simultaneously [35]. Before initiating the batch process, we first imported the CSV file containing the X, Y, and Z coordinates of the field’s GCPs. Next, we imported all image sets for that field (each image set is equivalent to a single flight) as separate chunks using the “Add Folder”command and selected the “multi-camera” option. For each chunk, we used the “Calibrate Reflectance” function in the Tools menu to calibrate at-sensor reflectance with the CSV file containing calibration values for each waveband. This CSV file was obtained by contacting MicaSense support and providing the serial number on the reflectance calibration panel included with the cameras. In the reflectance calibration step, we selected both “use sun sensor” and “use panels” to calibrate using both the values from the CSV file and the correction factors from the ambient light sensor mounted on the UAS, according to the recommendations in the Agisoft Metashape guide [35]. Once all chunks were imported and calibrated, we saved the project and proceeded to the batch process. Detailed steps for batch processing are provided in Supplmentary Table S2. After the batch process, we exported each orthomosaic as a GeoTIFF file. We used the “Draw Polygon” tool to delineate the area to export (removing sections outside the field), or “outer boundary”. We saved the polygon shapefile and used this file to set the same boundary for all orthomosaics at each site.

Segmentation configuration and performance evaluation

There are three segmentation methods implemented in MAUI: Otsu’s method with a selected color space, DeepLabv3, and a hybrid approach that combines SAM with the Excess Green index (ExG). The following sections describe how each method was applied in grapevine and hemp case studies. The methods were evaluated and compared to assess their effectiveness and adaptability in different crop types.

DeepLabv3 fine-tuning

To fine-tune the pre-trained DeepLabv3 model [36], we manually annotated a vineyard image dataset comprising 17 training patches, 7 validation patches, and 11 testing patches. For grapevine we divided the orthomosaics into 512$\times$512-pixel patches, applying padding to the image edges as needed to achieve the desired patch size. The patch dimensions were chosen in order to capture whole vine panels, shadow, and soil background pixels within each patch. Patch size is one of the customizable parameters in the DeepLabv3 module, enabling users to choose different dimensions based on the size of their crop unit.

The training process employed cross-entropy loss and an Adam optimizer with an initial learning rate of $2.5\times {10}^{-5}$ and a weight decay of 0.004 to optimize the network parameters. All other hyperparameters were kept at their default values (workers=4, batch_size=8). Model performance was evaluated using the mean intersection over union (mIoU) metric. Training was conducted for 1,000 epochs and the model weights achieving the highest mIoU score on the validation dataset were selected for testing. The segmentation performance of the selected model was then assessed on the test dataset.

For the hemp datasets, we re-trained the model using training and validation sets of 280 and 40 annotated 150 $\times$ 150-pixel image patches, respectively. We adjusted the batch number during training to accommodate the memory allocation of the computer GPU. We applied the re-trained model to the test set of 80 patches. For both the grapevine and hemp datasets, image patches were selected randomly from orthomosaics acquired in 2022 and 2023.

Hybrid method configuration

For hemp, the threshold to generate a coarse mask based on ExG values was set to 0.2 based on a manual visual inspection of masks resulting from various threshold values. Image pixels with ExG >0.2 were coded as 1 (foreground/crop), and pixels with ExG <0.2 were coded as 0 (background) (Fig. S1). Next, DBSCAN clustering was applied to the ExG mask to group nearby data points based on density, facilitating the identification of individual hemp plants. Tuning DBSCAN parameters (eps = 8, min_samples = 100 in this study) enabled adaptation to variations in plant distribution and field conditions, ensuring accurate clustering. The bounding box size was 160 $\times$ 160 pixels.

For grape, we made slight modifications to adapt the hybrid method. Because the vineyard did not follow a regular pattern of vegetation density clustering, we did not use DBSCAN to perform vine clustering before applying SAM-only or SAM $\times$ ExG segmentation. To implement SAM in grapevine, rather than use a point prompt based on DBSCAN clustering, we used the previously mapped panel geometries as bounding box prompts.

Performance evaluation

We used the mIoU metric to evaluate model performance. The mIoU is a statistic that quantifies how closely a model-generated segmentation mask matches the true, annotated mask for a set of test images and is the most widely used metric to evaluate accuracy in image segmentation [37]. The IoU was calculated according to Eq. 1, and the mean IoU was the average IoU for all test images. We used Student’s t test to evaluate the statistical significance of differences in mIoU scores between segmentation methods [38].

$$\begin{array}{c}\hfill IoU=\frac{AreaofIntersection}{AreaofUnion}\end{array}$$ 1

We also calculated F1, precision, recall, and accuracy for all crop masks in the test datasets. F1 is the harmonic mean of precision and recall and is calculated according to Eq. 2, where TP is the number of true positives (crop pixels correctly identified as crop pixels); FP is the number of false positives (non-crop pixels identified as crop pixels); and FN is the number of false negatives (crop pixels identified as non-crop pixels). Precision is the ratio of FP to the sum of TP + FP and recall is the ratio of TP to the sum of TP + FN.

$$\begin{array}{c}\hfill F1=\frac{2\times TP}{2\times TP+FP+FN}\end{array}$$ 2

Spectral trait extraction and comparison

We used the canopy masks generated by the segmentation methods described above to extract and compare the spectra of individual hemp plants and grapevine panels. The vineyard and hemp field layouts were as described in Table 1 and Fig. 2. For further details on the vineyard and hemp sites, we refer readers to Kanaley et al. (2024) [39] and Suchoff et al. (2024) [40].

For the grapevine and hemp case studies, we extracted 10-band reflectance from each plant (for hemp) or panel (for grapevine). The segmentation canopy masks have the same dimensions as the multispectral orthomosaics, so that multiplying each binary mask matrix by the image matrix yielded a new dataset containing reflectance for crop pixels only (Fig. 3). We used bounding box files corresponding to single plants (hemp) or multi-plant treatment groups (grapevine) to assign crop pixels to the appropriate experimental units. Hemp bounding boxes were generated in the DBSCAN step described above, while grapevine bounding boxes were generated with points collected using a mobile RTK GNSS receiver (SwiftNav, California, USA). The output was a CSV file with pixel coordinates and reflectance values for all experimental units. These CSV files were indexed by the unique identifier assigned to each hemp plant or grapevine panel so that any additional data collected for these units could easily be appended to the same file for downstream phenotypic analyses.

To compare the reflectance of individual plants (hemp) and plant experimental units (grapevine panels), we used the CSV files to calculate the normalized difference vegetation index, NDVI, for samples from each use-case. NDVI was computed according to Eq. 3, where NIR is near-infrared reflectance and red is red reflectance measured at band 5 and band 3 of the MicaSense RedEdge-MX^TM camera, respectively (Table S1). We calculated the Mann–Whitney U statistic [41] and corresponding p-value to test whether NDVI distributions were significantly different between samples. The nonparametric Mann–Whitney test was selected to account for the non-normal distributions of our reflectance datasets.

$$\begin{array}{c}\hfill NDVI=\frac{NIR-red}{NIR+red}\end{array}$$ 3

Hardware and software

All experiments were implemented on a Windows operating system with an NVIDIA Titan RTX GPU that operates at a base speed of 1350 MHz. Code for all models was written and executed in Python 3.11 using the packages OpenCV (v.4.9), NumPy (v. 1.26), scikit-learn (v.1.3), Pillow (v.9.5), GDAL (v. 3.6), Matplotlib (v.3.7), PyTorch (v.2.2.1), torchvision (v.0.15.2) and segment-anything (v. 1.0). We used the rasterio (v.1.3.9), rioxarray (v.0.15), and geopandas (v.0.14.2) packages to perform spectral trait extraction. Code to run all segmentation models is publicly available on the project GitHub: https://github.com/cu-cairlab/MAUI/tree/main.

Results

Evaluation of the different segmentation methods indicated that the unsupervised SAM$\times$ExG approach was most suitable for a wide-spaced row crop system (hemp), while the supervised DeepLabv3 model yielded the most accurate crop mask for the trellised vineyard. Our case studies further demonstrated that choosing an appropriate segmentation method enabled identifying spectral regions where reflectance values differ between individual plant units. These results highlight the importance of a customizable pipeline to facilitate downstream applications of UAS data.

Image preprocessing

For both the vineyard and hemp datasets, we observed substantial variability in co-registration accuracy when we relied on Agisoft Metashape (AgiSoft LLC, St. Petersburg, Russia) alone for automatic co-registration, despite the presence of twenty GCPs in each field. The AROSICS library was a more efficient and reliable method for accurate co-registration across dates. Although AROSICS was designed to co-register multispectral satellite imagery, we found it worked well for UAS images at fine (1–3 cm$·$pixel^-1) spatial resolution. Moreover, AROSICS is an inexpensive, accessible tool compatible with Linux, Mac, and Windows operating systems.

Segmentation model performance

Segmentation model performance varied significantly between crop case studies. In grapevine, the supervised DeepLabv3 method outperformed all unsupervised methods (Tables 2 and  4). In contrast, the unsupervised SAM$\times$ExG method achieved the greatest mIoU score for hemp (Table 2).

Table 2.

mIoU scores and standard deviations for supervised and unsupervised segmentation methods tested on grapevine and hemp image patches

		Segmentation mIoU$_{}^1$
Crop	Test Dataset	SAM	ExG	SAM$\times$ExG	Otsu’s thresholding	DeepLabv3
Grapevine Vitis vinifera ‘Chardonnay’	11 patches (512 $\times$ 512px)	0.16a ± 0.09	0.22a ± 0.10	0.17a ± 0.10	0.19a ± 0.10	0.85b ± 0.13
Hemp Cannabis sativa ‘Carolina Dream’ and ‘BaOx’	80 patches (150 $\times$ 150px)	0.89c ± 0.12	0.93c ± 0.05	0.95c ± 0.05	0.59d ± 0.17	0.80e ± 0.12

$_{}^1$ Superscripts for mIoU scores indicate significant differences based on the student’s t-test (p < 0.001). mIoU scores with the same superscript are not significantly different. Letters ^a and ^b correspond to grapevine data, while letters ^c, ^d, and ^e correspond to hemp

Table 4.

Average F1, precision, recall, and accuracy with corresponding standard deviations for grapevine crop masks. Metrics are for the test dataset of 11 512$\times$512 pixel patches

Model	F1	Precision	Recall	Accuracy
SAM	0.28 ± 0.14	0.18 ± 0.10	0.89 ± 0.22	0.39 ± 0.26
ExG	0.35 ± 0.16	0.23 ± 0.11	0.86 ± 0.09	0.56 ± 0.19
SAM $\times$ ExG	0.32 ± 0.15	0.22 ± 0.12	0.76 ± 0.22	0.57 ± 0.18
Otsu’s thresholding	0.30 ± 0.15	0.21 ± 0.12	0.69 ± 0.15	0.58 ± 0.14
DeepLabv3	0.77 ± 0.12	0.68 ± 0.15	0.94 ± 0.04	0.92 ± 0.03

Grapevine canopy segmentation

The DeepLabv3 model achieved an mIOU of 0.85 on the test grapevine dataset. Otsu’s thresholding, ExG-only, SAM-only, and SAM$\times$ExG yielded significantly lower (p < 0.001) mIoU scores of 0.19, 0.22, 0.16, and 0.17, respectively (Table 2). F1, precision, recall, and accuracy were also highest for DeepLabv3 masks for the grapevine test data set (Table 4). It is notable that the recall exceeded the precision for all segmentation methods in grapevine. This indicates that all models, including DeepLabv3, tended to misclassify non-crop pixels as grapevine. The DeepLabv3 mask correctly classified most of the vine pixels but occasionally failed to identify longer shoots extending beyond the canopy center, especially when these shoots appeared superimposed on the grass in the orthomosaic (Fig. 4a and b). In some cases light green foliage at the edge of the grapevine canopy was misclassified as background (Fig. 4c and d). We also observed an edge effect where soil and/or shadow close to the canopy boundary was misclassified as grapevine. Despite these errors, the DeepLabv3 model correctly identified most shadows cast by the vines as background, while still correctly classifying shadowed foliage within the main canopy as “grapevine” despite the darker color (Fig. 4). This is a major challenge in grapevine segmentation.

Fig. 4.

DeepLabv3 vine mask for the UAS image acquired at the Cornell Pathology chardonnay block on 2023-08-16 (a). Insets in (b, c) and (d, e) show highlighted sections of the mask and corresponding RGB composites for rows 1 and 2 and rows 6 and 7, respectively

Hemp plant segmentation

The mIoU for SAM-only segmentation was 0.89, while the ExG-only method achieved an mIoU of 0.93. When combined (SAM$\times$ExG), the mIoU improved to 0.95 (Table 2). These results were obtained without any additional fine-tuning of the pre-trained SAM model. In contrast, Otsu’s thresholding and the DeepLabv3 model yielded significantly lower mIoU values (p < 0.001) of 0.59 and 0.85, respectively. F1 was also highest for the SAM $\times$ ExG method, though the differences in F1, precision, recall, and accuracy of different methods were not as large for the hemp dataset as for grapevine. We also observed smaller differences in recall vs. precision in the hemp dataset. Otsu’s thresholding yielded higher precision than recall, indicating that this method was slightly better at correctly classifying non-crop pixels than crop pixels in hemp, whereas we observed the opposite in grapevine. F1 and accuracy scores were higher for all methods for the hemp dataset than for grapevine (Table 3).

Table 3.

Average F1, precision, recall, and accuracy with corresponding standard deviations for hemp crop masks. Metrics are for the test dataset of 80 150 $\times$ 150 pixel patches

Model	F1	Precision	Recall	Accuracy
SAM	0.93 ± 0.08	0.90 ± 0.14	0.98 ± 0.03	0.98 ± 0.02
ExG	0.96 ± 0.03	0.95 ± 0.04	0.98 ± 0.03	0.99 ± 0.01
SAM $\times$ ExG	0.97 ± 0.03	0.98 ± 0.04	0.97 ± 0.04	0.99 ± 0.01
Otsu’s thresholding	0.74 ± 0.17	0.97 ± 0.08	0.62 ± 0.18	0.93 ± 0.05
DeepLabv3	0.88 ± 0.09	0.86 ± 0.12	0.92 ± 0.05	0.97 ± 0.01

The combination of SAM and ExG effectively segmented the plant target, as demonstrated in Fig. 5. This method was particularly well suited for a wide-spaced horticultural system where black plastic mulch and weed fabric provided a homogeneous background and clear contrast in reflectance with the crop. ExG also enabled efficient separation of shadows cast by plants and the plant foliage (i.e. separating non-target and target). However, in some areas where neighboring plant canopies overlapped, the SAM$\times$ExG segmentation yielded a single mask encompassing both plants, failing to separate the two neighbors (Fig. 6). This limitation would be most apparent in fields with dense vegetation or irregular plant spacing.

Fig. 5.

Segmentation performance of the ExG, SAM, and SAM$\times$ExG methods on two hemp plants. In (a) a non-target weed, indicated by the black arrow, is misclassified as hemp by ExG and accurately classified as background by the SAM and SAM$\times$ExG methods. In (b) SAM incorrectly identifies non-target plastic mulch (indicated by the black arrow) as part of the plant, whereas ExG excludes the mulch

Fig. 6.

SAM$\times$ExG hemp mask for the UAS image acquired on 2023-08-10 at the Cornell wide-spaced hemp breeding trial (a). Insets in (b, c) and (d, e) show highlighted sections of the mask and corresponding RGB composites for rows 7 and 10 and rows 9 and 12, respectively

Effects on spectral extraction

Accurate vine pixel classification enabled comparison of spectral traits between treatment groups. Figure 7a illustrates the contrast between spectra extracted for two different grapevine panels using the DeepLabv3 method. The panels show differences in red and NIR reflectance, regions of the electromagnetic spectra known to correlate with plant biochemical and structural traits [42, 43]. The normalized difference ratio of red and near infrared (NDVI) distributions for the two panels are statistically different regardless of which segmentation method was used, but there are notable differences in the inter-quartile ranges (Fig. 7b, capped lines) for the different methods. The decreased variance in NDVI range for DeepLabv3 is likely caused by the lower contribution of soil and shadow pixels to the reflectance signal due to more accurate crop segmentation. For comparison purposes, spectra extracted via the additional segmentation methods are included in Fig. S3a–d.

Fig. 7.

Spectra extracted for individual grapevine panels (a) and hemp plants (c) using the DeepLabv3 and SAM$\times$ExG segmentation methods, respectively. b and d show the corresponding NDVI distributions of pixels isolated by each method. The row label in b is the row number assigned to two different grapevine panels. The plant_id label in d indicates the number assigned to two different hemp plants. Asterisks indicate statistically significant differences (p <0.001, Mann–Whitney test) in the distributions of NDVI values for the two samples in each case study. Numerical suffixes on the x-axis correspond to the center wavelengths for each waveband. Spectra were extracted from UAS images acquired on 2023-08-16 (grapevine) and 2022-07-26 (hemp)

The SAM$\times$ExG segmentation achieved the highest mIoU score and enabled clear spectral differentiation between plant units. In Fig. 7c, we observe differences in both the magnitude and slope of the reflectance profiles of the two hemp plants. All segmentation methods facilitated detecting significant differences (Mann Whitney test, p < 0.001) in NDVI between individual plants, though we again observed different degrees of variance in the NDVI distributions as indicated by the inter-quartile ranges (Fig 7d). Taken together with our findings in grapevine, this result demonstrates how accurate crop segmentation can facilitate monitoring and comparing individual plants in large field trials. Simplifying the extraction of plant spectra allows domain scientists to more readily link reflectance data to potential biochemical and biophysical drivers of the observed spectral differences, a critical step towards using UAS imagery to enhance the speed, scale and accuracy of specialty crop phenotyping.

Discussion

Since the emergence of field-based high-throughput phenotyping around 2014, uncrewed aerial systems (UAS), particularly consumer-grade drones, have become a popular tool for researchers to collect field imagery datasets quickly and affordably. Numerous studies have demonstrated the efficiency and accuracy of UAS-based phenotyping for applications including yield prediction in winter wheat [44], plant height estimation in sorghum [45], and quantifying nitrogen stress tolerance in maize [46]. Importantly, these and other studies have found that breeding programs can improve their selection processes by combining genotypic data with the detailed phenotypic information that UAS provide across large-scale field trials [44, 47].

Despite these successes, integrating UAS imaging and analysis into routine research for specialty crops faces significant barriers. Chief among these is the need for a streamlined protocol encompassing data acquisition, plant segmentation, and export of reflectance traits to enable downstream analysis [48]. MAUI was developed to address these challenges and pave the way for seamless integration of UAS into routine research projects, particularly for specialty crops. The system is structured around modular components that define the overall processing pipeline, enabling the standardization of UAS-based data collection and analytical workflows. This structure provides users with the flexibility to incorporate their preferred methods or the latest technologies for optimal performance. The inclusion of two different end-to-end use case examples differentiates MAUI from previously published open-source UAS image processing pipelines for plant science applications. Although these pipelines provide robust solutions for single crops, MAUI enables analyzing image data for trellised or row-cropped horticultural systems within the same framework. This meets an important need for interdisciplinary researchers, extension specialists, and industry professionals who aim to deploy the same UAS system to monitor a variety of crops.

Image preprocessing

Although we used Agisoft Metashape (AgiSoft LLC, St. Petersburg, Russia) to generate orthomosaics from the raw MicaSense images, this is not the only image-processing software available. Pix4D (Pix4D S.A., Prilly, Switzerland) is another widely used commercial photogrammetry software, though like Agisoft Metashape (AgiSoft LLC, St. Petersburg, Russia), users pay a subscription fee. Researchers have also used OpenDroneMap, an open-source platform, to orthorectify and mosaic UAS imagery with good results [24, 25, 49]. While institutions with access to subscription-based platforms may prefer proprietary software for ease-of-use and additional functionalities, open-source alternatives like OpenDroneMap and COLMAP offer comparable basic options at no cost [49, 50].

Crop segmentation in grapevine and hemp

Though previous studies have developed custom segmentation algorithms for grapevines in Mediterranean climates with great success [8, 11, 51], we set out to demonstrate an approach that is suitable for grapevines and trellised crops in cool and temperate regions characterized by substantial non-crop vegetation within the field. In these areas, segmentation algorithms must be able to distinguish between multiple types of plants. As we show here, grapevines often exhibit similar color features to grass or cover crops in UAS imagery, making it challenging for unsupervised methods to distinguish between them accurately. Unlike hemp fields, where wide inter-plant spacing and black weed fabric facilitates density-based clustering to isolate single plants, in trellised systems the canopies of neighboring plants can overlap significantly and are often indistinguishable (see Figs. 4 and 5). In addition, canopy size and shape changes rapidly as plants mature, especially when nutrients and rainfall are plentiful, and the canopy can become dense as shoots of neighboring plants intertwine. Trellis architecture also impacted canopy appearance. The vineyard in this study was planted on a vertical shoot positioned trellis, which compressed the canopy area to a narrow strip of foliage when viewed from above. Other trellis architectures, such as pergola or tendone, allow shoots to drape on either side of the wires, resulting in increased canopy width in aerial imagery [52]. In vineyards in dry climates, where inter-row vegetation is sparse or nonexistent, segmentation accuracy may improve due to higher contrast between bare soil and vine. The climatic and agronomic characteristics of our vineyard site motivated the decision to perform a supervised semantic segmentation, incorporating patterns at multiple spatial scales to capture more contextual information and segment the vineyard dataset.

Here, the DeepLabv3 model performed well at isolating groups of vine pixels adjacent to inter-row vegetation, though the method still struggled in some edge cases as illustrated in Fig. 4. Most misclassifications occurred when longer shoots overlapped with grass between the vineyard rows, possibly due to confusion between similar green vegetation features. Within the grapevine canopy, the DeepLabv3 model did not exclude shadowed vine pixels from the final mask. Although these pixels correspond to the target crop, it is worth noting that their reflectance profiles will alter the aggregate vine spectra by decreasing the magnitude of reflected light. These are important factors to consider for downstream analyses, especially when groups of pixels are averaged (Fig. 7) to calculate spectral traits of different plant units. The reflectance of soil pixels, for example, increases consistently from the visible to the near-infrared, while vegetation reflectance has characteristic absorption features (low reflectance) in the 650–680 nm (red) and 400–500 nm (blue) regions of the spectrum where chlorophyll molecules absorb solar radiation more strongly [53]. Thus, including soil pixels that were misclassified as crop will alter reflectance profiles, potentially limiting the ability to accurately detect plant responses to different biotic or abiotic factors in phenotyping trials. Previous work on winter wheat demonstrates that background soil reflectance accounts for up to 40% of the variability in NDVI of the crop canopy, and these variations are also heavily influenced by soil type [54]. Similarly, misclassified cover crop pixels can impact the accuracy of studies that aim to quantify and compare responses of different experimental units. For example, if a model misclassified clover as grapevine in one section of a vineyard and misclassified rye as grapevine in another area, the reflectance traits of the two sections may differ because of the variability introduced by the reflectance of two different cover crops, rather than by differences in grapevine phenotypes.

Prior work indicates that CNN model performance, though robust for datasets with varying illumination conditions and complex backgrounds, depends on the size and quality of the labeled training data [10, 15]. Thus, one way to maximize the accuracy of the current pipeline for a given crop type is to prepare a set of high-quality training image patches. We use a set of 35 labeled vineyard image patches to highlight the high accuracy attainable with a small input dataset. Training on a larger dataset that includes more regions of vine and grass overlap may further reduce misclassifications. However, the need for high-quality annotated training data also increases the time and labor required to develop an accurate supervised segmentation model. Although it requires more human annotation effort than threshold-based techniques and vision foundation models, the supervised DeepLabv3 model was the most reliable option for vineyard canopy segmentation in this study.

Conventional segmentation methods like ExG thresholding achieve their low computational cost by relying on RGB imagery to identify crops. While appropriate in some cases, this may limit segmentation accuracy in variable lighting conditions or when plant health is compromised by nutrient deficiency, drought, or disease, all of which can change the ratios of red, green and blue light reflected by the plant canopy. Studies that leverage the NIR and red-edge bands to segment different vegetation types from non-vegetation backgrounds achieve high performance due to the sensitivity of these bands to plant tissue [11, 51]. Additionally, when solar illumination conditions vary substantially throughout a flight, the contrast between vegetated and non-vegetated surfaces is more stable in the near-infrared than in the visible range [55].

Users seeking to improve the custom vision foundation model presented here can use different prompting strategies or combine SAM’s outputs with additional segmentation steps. For instance, integrating different spectral bands or other vegetation indices to determine an optimal crop segmentation threshold could improve model robustness across different environmental conditions. It is notable that, though Otsu’s thresholding does incorporate the NIR band, this method did not perform as well as the ExG thresholding approach, with a drop in mIOU of 0.36 and 0.64 compared to the best-performing methods for hemp and grapevine, respectively (Table 2). This finding further highlights the importance of tailoring image segmentation techniques to the target crop.

Computational resources

While learning-based segmentation models provide highly accurate results, they typically require significantly more computational resources than conventional computer vision algorithms, such as threshold-based segmentation. However, advancements in consumer-grade general-purpose graphics processing units (GPGPUs) and corresponding software support (e.g. NVIDIA CUDA and PyTorch) have made it possible to train and deploy modern deep learning models (e.g. DeepLabv3) on desktop computers. In this study, an NVIDIA Titan GPU was used to train and evaluate DeepLabv3 for plant segmentation in grapevine and hemp research. As a result, these models are expected to be largely accessible to plant researchers with minimal computational constraints.

Deploying a vision foundation model like SAM can also be achieved using consumer-grade GPUs, eliminating the need for specialized hardware. However, training or fine-tuning foundation models demands substantial computational resources, often requiring hundreds to thousands of GPUs running for weeks or months. Such resources are typically available only through specialized computational infrastructure, such as cloud-based GPU servers, which can pose a barrier for researchers seeking to train or fine-tune these models. Our experiments indicate that with appropriate image preprocessing using conventional computer vision techniques, foundation models can achieve satisfactory segmentation accuracy. This suggests that plant science researchers may not need to worry about the computational burden of foundation model training, but rather should focus on ensuring sufficient resources for inference. We provide inference times for the DeepLabv3 and SAM$\times$ExG models applied to vineyard and hemp orthomosaics in Table S3.

MAUI represents a starting point for advancing specialty crop research and breeding and lays the groundwork for further development. Although we highlight DeepLabv3 and SAM-based approaches to canopy segmentation, conventional methods such as support vector machines and k-means clustering are viable, well-studied alternatives with lower computational requirements. These can be readily incorporated into the MAUI pipeline by modifying the segmentation module presented here, or by adding custom modules to implement and compare the different segmentation techniques on the same datasets. Future studies can also investigate computational efficiency and scalability of the tested models under a broader range of field conditions. Assessing crop segmentation performance for thermal imagery datasets would be another fruitful area for further investigation. To promote accessibility, MAUI is containerized alongside an open-source codebase and documentation. This approach enables users, particularly plant scientists, to easily deploy MAUI and customize the pipeline to meet specific needs. Our next steps include developing a graphical user interface (GUI) for MAUI to further simplify integrating UAS imaging and analysis into routine workflows.

Conclusion

We developed a customizable pipeline for specialty crop researchers to facilitate rapid and accurate spectral data extraction. The pipeline accounts for the diverse structure of specialty crop production systems and ensures that remotely sensed data can be appropriately attributed to the target crop while minimizing noise from the surrounding environment. Our image segmentation workflow handles agronomic heterogeneity and takes advantage of inherent spatial patterns so that researchers can accurately identify plants of interest. To demonstrate the utility of this pipeline for domain scientists, we show that our workflow enhances the ability to isolate and compare spectra of individual crop units in grapevine and hemp. By standardizing and streamlining image processing and spectral extraction, we provide a tool to allow users to easily incorporate UAS imagery in specialty crop research.

Abbreviations

CNN

Convolutional neural network

DBSCAN

Density based spatial clustering of applications with noise

DEM

Digital elevation model

ExG

Excess green index

GCP

Ground control point

HSV

Hue-saturation-value

mIoU

Mean intersection over union

NIR

Near infrared

NDVI

Normalized difference vegetation index

RGB

Red-green-blue

RTK

Real time kinematic

RGN

Red-green-near infrared

SAM

Segment anything model

UAS

Uncrewed aerial system

Author contributions

MM and KK collected all UAS datasets, generated orthomosaics, and contributed equally to tables and figures. MM tested ExG, SAM, SAM$\times$ExG and DeepLabv3 models on the hemp dataset. TQ wrote the code for ExG, SAM, and SAM$\times$ExG segmentation and mIoU calculation and the corresponding methods section. EL wrote the code for DeepLabv3 segmentation and the corresponding methods section. SS created the Docker environment for all segmentation modules and organized the entire codebase for publication on GitHub. DS built the UAS imaging system and trained KK and MM on its operation. KK performed image co-registration, wrote the code for Otsu’s thresholding method, tested all segmentation methods on both vineyard and hemp datasets and was the primary contributor to manuscript drafting. All authors read and approved the final manuscript.

Funding

This research was supported in part by the intramural research program of the U.S. Department of Agriculture, National Institute of Food and Agriculture (USDA NIFA), Hatch accession no. 7003024 and by the USDA NIFA Small Farms Program, Grant no. 2022-68006-36148. MM was supported by a National Science Foundation NRT Training Grant in Digital Plant Science, Grant no. 1922551. This work was partially supported by funding from the New York State Department of Agriculture and Markets Grant no. CM04068, by the Foundation for Food and Agriculture Research, Grant no. 22-000414, and by the Cornell Institute for Digital Agriculture Research Venture Fund.

Data availability

The image datasets used during the current study are available from the corresponding author on reasonable request. All code to implement the image segmentation methods is publicly available on GitHub at: https://github.com/cu-cairlab/MAUI/tree/main

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no Conflict of interest.