Skip to main content
NCBI home page
Applications in Plant Sciences logoLink to Applications in Plant Sciences
. 2026 Oct 1:e70085. Online ahead of print. doi: 10.1002/aps3.70085

Petal to the metal: The slow road to automating large‐scale phenology labeling for herbarium specimens

Erin L Grady 1,✉, Raphael LaFrance 1, Daijiang Li 2, Russell Dinnage 3,4, Ellen G Denny 5, John Deck 6, Robert P Guralnick 1
PMCID: PMC13628181  PMID: 42824846

Abstract

Premise

Herbarium specimens represent critical historical records of plant phenology, yet automating annotation of reproductive structures remains challenging given the diversity of floral morphologies, specimen age and quality, and image quality.

Methods

Here, we present a machine learning pipeline that uses an ensemble modeling approach to detect flowers on herbarium specimens and deliver these data to the phenology research community. After testing multiple strategies for generating training data, we found that in‐house expert‐curated annotations were essential for producing reliable results.

Results

Expert validation found that the modeling pipeline detected present floral structures with relatively strong accuracy but had moderately high false negative rates. Applying the ensemble to our filtered final image dataset of 22 million records resulted in 11.1 million records labeled with flowers present. However, only 2.9 million of these contained the complete metadata necessary for downstream phenology research, highlighting the need for full label digitization efforts.

Discussion

The dataset presented here represents a large compilation of historical herbarium‐derived phenology records available as a resource for the phenology community. We end by demonstrating how integrating these machine‐labeled records into Phenobase, a publicly available phenology database, expands taxonomic and temporal coverage for large‐scale phenological analyses.

Keywords: computer vision, data integration, herbarium specimens, machine learning, plant flowering, plant phenology

Natural life cycle events, such as plant flowering, fruiting, and leaf‐out, occur annually on seasonally defined schedules shaped by environmental cues. The timing and sequence of these events, which vary by species and environment, underpin many biological and human processes, including species interactions (Kudo et al., 2004), carbon cycling (Richardson et al., 2013), resource acquisition (Deacy et al., 2017), agriculture (Yadav et al., 2023), tourism (Nagai et al., 2019), and allergy management (Manangan et al., 2021). Phenology, the study of the timing of these seasonal events, is a branch of inquiry dedicated to identifying their environmental drivers and the consequences that arise when these processes are disrupted.

A fundamental question in phenology is how different species and ecosystems have responded and will continue to respond to accelerating human‐driven environmental change. The rate and direction of these phenological shifts are of particular interest as they are often considered one of the first and best indicators of broader ecological disruption (Menzel et al., 2006). Such shifts can interfere with key interactions that require phenological synchrony, such as plant–pollinator relationships, and can have cascading consequences for species' population health, ecosystems, and people (Visser and Gienapp, 2019). Thus, access to data capturing the timing of key life stages across time, space, and taxa is increasingly of interest and importance across many scientific and applied disciplines.

Concerted phenological monitoring has been a key means to generate data needed to document phenological change. However, many regions across the globe do not have phenology monitoring programs, and in areas that do, the temporal extent is often limited. For example, the USA National Phenology Network began coalescing observational monitoring in 2007 (Crimmins et al., 2022), but national‐scale phenology data before this date are limited and often of short duration (with some exceptions; see, for example, Hough's efforts from 1851–1859; Guralnick et al., 2024). Beyond direct phenological monitoring, there are new resources, such as iNaturalist (https://www.inaturalist.org/), producing digital vouchers from which phenological information can be secondarily derived (Dinnage et al., 2025). Such contemporary contributory science platforms, like iNaturalist, harness community members across the globe to generate large quantities of observations linked to images and now include means to annotate phenological states. While these resources are helping to further fill spatial and taxonomic data gaps and providing new opportunities for phenology research at scale, they are even more limited in temporal extent compared to monitoring data.

A critical component for defining and predicting phenological change is understanding phenology and its trends over decadal to century‐level timescales. Herbaria contain specimens dating back hundreds of years and have the potential to offer valuable historical phenology data, complementing contemporary monitoring initiatives (Davis et al., 2015; Willis et al., 2017). Recent efforts to image and digitize herbarium specimens have allowed for broader use of these plant collections (Soltis, 2017), with information on over 108 million angiosperm specimens (~40 million with associated images) available on the Global Biodiversity Information Facility (GBIF; https://www.gbif.org/) as of October 2025. Ideally, each of these herbarium specimens contains valuable information about the location, collection date, species identity, and occasionally the phenological state of that plant individual. For the vast majority of plants lacking phenology annotations, it requires significant effort to classify which, if any, phenological phases are visibly present on specimens.

Automating phenology annotation for digitized herbarium specimens is a clear next step, one that would unlock vast amounts of historical data and enable phenological change research across diverse taxa and environments. Recent machine learning efforts to classify phenological phases in photos of live plants have been successful (Dinnage et al., 2025); however, herbarium specimens present a unique set of challenges. Dried and pressed plant specimens often lose color and structure over time, reproductive structures can be lost during processing, some taxa have structures that are undetectable from digitized images alone or without dissection, and specimens contain non‐plant material such as labels and herbarium stamps that may interfere with machine detection (Pearson et al., 2020). Additionally, the accuracy of machine phenology annotation is highly dependent on the annotator who creates the training data used to calibrate models, and curating a reliable and taxonomically diverse training dataset requires extensive effort and careful consideration.

Several works have applied machine learning to herbarium specimens, with greater success in species identification than phenological recognition tasks (Hussein et al., 2022). Phenological efforts have often focused on small, curated species and image sets (Goëau et al., 2020; Younis et al., 2020), and many have reported challenges annotating specific phenological structures such as flowers (Lorieul et al., 2019; Younis et al., 2020). More recent efforts have successfully addressed specific phenology research questions using models trained by compositing many datasets from multiple sources (Williamson et al., 2026). Our aim here, however, was not to produce a standalone research product or answer a particular question, but to develop a trustable, usable, and integrable data resource for the broader research community. Doing so required careful curation of our own training data, which allowed us to leverage information gathered during training to reduce noise in the final data product.

Here, we present an automated phenology annotation pipeline that accurately detects floral structures present on digitized herbarium specimens. With the goal of maximizing geographic and taxonomic coverage across the angiosperm tree of life, this effort unlocks vast quantities of previously limited historical phenology information. We outline the challenges, successes, complexities, and strategic decisions involved in developing the pipeline, emphasizing the need for high‐quality training data generated by botanically knowledgeable individuals and rigorous pre‐ and post‐modeling filtering. We discuss the tradeoff between strict data filtering for improved model accuracy and data retention to maximize the volume of downstream historical records annotated with phenology information. We highlight a surprising and disappointing discovery that the majority of specimens lack digital metadata critical for phenological annotation, such as date and location. Finally, we demonstrate the vast scale of previously unavailable historical phenology data that are now available to stakeholders through this work and discuss the integration of these data into Phenobase (Phenobase Collaboration, 2026), an online database of millions of research‐ready, community‐sourced phenology data from field images and from structured in situ phenology monitoring programs.

METHODS

Gathering specimen image and metadata

We opted to gather data for this work from GBIF, which provides access to billions of species occurrence records and, in many cases, associated images. We searched GBIF in mid‐2023 for all occurrences of “Magnoliopsida” and “Liliopsida,” using filters to select the basis of record as “preserved specimen” and media type as “images.” This produced 33,691,939 occurrence records with 35,314,854 corresponding multimedia records. We did not filter out records that were missing coordinates or location information so that when the metadata of a record is updated in the future, its associated phenological information will have already been extracted. We downloaded this dataset and associated images in October 2024 and, using a nearest pixel filter, programmatically resized each image to 1024 pixels along the shortest edge to reduce storage and satisfy graphics processing unit (GPU) memory limitations. Because automated services to download images are often down and media links can change, the number of images downloaded is a subset of the total number of records in the original GBIF dataset. In sum, after contending with link issues and removing small or low‐quality images and images associated with taxa that we ultimately decided to filter out (described below), we collected 22,447,563 images for downstream labeling and model training.

Labeling training data

A challenging aspect of developing a reliable automated pipeline for phenophase annotation on herbarium specimens across the angiosperm tree of life was generating enough high‐quality training data. Our goal was to produce a trusted historical phenology data resource for broad use, and we therefore deliberately chose to manually generate our own training data rather than utilizing data from multiple external sources. As mentioned above, manually scoring pressed plant specimens for phenophases is time consuming and difficult, and a key challenge is balancing the need for botanical expertise with the capacity of expert annotators. Other initiatives have employed volunteers, undergraduate students, and paid non‐experts with relative success (Zhou et al., 2018; Brenskelle et al., 2020); however, these studies focused on a few key taxa with easily distinguishable structures. Limiting annotations to a few taxa at the genus or family level allows for more in‐depth, targeted training and familiarity with target taxa, which likely leads to better results. We were unsure whether employing non‐experts would be successful when scoring thousands of diverse taxa.

We first tested whether a paid undergraduate student could produce accurate phenophase annotations across all flowering plants. We employed a student to score a random selection of 3525 specimens for the presence of open flowers. If the student was uncertain or unable to make an informed decision about the digitized image, they had the option to mark “unknown.” To evaluate the accuracy of the undergraduate's annotations, we had a botanically knowledgeable individual validate their annotations. We found the undergraduate's open flower annotations were not reliable for model training, as they were only in agreement with the expert annotations 69% of the time. Based on these results and other work that suggests that the knowledge of the annotator is one of the most important variables in annotation accuracy (Brenskelle et al., 2020), we made the conscious choice to employ one in‐house botanically knowledgeable individual to generate all training data. This decision greatly reduced our capacity to generate high quantities of training data, but ensured high quality and consistency, which was our priority.

In addition to the first set of specimens that was annotated to validate the undergraduate annotations, the expert annotated a second additional set of 5479 specimens for open flowers to increase the amount of training data. While compiling the second set, we randomly selected specimens in families that were underrepresented in the first set in an effort to promote taxonomic evenness. Additional records were added to the training data iteratively through repeated rounds of image validation throughout model training. In total, the expert annotator generated 9004 annotations for training.

Filtering training data

To ensure accurate and reliable results, we opted to filter out taxa that were challenging to manually annotate before model training. First, we removed genera that had more than five records in the training dataset and an “unknown” annotation rate greater than or equal to 25%. After removing these genera, we removed families following the same criteria. Many of these removed taxa had small reproductive structures, such as Chenopodiaceae, or had flowers and fruits that are difficult to distinguish from each other in images alone, like Poaceae. To ensure data quality, we chose to filter these taxa from the pipeline entirely. Throughout this process, we found it critical to acknowledge and respond to limitations in human expertise, machine capabilities, and overall capacity. The final lists of removed genera and families are available in Appendix S1 (see Supporting Information with this article).

Model training and post‐training thresholding

Model ensembling approach

One challenge with training models on all herbarium data is the vast and complex floral diversity in angiosperms, which challenges models to properly abstract features that can apply broadly and that are not easily confused with other objects. While single models often demonstrated modest performance, we postulated that an ensemble of models might perform better than any single model given this variability. We tried several different neural networks, architectures, and model parameters, and ultimately determined that an ensemble modeling approach, taking the consensus of three models, was most effective based on downstream validation statistics. Note that all three models were fine‐tuned versions of models that were previously pretrained on ImageNet data (Deng et al., 2009).

Despite differences across the three models (EffNet‐CNN [Thanikachalam et al., 2025], a vision transformer model trained using mean squared error [ViTMSE], and a vision transformer model trained using cross‐entropy [VITCE] [Dosovitskiy et al., 2021], summarized in Table 1), all training followed the same general steps. First, expert‐labeled records were split into 60% training, 20% testing, and 20% validation sets. After low‐quality images were filtered out, the images were resized to match the size requirements of the specific model (Table 1). Images in the training dataset were augmented using random horizontal and vertical flips and AutoAugment (Cubuk et al., 2018), which uses multiple techniques including random rotation, color jitter, sharpness adjustment, and image shear. This was done to diversify training examples, improve model generalization, and reduce overfitting. Next, all models were color adjusted to use the ImageNet color mean and standard deviation (Deng et al., 2009). Each model was saved to five checkpoints (i.e., interval‐based snapshots of model weights) during training. The final best model was chosen as the checkpoint with the best F1 score on the holdout validation dataset. The F1 score is the harmonic mean of precision (how many positive model predictions are actually positive?) and recall (how many actual positives did the model detect?). We chose the F1 score because it is particularly useful for assessing predictive performance for the positive class (“flowers present”), which is the focus of this effort. This overall approach is standard for model selection and ensures comparable scoring across model approaches (e.g., EffNet‐CNN versus ViT) and that models are tuned, selected, and ensembled based on their best validation performance.

Table 1.

Description of the three models included in the pipeline. The threshold values are cut‐offs applied to each model's continuous output probabilities.

Model typea Image size requirements Threshold low Threshold high Loss function
EffNet‐CNN 528 × 528 pixels 0.05 0.95 Mean squared error
Visual transformer network (ViTMSE) 384 × 384 0.5 0.7 Mean squared error
Visual transformer network (ViTCE) 384 × 384 0.5 0.7 Cross‐entropy
Open in a new tab
a

The difference between the two vision transformer models is the loss function used. Mean square error loss functions are more akin to confidence scores and are typically Gaussian, while cross‐entropy is better suited for classification tasks and tends to skew continuous probabilities more strongly to 0 or 1.

We ran models using a single A100 GPU on the University of Florida's HiPerGator. Each model was trained for at least 200 epochs (i.e., full passes through the training data). After each epoch, model performance was evaluated on a held‐out validation set to monitor generalization to unseen images and select the best‐performing checkpoints (see above). We used several metrics for evaluating the model, including F1 score, accuracy, validation set loss, precision, and recall. After training, each checkpoint from all three models was post‐processed to determine the probability thresholds that maximized accuracy. We divided the range (0.0–1.0) into three regions defined by two thresholds. Images with scores above the high threshold were considered a positive indication of the trait, while images with scores below the low threshold were considered a negative indication of the trait. Images with scores between the high and low thresholds were classified as “equivocal,” meaning the model could not decide whether the trait was present or absent. We adjusted those two thresholds until maximum accuracy was reached for the images in the test (holdout) dataset. We also guarded against equivocal records constituting more than 30% of the test dataset. We used a batch size of 32, an initial learning rate of 5e‐5, a weight decay parameter of 0.01, and an AdamW optimizer (Loshchilov and Hutter, 2019).

For each image, each model generated a final outcome of positive, negative, or equivocal. To ensemble our three models, we used a two‐thirds “majority rule” approach to apply a final label to each image. Images classified as positive by two or more models were labeled as “present” and those classified as negative by two or more models were labeled as “absent.” Note that the ensemble itself can be undecided (equivocal), for instance, if images are classified as positive, negative, and equivocal by the three models or receive two or more equivocal votes from the three model outputs. Images in those scenarios were labeled as “equivocal.”

Model validation

It is important to note that, because herbarium specimens often contain only a portion of a plant individual and absence of open flowers on the whole plant cannot be inferred, we focused primarily on the accuracy of “present” labels. For this reason, only images machine‐labeled as “present” were retained in the final dataset published to Phenobase, described below. We mention this here to clarify that the choices made in our methodology are grounded in the emphasis on “present” data only.

We validated ensemble results both using held‐out test data and through additional expert validation. The held‐out test data included 1441 specimens labeled present or absent for open flowers. These test data validation results are critical for understanding how well the ensemble performs at the task it was trained to do. For expert validation, we machine‐labeled a random subset of images (not included in training) and one author (E.L.G.) validated 600 specimens that were machine‐labeled as “present.” When conducting this expert validation, we opted to validate “present” data at a broader phenophase level (i.e., “flowers”). Consistent with the Plant Phenology Ontology (Stucky et al., 2018), “flowers” here refers to any floral structure, including both flower buds and open flowers. Although the models were originally trained on “open flowers,” we opted to conduct expert validation at the broader “flower” level, as these tasks are difficult and we thought that “flower” better reflected the features detected by the ensemble models. This is appropriate only because the final dataset exclusively includes images labeled as “present.” It is therefore in the best interest of data users to validate these images at a level in the ontology that yields the most accurate results. As mentioned below, this distinction and broader definition of “flower” is made clear to data users in Phenobase. Lastly, to further understand instances where the model consensus was incorrect, we anecdotally looked at false positives and false negatives to understand general patterns in the mistakes (see the Results section, below).

Machine labeling and integration with other sources

After model validation, the remaining unlabeled images (from the downloaded set of 22,447,563 images) were passed through the ensemble model pipeline. We retained only images labeled as “present” by at least two out of three models and the data fields were standardized for integration in Phenobase, which has a defined set of required and optional fields. As we discuss below, we then dropped records that had been labeled but were missing these key fields that are required for phenology research. Key fields here include scientific name, event date, and decimal latitude and longitude. The remaining data were made publicly available through the Phenobase web portal (Phenobase Collaboration, 2026). Data labeled as “present” but missing key fields are publicly available on Zenodo for potential future use once the necessary metadata are digitally transcribed (see Data Availability Statement).

Evaluating specimens missing critical fields

A key challenge and major limitation during this process was that a large number of specimens did not include date or location information in their digital metadata. To further investigate this issue, we randomly selected 100 specimens labeled as “flowers present” that were missing both date and location information and then manually annotated these specimens for whether or not they included specific (day, month, and year) date information and georeferenceable locality information on the specimen label.

RESULTS

Labeling training data

The lead author (E.L.G.), who served as our botanical expert, generated a total of 9004 annotations for open flower presence on individual specimens. After removing 49 genera and an additional seven families (Appendix S1), 1660 genera and 261 families remained. Our final training dataset contained 7205 specimen records, 4319 annotated as open flowers present and 2886 annotated as open flowers absent. These records represented 261 plant families; the plant families represented in our training dataset, along with the proportion of training data from each family, can be found in Appendix S2. Once expert‐labeled records were split, 4324 images were included in the training set, 1442 in the testing set, and 1439 in the validation set.

Model fitting and validation

We utilized three models (Table 1) and ensembled their results to predict the presence of flowers. Our training model accuracy scores across all three models varied between 88.3% and 92.5%. In general, the ViT models performed better (92.2% and 92.5% accuracy) than the EffNet‐CNN model (88.3% accuracy). Overall, from the full ensemble, validation on the held‐out test data found 96.3% accuracy for detecting present open flowers and 86.8% accuracy for detecting absent open flowers (Appendix S3). Expert validation on records that were not in the initial training, testing, or validation processes found the ensemble models to be 90.5% accurate for detecting present flowers. The discrepancy between held‐out test data and expert validation, while surprising because expert validation was conducted with the broader “flower” definition (described above), was likely due to expert‐validated data being more diverse than training data. Investigation of false positives from the expert validation found that the majority of false positives were fruits mistaken for flowers, although some sheets had no reproductive structures present and one sheet had a fruit illustration (Figure 1). Investigation into false negatives found a variety of reasons for the mistakes, including situations where flowers were not attached to the main plant, flowers may have been confused as fruits, or flowers were challenging to see due to factors like lighting or flower size (Figure 2).

Figure 1.

Figure 1

Open in a new tab

Examples of false positives, situations where the ensemble model consensus scored the image as “flowers present,” but there are no flowers present in the image. (A, B) Examples where fruits were confused for flowers. This was the most common cause of false positives, accounting for over half of cases. (C) An example where leaves were confused for flowers. Around 30% of false positives were situations where vegetative structures were confused for flowers.

Figure 2.

Figure 2

Open in a new tab

Examples of false negatives, situations where the ensemble model consensus scored the image as “flowers absent,” but there are flowers present in the image. (A) An example of a situation where there are obvious flowers on the sheet and the machine simply missed them. (B) An example of a situation where the floral structure is not connected to the plant. This often caused the machine to miss the flowers in the image. (C) An example where flowers are present but challenging to see.

Machine labeling and integration with other sources

After downloading and filtering out problematic images (images that could not be downloaded, images that cannot be read, or images that were either less than 10 kB or greater than 32 MB) and images of hard‐to‐annotate genera and families (Appendix S1), we were left with 22,447,563 images of digitized herbarium specimens. These images were fed through the ensemble modeling pipeline and annotated for present or absent flowers. Of these records, 11,131,820 were labeled as flowers present (meaning they were scored as present by at least two‐thirds of the models), but only 2,913,353 of these contained valid latitude, longitude, and date information (Figure 3). A total of 5,748,788 images were scored as flowers absent (meaning they were scored as absent by at least two‐thirds of the models), and 5,566,955 images were equivocal (meaning there was no consensus among the models). Only records scored as flowers present and with the needed metadata were retained and integrated into Phenobase (Phenobase Collaboration, 2026). These records were given the “flower” label in Phenobase, consistent with validation processes and with the Plant Phenology Ontology (Stucky et al., 2018). Biological definitions for these labels are provided to Phenobase data users both in the portal and in a file included with the download. Phenobase records each contain metadata essential for phenological research including, but not limited to, scientific name, year, day of year, location, and data source. Each entry also includes a source link, allowing data users to readily audit the machine labels.

Figure 3.

Figure 3

Open in a new tab

The percentage of all records (22,447,563) pushed through the ensemble pipeline labeled as “flowers present,” “flowers absent,” and “equivocal,” highlighting that the majority of records labeled as “flowers present” were unusable because they were missing critical date and/or location information. Only records in the “flowers present” category that had all critical fields were retained, representing 12.9% of all records.

Data coverage and evaluating specimens missing critical information

The 2,913,353 records annotated as flowers present represented 383 plant families and 8883 genera. They represent 320 collection years, with the earliest record dated in 1501 (Figure 4). We evaluated a small subset of the 8,218,467 records that were missing locality and date information to determine which fields were missing and whether those issues could be resolved with more complete digitization of labels. Out of the 100 random specimens that were missing date and latitude/longitude information, 65 had a date (specific day and year) on the specimen label while 35 did not; 85 had a locality that is georeferenceable and localizable to a unit lower than country, while 15 did not. A total of 57 out of 100 had both a georeferenceable specific locality and date information on the label. Only 44 of the 100 records had images retrievable from the original URL, while for 56 records the image URL failed.

Figure 4.

Figure 4

Open in a new tab

The number of records labeled as “flowers present” in the final dataset per decade, beginning in 1800. The dashed line indicates 2007, the year the USA National Phenology Network was founded and the year before iNaturalist began.

DISCUSSION

While machine‐automated annotation is a clear next step in addressing bottlenecks and delivering large quantities of valuable data to the research community, some annotation tasks are harder than others. Furthermore, tasks that are relatively straightforward in one context, such as annotating flowers in photographs of live plants, can be much more challenging in other contexts, such as attempting to do the same for dried, pressed, and aged plants on herbarium sheets. Other groups have produced herbarium data using machine labeling approaches to answer their own research questions about machine learning, certain taxa, or regions, and many models showcase reasonably high precision and recall (Lorieul et al., 2019; Goëau et al., 2020; Younis et al., 2020; Williamson et al., 2026). Here, however, our focus was on creating a reliable resource at the broadest possible scale and for the phenology research community as a whole; we were not focused on answering specific research questions. Our first priority was that stakeholders could trust, understand, and easily use these data. A key challenge for us was to develop a set of processes that we felt met a reasonable bar for trustworthiness and to clearly communicate where these processes may fall short. Here we are not wholly concerned with the size of the data product. Rather, creating trustworthy data in this context meant carefully vetting the quality of the training data, rigorously filtering data at multiple points, evaluating validation statistics, and being thoughtful about the data structure and documentation made available to stakeholders so they can best use these data and understand their value and limitations.

We also caution that the most appealing process for annotating flowers may be difficult at the taxonomic, spatial, and temporal scale we attempted here. Our initial hope when generating training data was that we could produce flower annotations utilizing a team of people, likely undergraduate students, with limited botanical training. While this approach might work when focusing on a small subset of plants, such as a genus or family, we found it untenable when scaling to all angiosperms. A vast, complicated diversity of inflorescence morphologies interacts with post‐mortem processes to make it especially challenging to detect flower presence, even for an expert botanist. We argue that the ideal solution for creating high‐quality annotations is delegating annotation tasks to botanists working on taxonomic groups where they have expertise, but this is also impractical given capacity and logistical constraints. Our solution here was using one expert with ample practice scoring sheets, which has both benefits and drawbacks. One major benefit is that there is likely more consistency and higher quality with this method than a crowdsourced solution. This is especially important in dealing with the many cases where flower presence was uncertain, which was critical for removing certain families and genera from downstream annotation labeling. However, a major drawback in using only one annotator to generate training data is that this method is less scalable, generating significantly fewer annotated records to use for training. Finally, more focused expertise in certain taxonomic groups may have produced fewer “unknown” labels and higher‐quality training data for those groups that could lead to inclusion in downstream modeling.

A key step in our process was making thoughtful, informed decisions about removing taxa that we believed were too challenging to score correctly. While annotating training data, the annotator could choose “present” or “absent” options to inform the models, but critically they also had an “unknown” option that they were encouraged to select whenever they could not make a confident determination. Ultimately, the proportion of “unknown” annotations, for both individual genera and families, was what we used to determine which taxa were simply too challenging to score consistently for flower presence. There is likely no perfect cutoff, and the decision to remove taxa for which more than 25% of records were too difficult to label represents a tradeoff between data quantity and quality. While it is always disappointing to exclude data, we prioritized maintaining stakeholder trust by favoring reliability over volume. We maintain that when attempting to scale challenging tasks such as phenological annotation of herbarium specimens, it is critical to acknowledge both human and technological limits and adjust methods accordingly.

Even with substantial taxonomic filtering, model performance was modest on both held‐out test data and expert‐validated data. Given the challenges in fitting a single model, we opted for an ensemble approach, which gave us more power to interrogate multiple models and arrive at a consensus. Ensembling is particularly useful because it reduces idiosyncratic errors from any single model. This strategy required more upfront effort but ultimately produced a more rigorous labeling workflow. As in other approaches we have published (Dinnage et al., 2025; Grady et al., 2026), our interest is in present phenophases, and the ensembled models are fairly good at avoiding false positives; however, we note that the model still produced a substantial number of false absences. Because we explicitly tuned the models to minimize false positives at the cost of increasing false negatives, it means we are likely throwing out many images that have flowers.

The filtering process did not end after we completed machine labeling. Out of 11 million records labeled as flowers present, we found that more than 70% lacked a reported latitude/longitude and/or a date. This is consistent with the overall proportion of specimen records on GBIF that have missing coordinates or spatial issues. When we queried GBIF on 16 December 2025, for example, there were 52,254,706 records with associated specimen images for Magnoliopsida and Liliopsida, but only about 32% of them (16,786,460) had latitude and longitude coordinates and had no geospatial issues. The percentage of incomplete data is disappointingly high and effectively limits the utility of these data for research, as date and location information are generally required for these data to be research ready (sensu Soltis et al., 2018). The root cause of this issue appears to be the publication of “stub records” to GBIF. These stub records may have taxon names but often lack dates and/or georeferences and are generally from very large collections. For example, one provider with records that often lacked date and geospatial data was the Museum National d'Histoire Naturelle de Paris, which published 5,031,777 angiosperm records with images. Furthermore, we should note that many of the images we annotated are no longer available on GBIF. Out of the 100 we checked, more than half no longer had resolvable images online; most of those also came from publishers who often supply a large volume of images. These significant issues highlight the need for efforts to expedite full label digitization to avoid these issues of missing metadata, as many labels do contain dates and can be georeferenced. Efforts are needed to support the best use of primary and secondary data (sensu Soberón and Peterson, 2004), such as phenology annotations generated from images. Improving link persistence for images tied to specimens will also better ensure reproducibility and stability when relating specimen images, core specimen metadata, and secondary data derived from those images.

Despite setbacks, our ensemble modeling flower annotation pipeline unlocks vast new global phenology data at new temporal scales. Although in situ phenology data collection networks (Crimmins et al., 2022) and other initiatives to unlock phenology information from sources like iNaturalist (Dinnage et al., 2025) are providing large quantities of contemporary phenology data, data predating these initiatives are less common. They are often in the form of tabular reports that were monitored at one site over many years, although some larger citizen science projects covered multiple sites and years (e.g., the New York Phenology Project [Fuccillo Battle et al., 2022] and Eastern USA Phenology Project [Guralnick et al., 2024] in the middle and late 1800s). The almost three million labeled records that we present here represent a large dataset of historical (77% are pre‐2000) phenology data that is now accessible to the research community.

All records labeled as flowers present from this effort are available through Phenobase, an open‐access datastore of plant phenology information that integrates in situ data from phenology monitoring programs, machine‐labeled data from iNaturalist images, and now machine‐labeled data from herbarium specimens. Our pipeline produces phenology labels and fields consistent with the Phenobase data structure and Plant Phenology Ontology (Stucky et al., 2018) and uses a controlled vocabulary for other fields, ensuring that data users can search for phenology records at any level of granularity and get consistent results. Future work may include efforts to label herbarium specimens for fruits. However, our early explorations found this task to be even more challenging than labeling for flowers, likely requiring greater human effort and more rigorous filtering steps. In summary, despite the challenges, providing information on historical flowering dates across the globe and integrating with other data sources enables new analyses and insights spanning regions and taxa, enabling increasingly broader questions to be asked across temporal, taxonomic, and spatial scales.

AUTHOR CONTRIBUTIONS

R.P.G., D.L., and R.L. initially conceived of this effort. E.L.G. operationalized most aspects of gathering training data, in consultation with R.L. and R.P.G. R.L. developed the modeling framework for this effort with help from R.D. R.L. produced final models and descriptive statistics and E.L.G. led expert model validation. E.G.D. and J.D. helped ensure that data in the right format could be ingested into Phenobase. E.L.G. and R.P.G. archived data. E.L.G. and R.P.G. wrote the manuscript with help from R.L., R.D., and D.L. All authors contributed to drafts and gave final approval for publication.

Supporting information

Appendix S1. List of difficult‐to‐annotate genera and families removed from training and downstream data.

APS3-9999-e70085-s002.docx (3MB, docx)

Appendix S2. List of families represented in the training data, including the number of images and percent of total training images per family.

APS3-9999-e70085-s003.csv (7.5KB, csv)

Appendix S3. Validation results for the held‐out test data.

APS3-9999-e70085-s001.docx (2.4MB, docx)

ACKNOWLEDGMENTS

The authors would like to thank the many individuals who have contributed to collecting, processing, digitizing, and maintaining herbarium specimens, as this effort would not be possible without their dedication. Phenobase is a team effort, and the authors acknowledge team members Carrie Seltzer, Ramona Walls, and Taran Lichtenberger. UFIT Research Computing provided computational resources and support. This work was funded by the National Science Foundation (DBI2223512 to R.P.G. and DBI2223508 to D.L.).

DATA AVAILABILITY STATEMENT

The final dataset of present‐only labeled specimens is available on the Phenobase data portal (https://phenobase.netlify.app/). Training data and final ensemble output can be found on Zenodo (https://doi.org/10.5281/zenodo.17675089). The ensemble models and a corresponding JSON file with model metadata data are housed on Zenodo (https://doi.org/10.5281/zenodo.17079402). Code used for this project can be found on GitHub (https://github.com/rafelafrance/phenobase/tree/v1.0.0). Images used in training, validation, and testing are available on Zenodo (https://zenodo.org/records/17675089). Suggestions for how to best use the pipeline presented here are available on GitHub (https://github.com/rafelafrance/phenobase/blob/main/docs/ensemble_tutorial.md).

REFERENCES

  1. Brenskelle, L. , Guralnick R. P., Denslow M., and Stucky B. J.. 2020. Maximizing human effort for analyzing scientific images: A case study using digitized herbarium sheets. Applications in Plant Sciences 8(6): e11370. 10.1002/aps3.11370 [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Crimmins, T. , Denny E., Posthumus E., Rosemartin A., Croll R., Montano M., and Panci H.. 2022. Science and management advancements made possible by the USA National Phenology Network's Nature's Notebook platform. BioScience 72(9): 908–920. 10.1093/biosci/biac061 [DOI] [Google Scholar]
  3. Cubuk, E. D. , Zoph B., Mane D., Vasudevan V., and Le Q. V.. 2018. AutoAugment: Learning augmentation policies from data. arXiv:1805.09501 [preprint]. Available at 10.48550/arXiv.1805.09501 [posted 24 May 2018; accessed 21 August 2026]. [DOI]
  4. Davis, C. C. , Willis C. G., Connolly B., Kelly C., and Ellison A. M.. 2015. Herbarium records are reliable sources of phenological change driven by climate and provide novel insights into species' phenological cueing mechanisms. American Journal of Botany 102: 1599–1609. 10.3732/ajb.1500237 [DOI] [PubMed] [Google Scholar]
  5. Deacy, W. W. , Armstrong J. B., Leacock W. B., Robbins C. T., Gustine D. D., Ward E. J., Erlenbach J. A., and Stanford J. A.. 2017. Phenological synchronization disrupts trophic interactions between Kodiak brown bears and salmon. Proceedings of the National Academy of Sciences, USA 114(39): 10432–10437. 10.1073/pnas.1705248114 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Deng, J. , Dong W., Socher R., Li L. J., Li K., and Fei‐Fei L.. 2009. ImageNet: A large‐scale hierarchical image database, 248–255. 2009 IEEE Conference on Computer Vision and Pattern Recognition, 20–24 June 2009, Miami, Florida, USA.
  7. Dinnage, R. , Grady E., Neal N., Deck J., Denny E., Walls R., Seltzer C., et al. 2025. PhenoVision: A framework for automating and delivering research‐ready plant phenology data from field images. Methods in Ecology and Evolution 16(8): 1763–1780. [Google Scholar]
  8. Dosovitskiy, A. , Beyer L., Kolesnikov A., Weissenborn D., Zhai X., Unterthiner T., Dehghani M., et al. 2021. An image is worth 16×16 words: Transformers for image recognition at scale. arXiv 11929 [preprint]. Available at 10.48550/arXiv.2010.11929 [revision posted 3 June 2021; accessed 4 September 2026]. [DOI]
  9. Fuccillo Battle, K. , Duhon A., Vispo C. R., Crimmins T. M., Rosenstiel T. N., Armstrong‐Davies L. L., and de Rivera C. E.. 2022. Citizen science across two centuries reveals phenological change among plant species and functional groups in the Northeastern US. Journal of Ecology 110(8): 1757–1774. 10.1111/1365-2745.13926 [DOI] [Google Scholar]
  10. Goëau, H. , Mora‐Fallas A., Champ J., Love N. L. R., Mazer S. J., Mata‐Montero E., Joly A., and Bonnet P.. 2020. A new fine‐grained method for automated visual analysis of herbarium specimens: A case study for phenological data extraction. Applications in Plant Sciences 8(6): e11368. 10.1002/aps3.11368 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Grady, E. L. , Denny E. G., Seltzer C. E., Deck J., Li D., Dinnage R., and Guralnick R. P.. 2026. Turning a new leaf: PhenoVision provides leaf phenology data at the global scale. Applications in Plant Sciences 14(5): e70067. 10.1002/aps3.70067 [DOI] [Google Scholar]
  12. Guralnick, R. , Crimmins T., Grady E., and Campbell L.. 2024. Phenological response to climatic change depends on spring warming velocity. Communications Earth & Environment 5: e634. 10.1038/s43247-024-01807-8 [DOI] [Google Scholar]
  13. Hussein, B. R. , Malik O. A., Ong W. H., and Slik J. W. F.. 2022. Applications of computer vision and machine learning techniques for digitized herbarium specimens: A systematic literature review. Ecological Informatics 69: e101641. 10.1016/j.ecoinf.2022.101641 [DOI] [Google Scholar]
  14. Kudo, G. , Nishikawa Y., Kasagi T., and Kosuge S.. 2004. Does seed production of spring ephemerals decrease when spring comes early? Ecological Research 19: 255–259. 10.1111/j.1440-1703.2003.00630.x [DOI] [Google Scholar]
  15. Loshchilov, I. , and Hutter F.. 2019. Decoupled weight decay regularization. In: International Conference on Learning Representations (ICLR 2019). arXiv:1711.05101v3 [preprint]. Available at: https://arxiv.org/abs/1711.05101 [submitted 14 November 2017; revised 4 January 2019; accessed 4 September 2026].
  16. Lorieul, T. , Pearson K. D., Ellwood E. R., Goëau H., Molino J.‐F., Sweeney P. W., Yost J. M., et al. 2019. Toward a large‐scale and deep phenological stage annotation of herbarium specimens: Case studies from temperate, tropical, and equatorial floras. Applications in Plant Sciences 7(3): e1233. 10.1002/aps3.1233 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Manangan, A. , Brown C., Saha S., Bell J., Hess J., Uejio C., Fineman S., and Schramm P.. 2021. Long‐term pollen trends and associations between pollen phenology and seasonal climate in Atlanta, Georgia (1992–2018). Annals of Allergy, Asthma & Immunology 127(4): 471–480.e4. 10.1016/j.anai.2021.07.012 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Menzel, A. , Sparks T. H., Estrella N., Koch E., Aasa A., Ahas R., Alm‐Kubler K., et al. 2006. European phenological response to climate change matches the warming pattern. Global Change Biology 12: 1969–1976. 10.1111/j.1365-2486.2006.01193.x [DOI] [Google Scholar]
  19. Nagai, S. , Saitoh T. M., and Yoshitake S.. 2019. Cultural ecosystem services provided by flowering of cherry trees under climate change: A case study of the relationship between the periods of flowering and festivals. International Journal of Biometeorology 63: 1051–1058. 10.1007/s00484-019-01719-9 [DOI] [PubMed] [Google Scholar]
  20. Pearson, K. D. , Nelson G., Aronson M. F. J., Bonnet P., Brenskelle L., Davis C. C., Denny E. G., et al. 2020. Machine learning using digitized herbarium specimens to advance phenological research. BioScience 70(7): 610–620. 10.1093/biosci/biaa044 [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Phenobase Collaboration . 2026. Phenobase Web Application. Available at website https://phenobase.netlify.app/
  22. Richardson, A. D. , Keenan T. F., Migliavacca M., Ryu Y., Sonnentag O., and Toomey M.. 2013. Climate change, phenology, and phenological control of vegetation feedbacks to the climate system. Agricultural and Forest Meteorology 169: 156–173. 10.1016/j.agrformet.2012.09.012 [DOI] [Google Scholar]
  23. Soberón, J. , and Peterson A. T.. 2004. Biodiversity informatics: Managing and applying primary biodiversity data. Philosophical Transactions of the Royal Society of London, Series B Biological Sciences 359(1444): 689–698. 10.1098/rstb.2003.1439. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Soltis, P. S. 2017. Digitization of herbaria enables novel research. American Journal of Botany 104: 1281–1284. 10.3732/ajb.1700281 [DOI] [PubMed] [Google Scholar]
  25. Soltis, P. S. , Nelson G., and James S. A.. 2018. Green digitization: Online botanical collections data answering real‐world questions. Applications in Plant Sciences 6(2): e1028. 10.1002/aps3.1028 [DOI] [Google Scholar]
  26. Stucky, B. J. , Guralnick R., Deck J., Denny E. G., Bolmgren K., and Walls R.. 2018. The plant phenology ontology: A new informatics resource for large‐scale integration of plant phenology data. Frontiers in Plant Science 9: e517. 10.3389/fpls.2018.00517. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Thanikachalam, R. , Muniasamy A., Alasmari A., and Thavasimuthu R.. 2025. EffNet‐CNN: A semantic model for image mining and content‐based image retrieval. Computer Modeling in Engineering and Sciences 143: 1971–2000. [Google Scholar]
  28. Visser, M. E. , and Gienapp P.. 2019. Evolutionary and demographic consequences of phenological mismatches. Nature Ecology & Evolution 3: 879–885. 10.1038/s41559-019-0880-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Williamson, D. R. , Prestø T., Westergaard K. B., Trascau B. M., Vange V., Hassel K., Koch W., and Speed J. D. M.. 2026. Long‐term trends in global flowering phenology. New Phytologist 251: 707–720. 10.1111/nph.70139 [DOI] [PubMed] [Google Scholar]
  30. Willis, C. G. , Ellwood E. R., Primack R. B., Davis C. C., Pearson K. D., Gallinat A. S., Yost J. M., et al. 2017. Old plants, new tricks: Phenological research using herbarium specimens. Trends in Ecology & Evolution 32(7): 531–546. 10.1016/j.tree.2017.03.015 [DOI] [PubMed] [Google Scholar]
  31. Yadav, S. , Korat J. R., Yadav S., Mondal K., Kumar A., Homeshvari, and Kumar S.. 2023. Impacts of climate change on fruit crops: A comprehensive review of physiological, phenological, and pest‐related responses. International Journal of Environment and Climate Change 13(11): 363–371. 10.9734/ijecc/2023/v13i113179. [DOI] [Google Scholar]
  32. Younis, S. , Schmidt M., Weiland C., Dressler S., Seeger B., and Hickler T.. 2020. Detection and annotation of plant organs from digitised herbarium scans using deep learning. Biodiversity Data Journal 8: e57090. 10.3897/BDJ.8.e57090 [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Zhou, N. , Siegel Z. D., Zarecor S., Lee N., Campbell D. A., Andorf C. M., Nettleton D., et al. 2018. Crowdsourcing image analysis for plant phenomics to generate ground truth data for machine learning. PLoS Computational Biology 14(7): e1006337. 10.1371/journal.pcbi.1006337 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Appendix S1. List of difficult‐to‐annotate genera and families removed from training and downstream data.

APS3-9999-e70085-s002.docx (3MB, docx)

Appendix S2. List of families represented in the training data, including the number of images and percent of total training images per family.

APS3-9999-e70085-s003.csv (7.5KB, csv)

Appendix S3. Validation results for the held‐out test data.

APS3-9999-e70085-s001.docx (2.4MB, docx)

Data Availability Statement

The final dataset of present‐only labeled specimens is available on the Phenobase data portal (https://phenobase.netlify.app/). Training data and final ensemble output can be found on Zenodo (https://doi.org/10.5281/zenodo.17675089). The ensemble models and a corresponding JSON file with model metadata data are housed on Zenodo (https://doi.org/10.5281/zenodo.17079402). Code used for this project can be found on GitHub (https://github.com/rafelafrance/phenobase/tree/v1.0.0). Images used in training, validation, and testing are available on Zenodo (https://zenodo.org/records/17675089). Suggestions for how to best use the pipeline presented here are available on GitHub (https://github.com/rafelafrance/phenobase/blob/main/docs/ensemble_tutorial.md).

Articles from Applications in Plant Sciences are provided here courtesy of Wiley

RESOURCES