Validation of a Plant Identification Application Using Digital Images of Toxic Plants

Introduction

In 2019, over 45,000 plant exposures in humans were reported to American Poison Control Centers making plants one of the top 25 reported categories. Most of these plant exposures refer to an unknown plant [1]. Plant identification is challenging due to several limitations: lack of physical plant in the health care setting, remote location of expert, poor quality photograph, or lack of identifying characteristics such as fruit or flower [2]. Correct identification of the plant is useful in predicting expected toxicity and guiding treatment.

Innovations in technology have led to improvements in plant identification. In 1993, poison centers began using fax machines to send pictures of plants to botanists for identification [3]. In 2003, Leafsnap was created as the first smartphone application for identification of plants [4]. Since then, several new applications have been created. However, a comparison of machines versus botanists in 2014 based on the LifeCLEF collection of plant images demonstrated that botanists still outperformed machines [5]. Yet, use of smartphone applications for plant identification is more easily obtained than consultation with a botanist.

PictureThis is one of many commercially available smartphone applications available on Apple and Android devices and, when previously studied, was identified as one of the most accurate applications for identification of poisonous plants [6]. PictureThis has both an unpaid version with advertisements and a paid Premium version. The unpaid version allows for unlimited identifications, but the user must first exit out of the advertisement for the premium version each time the application is opened. The website advertises PictureThis as a way to protect your pets through toxic plant identification [7]. PictureThis provides the user with one most likely species and, when applicable, provides information on toxicity in pets. For certain toxic species such as common foxglove, it lists the symptoms expected if ingested in humans but does not provide information on treatment, disclosures regarding use for patient care, or advise the reader to call a poison control center. PictureThis is marketed as having over 98% accuracy when compared to human experts which is higher than identified in prior work [6]. Our study aimed to evaluate PictureThis on a larger collection of toxic plants. Given that poison centers often have images of plants rather than actual plant specimens, this study used digital photographs of toxic plants obtained from a textbook.

Methods

Pictures of poisonous plants were obtained from a digitalized textbook [8] viewed on a MacBook Air (Intel HD Graphics 6000). Each plant was identified via the unpaid (non-premium) version 3.4 of PictureThis on an iPhone 11 Pro in May 2021. The application was used at a comfortable arm length from the computer without any standardization in lighting to mimic real-world conditions. All pictures contained in this textbook were analyzed except for 25 photos: 2 of jewelry, 1 of dried plant roots, 1 of microscopic plant crystals, and 21 drawings of plants. Analysis of these results in “no plants found.” If there was more than one photo for a species, all photos were analyzed. The application was only considered to be accurate for genera if it identified the genera correctly on all photos when more than one photo was available. One attempt was used by the first author for identification of each picture. To evaluate for precision, species that were featured in multiple photos were compared for consistency. Using a random number generator, 20% of the pictures were selected for repeat analysis by the second author.

Digitalized photos of plants were considered representative of real-world conditions as digital images are often sent to poison control centers rather than the actual plant.

Results

Accuracy

A total of 369 pictures were analyzed including 237 plant species and 158 plant genera. PictureThis correctly identified both the genus and species in 64% (n = 237). The correct genus was able to be identified in 82% (n = 301). There were 68 pictures in total in which PictureThis was not able to identify the correct genera; this spanned 48 individual genera which are listed in Table 1.

Table 1.

Complete list of misidentified toxic plant genera.

Acokanthera

Aconitum

Adonis

Allium

Alocasia

Hippeastrum

Blighia

Calotropis

Cassia

Cestrum

Chrysanthemum

Colchicum

Conium

Corynocarpus

Crinium

Crotolaria

Cryptostegia

Datura

Dieffenbachia

Ephedra

Gelsemine

Hellebore

Hippomane

Hydrastis

Laburnum

Ligustrum

Lycoris

Manihot

Melia

Pedilanthus

Pentalinon

Pernettya

Philodendron

Pieris

Pteridium

Rhodotypos

Schoenocaulon

Scilla

Senecio

Sesbania

Solanum

Sophora

Strychnos

Taxus

Urginea

Viscum

Xanthosoma

Zigadenus

Precision

There were 100 plant species pictured more than once in this text. PictureThis gave consistent results on duplicate pictures in 71% (n = 71). Of these, PictureThis was consistently correct in 57% (n = 57), consistently incorrect in 8% (n = 8), and consistently correct on genus only in 6% (n = 6). On random sampling of 20% of the photos used for the initial analysis (n = 74), the first and second authors obtained the same result in 88% (n = 65).

Discussion

This study aimed to investigate the accuracy and precision of PictureThis in identifying toxic plants on a digital platform. PictureThis was able to identify the correct genera of digital images of toxic plants with overall 82% accuracy. PictureThis was less accurate (64%) in discriminating species within a genus of toxic plants. Toxicologists and poison specialists do not often have the actual plant at time of consult and must rely on a digital image. Medical decisions must often be made based on plant identification with appropriate clinical correlation. Although this study did not compare to botanist identification, this is not typically available with rapid turnaround in the clinical setting. Therefore, the findings in this study are applicable to clinical care.

Our study did not find PictureThis to be as accurate as previously described. Otter et al. found PictureThis to be 84–100% accurate [6]. This discrepancy is not unexpected as Otter et al. used live specimens of commonly encountered plant species compared with digital photos in our study. Our study produced conflicting results with Otter et al.’s study specifically for Datura, Dieffenbachia, and Conium. Otter et al. reported PictureThis to be 100% accurate compared with our study demonstrating 83%, 0%, and 33% accuracy at identifying Datura, Dieffenbachia, and Conium, respectively. Ingestion of Datura, Dieffenbachia, or Conium would likely produce clinical symptoms; therefore with appropriate clinical correlation, toxicity should not be missed despite misidentification by PictureThis. However, using PictureThis to identify plants for consumption is problematic as ingestion of Conium maculatum could produce not only undesirable symptoms but also possibly death.

To evaluate the precision, our study had two approaches. First, when more than one photo was available for a particular plant species, all photos were tested, and results were compared. PictureThis demonstrated poor precision at 71%. This could be explained by variations in the part of the plant photographed, photo quality, and distance from the camera. Second, a random sampling of photos was tested in duplicate by a second author. Precision was higher at 88%. This is more concerning and difficult to explain as both authors used the same photographs, an iPhone, and the same version of PictureThis. This could be caused by subtle differences in lighting or angle of the photograph. It could also be explained by uncertainty within the application; when the application was wrong, multiple repeat attempts on the same photo to obtain the correct answer sometimes resulted in different results which is why we used the first attempt only for this study to mimic real-world conditions. If used clinically, obtaining different results from different providers could lead to variations in care.

Using digital images allowed for a larger analysis of 158 plant genera and 237 individual species. By broadening selection from commonly encountered toxic plants, this study was able to identify 48 genera in which the application was unreliable. Table 2 lists 19 highly toxic plants that were misidentified and capable of producing significant toxicity. This reinforces that PictureThis should not be used to identify plants for consumption or replace a toxicologist or poison specialist when caring for patients with possible toxic exposures.

Table 2.

Misidentified plants with significant toxicity in humans and their corresponding toxin.

Plant	Toxin
Acokanthera oppositifolia	Cardioactive steroids
Aconitum napellus	Aconitine, Hypaconitine, Mesaconitine
Adonis amurensis	Cardioactive steroids
Blighia sapida	Hypoglycin A
Colchicum autumnale	Colchicine
Conium maculatum	Coniine and related nicotine-like alkaloids
Crotalaria sagittalis	Monocrotaline and other pyrrolizidine alkaloids
Datura sanguinea	Atropine, scopolamine, and other anticholinergic alkaloids
Gelsemium sempervirens	Gelsemine, gelsemicine, and related alkaloids
Helleborus niger	Hellebrin, helleborin, and helleborein, cardioactive steroids
Hippomane manicella	Hippomane A and B
Laburnum anagyroides	Cytisine, a nicotine-like alkaloid
Pieris floribunda	Grayanotoxins
Rhodotypos scandens	Potentially cyanogenic glycosides
Schoenocaulon drummondii	Veratrum alkaloids
Senecio jacobaea	Pyrrolizidine alkaloids
Solanum sp.	Solanine glycoalkaloids
Strychnos nux-vomica	Strychnine
Taxus sp.	Taxine alkaloids
Urginea maritima	Scillarin
Zigadenus nuttalli	Zygadenine, zygacine, isogermidine, neogermidine, and protoveratridine

Our study has several limitations. Digital photos were used to simulate real-world scenarios. Pictures of poisonous plants from a textbook are more likely to highlight the characteristic parts of the plant during peak growing season. Textbook pictures are likely better quality than pictures submitted by patients. The first author reviewed images in May 2021 and the second in September 2021. This artificial intelligence application is based on an online database of over 10,000 plant species. As subscribers use the application, there is an option to submit feedback. This application continues to improve as more data is gathered. Therefore, our results apply to this version only. Accuracy and precision of the most updated version of PictureThis at the time of publication could be different.

Conclusions

PictureThis had over 80% accuracy for identifying a wide variety of digital images of toxic plants by genus, but performed only at 64% for specific species. Often, consultation with a botanist is not available in real time at regional poison control centers. Given the inaccuracy and inconsistency of this PictureThis, this application should not be used to guide decisions in patient care and cannot replace consultation with a poison center or medical toxicologist.

Author Contribution

All authors contributed to the study conception and design. Data collection and analysis were performed by SM and EF. All authors read and approved the final manuscript.

Declarations

This research was exempt from institutional IRB review as it does not pertain to human subjects.

Conflict of Interest

None.