EVALUATION OF IN SILICO MODEL PREDICTIONS FOR MAMMALIAN ACUTE ORAL TOXICITY AND REGULATORY APPLICATION IN PESTICIDE HAZARD AND RISK ASSESSMENT

Abstract

The United States Environmental Protection Agency (USEPA) uses the lethal dose 50% (LD₅₀) value from in vivo rat acute oral toxicity studies for pesticide product label precautionary statements and environmental risk assessment (RA). The Collaborative Acute Toxicity Modeling Suite (CATMoS) is a quantitative structure-activity relationship (QSAR)-based in silico approach to predict rat acute oral toxicity that has the potential to reduce animal use when registering a new pesticide technical grade active ingredient (TGAI). This analysis compared LD₅₀ values predicted by CATMoS to empirical values from in vivo studies for the TGAIs of 177 conventional pesticides. The accuracy and reliability of the model predictions were assessed relative to the empirical data in terms of USEPA acute oral toxicity categories and discrete LD₅₀ values for each chemical. CATMoS was most reliable at placing pesticide TGAIs in acute toxicity categories III (>500 – 5,000 mg/kg) and IV (>5,000 mg/kg), with 88% categorical concordance for 165 chemicals with empirical in vivo LD₅₀ values ≥500 mg/kg. When considering an LD50 for RA, CATMoS predictions of 2,000 mg/kg and higher were found to agree with empirical values from limit tests (i.e., single, high-dose tests) or definitive results over 2,000 mg/kg with few exceptions.

1. Introduction

1.1. Use of the acute oral toxicity test for pesticide safety assessment

The United States Environmental Protection Agency (USEPA) Office of Pesticide Programs (OPP) uses data specified in 40 Code of Federal Regulations (CFR) Part 158 (40 CFR 158, 2023) to make regulatory decisions regarding the effects of pesticides on human health and the environment. The in vivo rat acute oral toxicity test is included in 40 CFR 158 and is performed for all technical grade active ingredients (TGAIs) as well as for end-use product formulations of pesticide TGAIs mixed with other ingredients. The median lethal dose (LD₅₀), a statistically derived single dose of test substance expressed in milligrams per kilograms body weight (mg/kg) of the test animal that can be expected to cause death in 50% of the animals when administered by the oral route, is the endpoint of this test. The USEPA Office of Prevention, Pesticides and Toxic Substances (OPPTS) test guideline 870.1100 (USEPA, 2002) lists three accepted approaches to assess acute oral toxicity in which progressive doses are given: the up-down procedure (OECD, 2022), acute toxic class method (OECD, 2002a), and the fixed dose method (OECD, 2002b). Results from these studies reported as a specific dose, such as 50 or 1,000 mg/kg, are termed definitive LD₅₀s, although each has a limit test option (i.e., a single high dose of either 2,000 mg/kg or 5,000 mg/kg) for suspected low toxicity substances. If there is no mortality observed with the limit test, the LD₅₀ is then considered to be non-definitive at >2,000 or >5,000 mg/kg. LD₅₀s may be reported individually for both male and female rats, or only for females, which tend to be more sensitive when there are differences in sensitivity (OECD, 2022). Sometimes, a combined LD₅₀ value (i.e., average) that does not distinguish by sex may be reported. The number of animals used in each of these three approaches usually ranges from five to nine (OECD, 2001). OPP received between 300 – 500 of these studies per year from 2017 through 2021, although most were for pesticide formulations rather than TGAIs. Thus, the number of animals used to conduct the acute oral toxicity test has recently ranged from about 1,500 – 4,500 per year, assuming the use of five to nine animals per test.

USEPA uses in vivo acute oral toxicity data for two main purposes. First, based on the LD₅₀, the acute oral toxicity classification of the test substance is assigned to one of four acute toxicity categories (Table 1) that range from the most potent category I to the least potent category IV (USEPA, 2018). The toxicity category determines the precautionary statement about the hazard associated with the product. That information is placed on pesticide labels in the form of hazard signal words and symbols, first aid information, precautionary statements, worker protection statements, and personal protective equipment information for product use (USEPA, 2018). Secondly, the acute oral LD₅₀ value is used as a surrogate to represent acute oral toxicity to all mammalian wildlife. When conducting a quantitative ecological RA, USEPA compares LD₅₀s to pesticide exposure estimates, according to methods described in USEPA (2004; 2012), to determine if a potential risk concern exists for acute oral toxicity. When the LD₅₀ is non-definitive based on results of a limit test, no mortality is observed, and the highest concentration tested is greater than environmentally relevant concentrations, then risk from acute oral exposure is considered low and a quantitative RA is typically not conducted (USEPA, 2011).

Table 1.

USEPA acute oral toxicity categories based on LD₅₀ values and associated label precautionary statements and signal words (USEPA, 2018).

Category	Oral LD50 (mg/kg)	Precautionary Statement	Signal Word
I	≤50	Fatal if swallowed	Danger
II	>50 – 500	May be fatal if swallowed	Warning
III	>500 – 5,000	Harmful if swallowed	Caution
IV	>5,000	No statement required	None required

1.2. Reducing animal use through New Approach Methods

USEPA is involved in several ongoing efforts to develop and use New Approach Methods (NAMs) to test chemicals for toxic effects as described in its NAMs workplan, first issued in June 2020 and updated in December 2021 (USEPA, 2021). A key element in USEPA’s strategy is to establish scientific confidence in NAMs and demonstrate their applicability to regulatory decision-making, thus ensuring protection of human health and the environment. A secondary benefit of adopting NAMs is reducing the use of animals. In silico approaches are one alternative to in vivo testing that has shown promise thanks to recent advances in computational resources and artificial intelligence/machine learning applications to large, curated training datasets.

An example of a widely used in silico method is the quantitative structure-activity relationship (QSAR), which is a modeling approach employed in chemistry, pharmacology, and toxicology to predict the biological activity, properties, or behavior of chemical compounds based on their structure. It involves analyzing the quantitative relationship between chemical structures (molecular descriptors) and their biological activities or other properties using mathematical models and statistical techniques. QSAR models can assist in drug design, toxicity prediction, environmental impact assessment of chemicals, and more. They are particularly useful when experimental data are limited or expensive to obtain, allowing predictions to be made based on structural characteristics (Mansouri et al., 2016; Mansouri et al., 2020).

Several recently published articles have demonstrated the reliability and utility of in silico acute toxicity models for several chemical spaces. For example, Bercu et al. (2021) investigated the Leadscope (Instem, 2023) QSAR model for its ability to predict United Nations Globally Harmonized System (GHS) hazard categories, based on rat oral acute toxicity, of chemicals including pharmaceuticals, plant protection products (pesticides), pharmaceutical and pesticide intermediates, and metabolites. They found it to be reliable in its hazard predictions for a wide range of chemical structures, spanning numerous industries. This evaluation noted the importance of a weight of evidence approach, which increased fit-for-purpose and accuracy metrics for the model predictions.

In other investigations, Graham et al. (2021) and Moudgal et al. (2023) independently examined the reliability of two QSAR models, Leadscope and the Collaborative Acute Toxicity Modeling Suite (CATMoS) (Kleinstreuer et al., 2018; Mansouri et al., 2021), at predicting acute oral toxicity of existing pharmaceutical compounds based on rat LD₅₀s. Both studies concluded that these models demonstrated a high degree of agreement with in vivo results and could be used most reliably to identify low acute oral toxicity chemicals with LD₅₀s >300 mg/kg (GHS categories 4, 5, and not classified).

1.3. The Collaborative Acute Toxicity Modeling Suite (CATMoS)

In this analysis, we focused on CATMoS, an in silico QSAR predictive tool for screening chemicals for acute oral toxicity based on two-dimensional molecular structures. It was released by the National Toxicology Program Interagency Center for the Evaluation of Alternative Toxicological Methods (NICEATM) within OPERA (Open Structure-Activity/Property Relationship App), a free and open-source/open-data suite of QSAR models¹ developed to support regulatory decisions (Mansouri et al., 2018), and is described in greater detail in the Supplemental Material file and in Mansouri et al. (2021). OPERA-implemented models can be run on chemicals one by one or in batch mode but can only be applied to well defined small molecule organic structures (i.e., no mixtures, no polymers, or substances of unknown or variable composition, complex reaction products, or biological materials known as UVCBs). Using information on chemical structure and rat acute oral toxicity data curated from public literature sources comprised largely of studies to support regulatory submissions, CATMoS was built with nearly 9,000 chemicals and tested with a validation set of approximately 3,000 chemicals. To provide quantitative benchmarks for assessing the performance of CATMoS predictions, the variability of the reference in vivo data was determined through a bootstrap analysis of a subset of 1,885 chemicals with two or more discrete empirical LD₅₀ values that provided a data-derived 95% confidence interval around the LD₅₀ of +/−0.24 log₁₀ mg/kg. Analysis of 2,441 chemicals with multiple categorical values (e.g., including limit tests) yielded categorical reproducibility ranging from 55% – 80% (Karmaus et al., 2022).

CATMoS predicts rat acute oral toxicity endpoints relevant to USEPA regulatory needs, i.e., discrete LD₅₀ values and USEPA toxicity categories I – IV (Strickland et al., 2018). OPERA’s output for CATMoS predictions also provides an indication of confidence in the accuracy of the discrete value or category predicted, and an assessment of whether the predicted test substance falls within the applicability domain of the models. Against multiple external validation sets, in silico CATMoS predictions performed as well as the in vivo acute oral toxicity test, using the repeatability of the in vivo data as a quantitative benchmark (Mansouri et al., 2021; Graham et al. 2021; Karmaus et al., 2022; Moudgal et al., 2023).

The purpose of this analysis was to determine how well CATMoS performs in the chemical space of conventional pesticides in predicting rat acute oral toxicity LD₅₀ values relative to in vivo empirical LD₅₀ values of TGAIs registered by USEPA in approximately the last 20 years.² The accuracy and reliability of model predictions in terms of USEPA toxicity categories I – IV and discrete LD₅₀ values were assessed.

2. Methods

2.1. Collection, evaluation and curation of empirical data used for analysis

We focused on the 195 conventional pesticides registered by USEPA from 1998 – 2020 given that more recent chemistries are potentially more representative of future chemistries.³ The latest versions of USEPA ecological and human health RAs were obtained from public sources by searching USEPA’s Pesticide Chemical Search dashboard⁴ and the US government website Regulations.gov.⁵ The data extracted from the RAs included empirical rat LD₅₀, pesticide common name, and Chemical Abstract Service (CAS) registry number. Pesticide chemical structures were obtained in Simplified Molecular Input Line Entry System (SMILES) notation from USEPA’s Distributed Structure-Searchable Toxicity (DSSTox) database.⁶ Seventeen pesticides were removed from the initial list including six inorganic chemicals, four rodenticides, two soil fumigants, and five chemicals that cannot be predicted by the CATMoS model (e.g., pesticides with ambiguous or undefined structures and mixtures of multiple compounds). Given that a mammal is the intended target pest for rodenticides, making them more toxic (typically category I or II), it is unlikely that USEPA would use an in silico NAM at this time to replace an animal test for these substances; therefore, they were removed from the list. As the acute oral toxicity study is not relevant unless the chemical also comes in a form that can be ingested, the two soil fumigants were excluded because their exposure route is through inhalation. One of the 195 pesticides had ester and acid forms that were modeled separately and were counted as two pesticides.

In most cases, USEPA relies on data submitted by registrants that conforms to the study design and conditions outlined in OPPTS 870.1100. The data extraction revealed the need to conduct additional curation in some cases to select the most appropriate empirical LD₅₀ value for comparison to the CATMoS prediction. For example, there were occasional differences in the LD₅₀ values reported in the ecological and human health RAs for the same pesticide, or the LD₅₀ used in the ecological RA was from a study in the mouse or other species rather than the rat. In those cases, it was determined through consultation with USEPA, and expert examination of the data used to support the RA, which value was most appropriate for comparisons made in this analysis. As a result, one pesticide was removed due to conflicting empirical in vivo LD₅₀ values from different sources that could not be resolved, and one had an LD₅₀ in the dog instead of the rat that was also removed because only rat data were used in the analysis.

There were 26 pesticides where the rat LD₅₀ used in the RA was based on a single sex, usually female, rather than on the more commonly averaged value of both sexes. Acute oral toxicity data used to build the CATMoS model were not distinguished by sex, leading to its inability to predict LD₅₀s by sex. Thus, for this comparison it was decided to recalculate the USEPA empirical LD₅₀ data for these 26 cases as the average of reported male and female rat values (Supplemental Table S1). In cases where one male/female value was definitive and the other was non-definitive, the definitive value was retained for comparison to the predicted value.

The specific rat LD₅₀s obtained from the USEPA pesticide RAs and used in this analysis generally were not in the peer-reviewed published literature or in the gray literature. In contrast, the LD₅₀ dataset used to develop the CATMoS model was collected from public sources including the open literature and readily accessible regulatory databases such as the Organisation for Economic Cooperation and Development (OECD) eChemPortal, the National Library of Medicine’s Hazardous Substances Data Bank (NLM HSDB), ChemIDplus databases, and the European Commission Joint Research Center’s (JRC) AcutoxBase (Karmaus et al. 2019; Kleinstreuer et al. 2018; NTP 2018). While the USEPA RAs were not part of that data curation, for some pesticides in the current analysis there was likely some overlap with values curated from these other sources if registrants also reported study results to non-USEPA databases queried during the CATMoS dataset building process. Therefore, for any chemicals in our analysis that were also in the CATMoS dataset, we noted when the empirical LD₅₀ values differed substantially. However, some differences are expected due to inherent variability in animal data such as the reporting of a single sex LD₅₀ rather than an averaged value from both sexes, and because the dataset also contained open literature data that may have differed from data generated by registrants for the same chemical. Adherence to standardized regulatory guideline protocols was confirmed when available during curation of the CATMoS dataset, but lack of methodological information could introduce greater variability in the LD₅₀ values if non-standardized and varying test procedures were included. Other aspects such as chirality effects or bridging approaches (sharing data between similar active ingredients) could also influence the variability in the data. Reflecting this variability in LD₅₀ values derived from different sources, there were 20 chemicals where LD₅₀s from USEPA and CATMoS datasets did not match, as shown in Supplemental Table S2 with both USEPA RA-derived and CATMoS dataset empirical in vivo LD₅₀ values for comparison purposes. In all cases these differences were resolved and did not result in any adjustment of empirical values in either dataset other than the above-mentioned averaging of USEPA male/female LD₅₀s.

The workflow used in the data curation, model development, and comparative analysis is shown in Fig. 1. After the RA dataset curation, there were a total of 177 pesticide TGAIs (67 fungicides, 56 herbicides, 46 insecticides/acaricides, three nematicides, four plant activators/growth regulators, and one reptilicide) used in the analysis (Table 2). The focus of the analysis was on parent compounds and no metabolites or degradants were included, mainly due to a general lack of information on the toxicity of these transformation products. Of the 177 chemicals, 115 were in the CATMoS dataset, although, as noted above for 20 of the chemicals that overlapped, the empirical values from USEPA did not match the experimental data collected from the open literature or other data sources during the development of the CATMoS model. When making predictions for chemicals already in the model’s knowledge base, it is important to note that CATMoS does not take the exact experimental value as the prediction, but rather generates a prediction derived from a consensus of the individual models integrated within the CATMoS suite. Overlapping chemical structures between comparative datasets and the CATMoS dataset do not necessarily impact the interpretation of model results because the consensus prediction is a multistep mathematical process and is not based on any specific empirical value in the model dataset, which may differ from that in the comparative dataset.

Figure 1.

Workflow detailing data curation, model building, and comparative analysis for assessing performance of CATMoS LD₅₀ predictions of TGAIs for potential use in regulatory application.

Table 2.

Pesticides included in the analysis and their chemical classes based on BCPC (2023).

FUNGICIDES
• Amides	ㅁ Valinamidecarbamates	• Organophosphates
Benalaxyl-M	Benthiavalicarb	Tolclofos-methyl
ㅁ Anilides	Iprovalicarb	• Oxazolidinediones
Fenhexamid	Valifenalate	Famoxadone
ㅁ Benzamides	• Aminopyrazolinones	• Piperdines
Fluopicolide	Fenpyrazamine	Fenpropidin
Fluopyram	• Antibiotics	• Piperidinyl-thiazole-isoxazolines
Zoxamide	Kasugamycin	Oxathiapiprolin
ㅁ Cinnamamide oximes	• Aryl phenyl ketones	• Pyrimidines
Dimethomorph	Metrafenone	ㅁ Anilinopyrimidines
ㅁ Cyanoacetamide oximes	Pyriofenone	Cyprodinil
Cymoxanil	• Azoles	Mepanipyrim
ㅁ Mandelamides	ㅁ Triazoles	Pyrimethanil
Mandipropamid	Bromuconazole	• Quinazolines
ㅁ Phenylacetamide oximes	Epoxiconazole	Proquinazid
Cyflufenamid	Flutriafol	• Quinolines
ㅁ Phenyloxoethy thiophenacetamides	Ipconazole	Quinoxyfen
Isofetamid	Mefentrifluconazole	• Quinones
ㅁ Pyrazole carboxamides	Metconazole	Dithianon
Benzovindiflupyr	Tetraconazole	• Spiroketalamines
Bixafen	Triticonazole	Spiroxamine
Fluindapyr	ㅁ Triazolinthiones	• Strobilurins
Fluxapyroxad	Prothioconazole	Fluoxastrobin
Inpyrfluxam	• Carbamates	Kresoxim-methyl
Isopyrazam	Diethofencarb	Mandestrobin
Penflufen	• Cyanoimidazoles	Picoxystrobin
Penthiopyrad	Cyazofamid	Pyraclostrobin
Sedaxane	• Dinitroanilines	Trifloxystrobin
ㅁ Pyridinecarboxamides	Fluazinam	• Sulfamoyltriazoles
Boscalid	• Dinitrophenols	Amisulbrom
ㅁ Sulfamides	Meptyldinocap	• Tetrazolyloximes
Tolyfluanid	• Imidazolines	Picarbutrazox
ㅁ Thiadiazolecarboxamides	Fenamidone	• Triazolopyrimidines
Isotianil	• Morpholines	Ametoctradin
ㅁ Thiazolecarboxamides	Fenpropimorph	• Unclassified
Ethaboxam		Furfural
• Amides	• N-phenyltriazolinones	• Triketones
ㅁ Chloroacetamides	Azafenidin	Benzobicyclon
Dimethenamid-P	Carfentrazone-ethyl	Bicyclopyrone
Pethoxamid	• Oxyacetamides	Mesotrione
ㅁ Sulfonanilides	Flufenacet	Tembotrione
Pyrimisulfan	• Phenoxys	• Ureas
• Aromatic acids	ㅁ Aryloxyphenoxypropionics	ㅁ Sulfonylureas
ㅁ Arylcarboxylic acids	Clodinafop-propargyl	Ethametsulfuron-methyl
Diflufenzopyr	Cyhalofop-butyl	Flazasulfuron
ㅁ Benzoic acids	• Pyrazoles	Flucarbazone-sodium
Bispyribac-sodium	ㅁ Benzoylpyrazoles	Foramsulfuron
ㅁ Benzyl ether acids	Pyrasulfatole	Iodosulfuron-methyl sodium
Methiozolin	Topramezone	Imazosulfuron
ㅁ Pyridinecarboxylic acids	ㅁ Phenylpyrazoles	Mesosulfuron-methyl
Aminopyralid	Pyraflufen-ethyl	Orthosulfamuron
Florpyrauxifen-benzyl	ㅁ Phenylpryrazolines	Propoxycarbazone
Halauxifen-methyl	Pinoxaden	Sulfosulfuron
ㅁ Pyrimidinecarboxylic acids	• Pyridazines	Thiencarbazone-methyl
Aminocyclopyrachlor	Pyridate	Trifloxysulfuron-sodium
• Cyclohexene oximes	• Pyridines
Tepraloxydim	Fluroxypyr-acid*
Tralkoxydim	Fluroxypyr-MHE (ester)*
• Isoxazoles	• Triazines
Isoxaflutole	Indaziflam
• Isoxazolines	Propazine
Pyroxasulfone	• Triazolones
• N-phenylimides	Amicarbazone
Butafenacil	• Triazolopyrimidines
Flufenpyr-ethyl	Cloransulam-methyl
Flumioxazin	Diclosulam
Fluthiacet	Florasulam
Saflufenacil	Penoxsulam
Tiafenacil	Pyroxsulam
• Amidines	• Pyrazoles	• Fluroalkenyls
Demiditraz	Ethiprole	Fluensulfone
• Beta-ketonitriles	Tebufenpyrad	• Organophosphates
Cyflumetofen	Tolfenpyrad	Fosthiazate
• Benzoylureas	• Pyrethroids	• Unclassified
Flufenoxuron	Alpha-cypermethrin	Tioxazafen
Lufenuron	Flumethrin
Novaluron	Imiprothrin	PLANT GROWTH ACTIVATORS/GROWTH
Noviflumruon	Metofluthrin	REGULATORS
Teflubenzuron	Momfluorothrin
• Dicyclhydrazines	Transfluthrin	• Benzothiadiazoles
Methoxyfenozide	• Pyridine azomethines	Acibenzolar-s-methyl
• Diamides	Pymetrozine	• Unclassified
Chlorantraniliprole	• Quinazolines	Ecolyst (PT807-HCl)
Cyantraniliprole	Fenazaquin	Forchlorfenuron
Flubendiamide	• Quinones	Prohexadione calcium
Tetraniliprole	Acequinocyl
• Hydrazides	• Semicarbazones	REPTILICIDES
Bifenazate	Metaflumizone
• Meta-diamides	• Tetramic acids	• Unclassified
Broflanilide	Spirotetramat	Acetaminophen
• Neonicotinoids	• Tetronic acids
ㅁ Butenolides	Spirodiclofen
Flupyradifurone	Spiromesifen
ㅁ Cyano imidamides	• Unclassified
Acetamiprid	Buprofezin
Thiacloprid	Etoxazole
ㅁ Nitroguanidines	Fenpyroximate
Clothianidin	Flonicamid
Dinotefuran	Lithium P-sulfonate
Thiamethoxam	Oxalic acide
• Sulfoximines	Picaridin
Sulfoxaflor	Pyridalyl
• Oxadiazines
Indoxacarb

^*The acid and ester forms of the herbicide fluroxpyr had different CAS numbers, structures, and LD50s and were analyzed separately.

2.2. Model predictions

In addition to the discrete LD₅₀ and categorical toxicity category predictions used in this analysis, OPERA provided the prediction confidence range for the discrete values denoting the uncertainty limits, an accuracy estimate index ranging from 0 to 1, and two levels of applicability domain assessments (AD): a global AD (0 or 1) based on the entire model dataset specifying whether the prediction is within the limits of the chemical interpolation space and a local AD index ranging from 0 to1 based on the weighted similarity to the nearest neighbors (most similar structures based on the model’s descriptor space) of the predicted chemical. The assessments of the applicability domains follow the approaches described in Mansouri et al. (2018). The accuracy index is calculated using the information of the five nearest neighbors and represents the concordance among the predictions of the single independent models and the accuracy of predictions in comparison to the experimental data, all weighted by similarity (defined by the models’ selected descriptors). Optional information such as the experimental values when available and the five nearest neighbors’ information (identifiers, experimental and predicted values) can also be obtained. Additional information about how the model operates and is used can be found in the Supplemental Material Section S1 and Mansouri et al. (2021).

As described above, differences noted in in vivo LD₅₀ values for the same test substance from different sources represent the inherent variability around in vivo experimental data. The analysis conducted during the CATMoS project that resulted in a margin of uncertainty of +/−0.24 log₁₀ mg/kg (Karmaus et al., 2022) is considered a range of inherent variability for empirical acute oral toxicity data based on studies from the open literature, curated where possible based on adherence to the OPPT/OECD guidelines. The variability is thought to encompass the biological variance within the test organisms, the range of study designs, and general repeat-test variability due to other factors such as laboratory conditions. This variability range may or may not differ for other acute oral repeat testing sources that were not considered during the CATMoS project due to study design or conditions, although this range showed almost no variation during the curation process that gradually eliminated multiple chemicals and study results based on low confidence data, e.g., due to transcription/typo errors or unrealistic doses. For the current analysis, a data-driven confidence range was calculated around each CATMoS predicted LD₅₀ based on the upper and lower confidence limits of +/−0.24 log₁₀ mg/kg and included for comparison purposes to help characterize the accuracy of the predictions. All empirical and predicted values used in the analysis along with upper and lower confidence limits around the predicted values are provided in Supplemental Table S3.

2.3. Approach to comparative analysis

Using the CAS numbers of the selected pesticide TGAIs as input to OPERA, CATMoS predictions were generated based on SMILES structures for comparison to rat acute oral LD₅₀s used in the RAs that were derived in guideline studies submitted to USEPA, both in terms of discrete LD₅₀ values and USEPA toxicity categories. The full OPERA (version 2.7) output of CATMoS predictions for the pesticides analyzed here is available in Supplemental Table S4.

Acute oral toxicity categories based on empirical in vivo LD₅₀s were compared to CATMoS-predicted categories to evaluate the model’s ability to provide a value consistent with the hazard associated with precautionary statements on current pesticide labels. This comparison was made based on whether CATMoS placed the pesticide in the same toxicity category as the in vivo data submitted to USEPA. Also, as part of the analysis, pesticides in categories III and IV (LD₅₀s >500 mg/kg) based on in vivo data were considered together and compared to CATMoS predictions of >500 mg/kg, thus testing whether CATMoS correctly predicts when a pesticide is in category III but may also fall in category IV, categories which represent less potent chemicals.

For the comparison of the discrete LD₅₀ values, the replicate data-driven upper confidence limits (UCL) and lower confidence limits (LCL) based on the confidence interval of +/−0.24 log₁₀ mg/kg representing acute oral toxicity test variability were applied to the predicted discrete LD₅₀ value to determine if the empirical LD50 from the USEPA RA fell within this confidence interval.

As many pesticide TGAIs display chirality or isomerism, and these chemicals with different three-dimensional forms can produce different toxicities, we investigated the potential impact on CATMoS predictions. TGAIs with various isomer mixtures can have differing biological activities and toxicity effects, which can be reflected both in empirical in vivo/in vitro data and in silico predictions as well. This is particularly challenging to modeling efforts designed for large, heterogeneous groups of chemicals where the machine learning algorithms work from two-dimensional structures without necessarily distinguishing between stereoisomers. CATMoS can only process two-dimensional structures, so it cannot predict toxicity of three-dimensional stereochemistry. Pesticides with chiral centers (no specific isomerism) were identified manually by USEPA, as well as from the chemical structures using the RDKit cheminformatic tool implemented in KNIME (Berthold et al., 2008), and the accuracy of CATMoS predictions of these versus achiral pesticides were compared to determine if chirality affected performance of the model.

3. Results

The empirical LD₅₀s of the pesticide TGAIs in this analysis ranged from 62 mg/kg to7,500 mg/kg (Supplemental Table S3), encompassing three USEPA acute oral toxicity categories (II – IV). Only 57 pesticides had definitive LD₅₀ values. The remaining 120 pesticides had non-definitive LD₅₀ values with 42 chemicals estimated at >2,000 mg/kg, and 78 chemicals estimated at >5,000 mg/kg. There were no pesticides in the analysis with empirical values in category I; 12 pesticides were in category II; 84 were in category III; and 81 were in category IV. Therefore, while the majority are less potent chemicals (category III or IV), the analysis covered a relatively wide range of toxicity, lacking only the most potent, category I substances (e.g., rodenticides having been excluded). The LD₅₀s predicted by CATMoS ranged from 23 mg/kg to 9,010 mg/kg, with 52 between >2,000 mg/kg and 5,000 mg/kg, and 24 at >5,000 mg/kg. All predictions were within the global AD of the model but with varying local AD indices.

3.1. Acute Oral Toxicity Category Comparisons

CATMoS model predictions agreed with or were more conservative than (over-predicted) the toxicity category based on the empirical in vivo LD₅₀ value for 94% of the pesticides and under-predicted the category for only 6% (Fig. 2A). Nearly all of the differences in predicted category were within +/−1 USEPA category (Fig. 2B); of the remaining four model predictions that were within +/−2 USEPA categories, three were cases where the model predicted a category II for a category IV pesticide (transfluthrin, tolylfluanid, broflanilide), while one was a case where the model predicted a category I for a category III pesticide (fluensulfone). For the 12 category II pesticides, five (42%) were under-predicted as category III, while for 84 category III pesticides, only five (6%) were under-predicted as category IV (Fig. 2C).

Figure 2.

CATMoS model LD₅₀ category prediction agreement with category based on empirical in vivo LD₅₀ data. (A) Agreement of empirical in vivo and model toxicity categories for 177 pesticides in terms of exact match, over-prediction (more potent than empirical) and under-prediction (less potent than empirical); (B) Agreement of empirical in vivo and model toxicity categories for 177 pesticides in terms of exact match, within +/−1 USEPA category or +/−2 USEPA categories; (C) Agreement of empirical in vivo and model toxicity categories for 177 pesticides by category in terms of exact match, over-prediction (more potent than empirical) and under-prediction (less potent than empirical).

Concordance analysis was used to compare toxicity categories based on CATMoS LD₅₀ predictions to empirical LD₅₀ values (Table 3). CATMoS predictions of category II were correct in 6/25 cases (24%) while 19 of those predictions (76%) were for chemicals that were less potent (category III/IV) according to empirical data. CATMoS predictions of category IV were correct in 19/24 cases (79%). Taken together, pesticides predicted by CATMoS to be in categories III or IV (n = 150) were, empirically, in either category III or IV (Table 3, area in the box) in 145 cases (97%), with just five cases in category II (3%).

Table 3.

Concordance of CATMoS predictions of acute oral toxicity categories I to IV with empirical in vivo toxicity categories. Bolded numbers indicate that predicted and empirical categories agree. Values within the box denote 145/150 CATMoS predictions of categories III or IV were, empirically, also in either category III or IV.

Toxicity Category based on CATMoS Prediction	Number of predictions	Toxicity Category based on Empirical In Vivo Test Data
		I	II	III	IV
I (<50 mg/kg)	2	-	1	1	-
II (50–500 mg/kg)	25	-	6	16	3
III (>500–5,000 mg/kg)	126	-	5	62	59
IV (>5,000 mg/kg)	24	-	-	5	19
III and IV combined	150	-	5	145

For pesticides with an empirical value of >500 mg/kg (category III or IV), the CATMoS prediction was also >500 mg/kg in 145/165 cases, meaning that a prediction of category III or IV agreed with empirical data 88% of the time. The remaining 20 pesticides with empirical LD₅₀s >500 mg/kg had CATMoS over-predictions that placed them in category II, or, in one case, in category I. Of 81 pesticides in category IV based on empirical data, CATMoS predicted three to be category II, 59 to be category III and 19 as category IV (Table 3). Thus, 23% were predicted correctly as category IV, and 73% were over-predicted as category III.

Overall, CATMoS was found to be a good predictor of toxicity categories III and IV as a combined group. While the model predictions were not always concordant with empirical values for pesticides identified as category II, CATMoS predictions of >500 mg/kg were concordant with empirical data of >500 mg/kg in nearly 90% of cases.

Concordance analyses based on toxicity category were also performed for the three main pesticide use sub-types from Table 2: fungicides (67), herbicides (56) and insecticides/acaricides/nematicides (49) to determine if the model predicted better for some sub-types than others (Supplemental Table S5). The type-relative, multiclass balanced accuracy (BA) values of 0.55, 0.56 and 0.68 for the fungicides, herbicides, and insecticides, respectively, indicated that the use sub-type specific concordance was like the overall concordance for the full list of chemicals (0.63) with the insecticides/acaricides/nematicides having the highest concordance of the three.

3.2. LD₅₀ Discrete Value Comparisons

When used in quantitative RA, a more accurate prediction of LD₅₀ is needed than for the acute toxicity categories. The discrete value LD₅₀ CATMoS predictions, along with the CATMoS UCL and LCL, and the empirical LD₅₀ values from in vivo rat studies submitted for pesticide TGAI registration and conducted under OPPTS 870.1100 are plotted in Fig. 3 and sorted from left to right in order of decreasing toxicity of the CATMoS prediction. (These data are also displayed as an X-Y plot in Supplemental Fig. S1). Empirical values (blue bars) reaching exactly 2,000 or 5,000 mg/kg represent limit tests that elicited no toxicity to the rat at those doses. Fig. 3 shows that, in general, CATMoS predictions of LD₅₀ are lower (more toxic) than empirical definitive rat LD₅₀s or limit tests. However, above a CATMoS-predicted LD₅₀ of 2,000 mg/kg, the empirical value is also equal to or above 2,000 mg/kg 96% of the time except in 3/75 cases: clodinafop-propargyl, 1,392 mg/kg; fenpropimorph, 1,670 mg/kg; and pyrimisulfan, 1,750 mg/kg.

Figure 3.

Comparison of empirical rat LD₅₀s to the CATMoS predictions for 177 pesticides with CATMoS upper confidence limit (UCL) and lower confidence limit (LCL) based on data-derived inherent variability of +/− 0.24 log₁₀ (Karmaus et al., 2022). Pesticides are sorted from left to right based on predicted toxicity, i.e., most toxic to least toxic. Empirical values associated with a non-definitive “>” value, including limit tests, are represented on this figure as point estimates at 2,000 mg/kg and 5,000 mg/kg. Red horizontal line denotes 1,300 mg/kg. Area in the black box indicates CATMoS point estimates above 2,000mg/kg considered reliable predictors of empirical values.

The ten CATMoS predictions between 1,300 and 2,000 mg/kg also had empirical values of >2,000 mg/kg (Fig. 3). Between 1,000 and 1,300 mg/kg, this relationship breaks down and there are seven under-predictions of toxicity in 23 cases (30%) with four of these outside the CATMoS confidence interval, and 12 cases (52%) where the empirical LD₅₀s were greater than 2,000 mg/kg (Supplemental Table S3). Applying the inherent variability estimate of +/−0.24 log₁₀ mg/kg to the potential thresholds of 1,000 and 1,300 mg/kg yielded a confidence interval at 1,300 mg/kg of 748 – 2,259 mg/kg, which includes the 2,000 mg/kg limit dose threshold. However, the confidence interval at 1,000 mg/kg was 575 – 1,738 mg/kg, which does not include the 2,000 mg/kg threshold. Therefore, a CATMoS prediction of >2,000 mg/kg appears to be a reliable indicator that the corresponding rat acute oral toxicity test would also produce an LD₅₀ >2,000 mg/kg (Fig. 3, area in the black box). There is some evidence that predictions as low as 1,300 mg/kg may be considered reliable indicators of low or no toxicity because the corresponding empirical LD_50s are >2,000 mg/kg, and the confidence limit includes the common limit dose of 2,000 mg/kg.

3.3. Chirality analysis

About 50% of the chemical structures in the list of pesticides were chiral molecules, which makes a good case for assessing their effect on the concordance analysis. We compared concordance of chiral and achiral chemicals based on six LD₅₀ bins with regulatory relevance (Supplemental Table S6 and S7). These bins were obtained by breaking down the four toxicity categories into finer detail, with 500 mg/kg being a boundary between more and less toxic chemicals, with a greater focus on the less toxic chemicals (highest bin was >2,000 mg/kg). The overall concordance was an average BA of 0.61 for the chiral and 0.62 for the achiral chemicals, showing that the model prediction accuracy was not affected overall by the chirality of chemicals. However, this does not mean that accuracy is the same for all structures. Certainly, some stereoisomers will have noticeably different LD₅₀ values, such as the pesticides alpha- and zeta-cypermethrin with LD₅₀s of >2,000 mg/kg and 86 mg/kg, respectively (USEPA, 2017). Thus, it can be informative to identify chiral centers during the assessment and reporting of the toxicity of chemical structures, and when considering the accuracy of CATMoS predictions.

4. Discussion

Alternatives to animal tests are being actively pursued globally, both as 3Rs (Refinement, Reduction, Replacement) initiatives and to improve predictability of human and ecological outcomes. The determination of acute oral toxicity is a typical requirement for many regulatory agencies with regards to ingestion, handling, transport, and incidental exposure to chemical products. In silico models offer the potential to replace animal tests when determining acute oral toxicity without compromising human health or protection of the environment for many classes of chemicals, including pesticides.

In general, most newer, non-rodenticide pesticides have a specific pesticidal mode of action and are considerably less toxic to mammals than older, broad-spectrum pesticides. Indeed, 165 of the 177 chemicals examined here had in vivo empirical LD₅₀s in the less potent categories III and IV (LD₅₀ >500 mg/kg). Our analysis of CATMoS predictions showed good agreement with the empirical values for these categories. This finding is in accordance with the results of the Graham et al. (2021) evaluation of CATMoS for 353 pharmaceutical compounds, which determined that the model can be confidently employed to identify compounds of low toxicity (LD₅₀ >300 mg/kg, i.e., Globally Harmonized System of classification and labelling of chemicals (GHS) category 4 and higher).

4.1. Model prediction outcomes in context of regulatory use

One reason for determining an LD₅₀ is to identify chemicals that pose a greater acute hazard to humans through the oral exposure route than those chemicals with low hazard potential. Therefore, over-predictions by the model, while not necessarily in strict agreement with the animal test value, nevertheless are adequate for labeling purposes. However, the practical outcome of CATMoS over-predictions of category IV pesticides as category III, for example, would have necessitated inclusion of a signal word and precautionary statement on the product label, whereas category IV pesticides are not required to have either.

For environmental RAs, predictions that portray the pesticide as more toxic than empirical in vivo results could lead to unfounded adverse risk determinations for mammalian wildlife. The general trend of over-prediction of toxicity shown by CATMoS led to the determination of a threshold above which the predictions were considered quantitatively reliable. Comparison of empirical LD₅₀ values and CATMoS LD₅₀ predictions showed that point estimate predictions above 2,000 mg/kg corresponded in 73 of 76 cases (96%), to empirical results from limit tests of >2,000 mg/kg in rat studies, commonly thought to be an indicator of non-toxicity.

The 145 of 150 pesticides predicted by CATMoS to have an LD₅₀ of >500 mg/kg (categories III and IV) also had empirical LD₅₀ values of >500 mg/kg, suggesting that there is high confidence that pesticides predicted to be categories III or IV are not more toxic. However, 5/150 were predicted to be category III when, empirically, they were category II. All of these underpredictions were near the lower boundary of category III between 500 and 1,200 mg/kg. Therefore, a category III prediction in this range may need to be supported by additional lines of evidence. In fact, Bercu et al.’s (2021) assessment of the Leadscope QSAR model showed that model predictions accompanied by additional information such as chemical properties and attributes of analogs, increased the percentage of predictions that were correct or more conservative for all GHS hazard categories, and especially for higher toxicity categories I and II. Indeed, using such a weight of evidence approach is considered best practice to improving the overall reliability of any prediction (Myatt et al., 2018).

CATMoS LD₅₀ predictions from 50 – 500 mg/kg (i.e., category II) tended to overestimate toxicity because 19 of the 25 chemicals had empirical LD₅₀ values well above 500 mg/kg, including 14 with empirical LD₅₀ values of >2,000 mg/kg and >5000 mg/kg (all in categories III or IV).

Finally, while there were too few pesticides in this analysis of recently registered TGAIs with empirical LD50 values in the more toxic categories I and II to draw useful conclusions from the comparisons with CATMoS predictions, we performed an evaluation of the model’s ability to accurately predict the toxicities of 16 older TGAIs (registered prior to 1998). Supplemental Table S8 shows that CATMoS correctly predicted 10 out of 11 category I pesticides with only one underprediction as a category II pesticide, and correctly predicted four out of five category II pesticides with one overprediction as a category I and no underpredictions. When combining these category II results with those from the main analysis, the model correctly predicted 10 out of 17, underpredicted five and overpredicted one. While CATMoS has shown that it can provide adequate hazard identification of toxicity categories III and IV, it also performed well for categories I and II in some cases. Nonetheless, USEPA is unlikely to consider a NAM approach for intended or suspected high toxicity chemicals at this time.

4.2. Incorporating variability into animal test and model LD₅₀s

LD₅₀ values are subject to variability whether derived from in vivo laboratory studies or in silico model predictions that are based on in vivo data. Incorporating variability into reference mammalian toxicity tests is an important consideration when setting performance expectations for NAMs (Pham et al., 2020; Rooney et al., 2021). Using +/−0.24 log₁₀ mg/kg confidence interval for CATMoS predictions described above and demonstrated in Fig. 3 builds confidence in the LD₅₀ predictions by helping the user to understand their relationship to the in vivo data submitted for pesticide registrations. Such error limits are driven by factors including dose-response, dose spacing, sex, and other laboratory test conditions, even when complying with standard protocols.

The confidence interval around the predictions may be particularly relevant in regulatory applications in cases where a range is calculated by the acute toxicity method used or a limit test is accepted and can be a more accurate representation of the in vivo LD₅₀. An example is the case of inpyrfluxam that was reported in the ecological and human RAs (USEPA, 2020a; USEPA, 2020b) as having an LD₅₀ for the female rat of 180 mg/kg (95% CI of 30 – 735 mg/kg) using the up-down method (USEPA, 2002; OECD, 2022) and as a range of 50 mg/kg >LD₅₀ <300 mg/kg using the acute toxic class method (USEPA, 2002; OECD, 2002a). CATMoS predicted the LD₅₀ for inpyrfluxam at 294 mg/kg with a confidence interval of 165 – 520 mg/kg, a good match with both estimates provided by the in vivo regulatory guideline methods.

CATMoS LD₅₀ predictions provide a two-level applicability domain assessment based on the distribution and density of the data available in the covered chemical space around the predicted chemical. In addition, a confidence index ranging from 0 to 1 also informs the estimated accuracy of a prediction based on the similarity of the chemical in question to its nearest neighbors and the availability of empirical data for those surrounding chemicals. For example, this confidence index assisted in assessing the reliability of the model estimate for fluensulfone, a category III pesticide predicted by CATMoS to be a category I that had a low confidence index of 0.28. We found that it had poor representation in the model dataset and its five nearest neighbors, all with low toxicity, were similar to each other but not very similar to this chemical. This type of assessment can be used to decide if another method besides CATMoS may be needed to determine acute oral toxicity. Experimental and predicted data, as well as the identifiers of the five nearest neighbors, can be added to the OPERA output for CATMoS predictions, as for all OPERA models. Although it was demonstrated that for the current list of pesticides chirality did not affect the reliability statistics, it still should be considered in cases where chirality is known to affect toxicity.

4.3. Using CATMoS to reduce animal testing

In general, a greater level of uncertainty in CATMoS estimates may be accepted for chemicals with a predicted lower toxicity because there is generally a greater margin of safety relative to exposure compared to more toxic chemicals. With regards to acute oral toxicity categories, CATMoS predictions were reliable for pesticide TGAIs of LD₅₀s >500 mg/kg (categories III and IV). The confidence index provided by the model could be used to decide if additional evidence or in vivo testing is warranted to confirm the appropriate toxicity category in cases where the prediction range straddles the boundary between category II and III. Quantitative use of the LD₅₀ model estimate in RA would be limited to predicted values of >2,000 mg/kg based on the agreement of CATMoS predictions and guideline data above the 2,000 mg/kg threshold.

4.4. Possible further work

While the CATMoS model was shown to perform well for predicting acute oral toxicity of lower-toxicity pesticide TGAIs, far more LD₅₀ tests are conducted annually to determine the acute oral toxicity of end-product pesticide formulations as part of the “six-pack” acute toxicity tests required under 40 CFR Part 158. A mathematical approach that applies a weighted-sum to the toxicity of individual ingredients, known as the GHS Mixtures Equation, has been shown to be useful in identifying low-toxicity formulations (Hamm et al., 2021). With further work, for example, use of in silico models such as CATMoS (that can predict only single molecules) in concert with the Mixtures Equation to predict the acute toxicity of new formulations may be evaluated for their combined predictive capabilities (e.g., Chusak et al., 2021).

As a follow up to this analysis, it is planned that the empirical data of the considered pesticides will be included in the knowledge base of the CATMoS consensus model to improve the predictions. Thus, it is anticipated that the predictions of these same chemicals as well as similar structures will be more accurate. Automatically generated chirality data can also be useful if provided in the CATMoS output with the predictions as an additional piece of information.

5. Conclusions

The USEPA uses the LD₅₀ value from in vivo rat acute oral toxicity studies for two main purposes. One is for informing human-health hazard evaluations and determining the precautionary product label statement about the hazard associated with the product. The second is for informing risk to all mammalian wildlife in environmental risk assessment. CATMoS is an in silico approach to predict those LD₅₀ values. When considering USEPA toxicity categories III and IV together, chemicals predicted by CATMoS to be in this range were consistent with in vivo category assignments 97% of the time. Due to some underestimates of category II pesticides as category III, a weight of evidence approach using multiple lines of evidence could help to understand the reliability of model predictions that are close to the lower boundary of category III. Predictions of pesticide TGAI discrete LD₅₀ values >2,000 mg/kg were found to align well with empirical LD₅₀ results from limit tests or definitive value studies exceeding 2,000 mg/kg. Our evaluation supports the potential use of CATMoS predictions of acute oral toxicity in rats when determining the need for whole animal studies.

Highlights.

• The CATMoS model was evaluated for predicting the rat LD₅₀ of new pesticide technical grade active ingredients (TGAI).

• Predictions were compared to empirical rat acute toxicity data of 177 TGAIs registered by the USEPA from 1998 – 2020.

• Model reliability was high when placing TGAIs in USEPA acute toxicity categories III and IV.

• CATMoS discrete value predictions of 2,000 mg/kg and higher were found to agree with empirical values with few exceptions.

• Our evaluation supports the potential use of CATMoS predictions of TGAI acute oral toxicity in place of animal studies.