Predicting toxicity of chemicals: software beats animal testing

Abstract

We created earlier a large machine‐readable database of 10,000 chemicals and 800,000 associated studies by natural language processing of the public parts of Registration, Evaluation, Authorisation and Restriction of Chemicals (REACH) registrations until December 2014. This database was used to assess the reproducibility of the six most frequently used Organisation for Economic Co‐operation and Development (OECD) guideline tests. These tests consume 55% of all animals in safety testing in Europe, i.e. about 600,000 animals. With 350–750 chemicals with multiple results per test, reproducibility (balanced accuracy) was 81% and 69% of toxic substances were found again in a repeat experiment (sensitivity 69%). Inspired by the increasingly used read‐across approach, we created a new type of QSAR, which is based on similarity of chemicals and not on chemical descriptors. A landscape of the chemical universe using 10 million structures was calculated, when based on Tanimoto indices similar chemicals are close and dissimilar chemicals far from each other. This allows placing any chemical of interest into the map and evaluating the information available for surrounding chemicals. In a data fusion approach, in which 74 different properties were taken into consideration, machine learning (random forest) allowed a fivefold cross‐validation for 190,000 (non‐) hazard labels of chemicals for which nine hazards were predicted. The balanced accuracy of this approach was 87% with a sensitivity of 89%. Each prediction comes with a certainty measure based on the homogeneity of data and distance of neighbours. Ongoing developments and future opportunities are discussed.

1. Introduction

Big data and artificial intelligence are the most recent additions to the toolbox of toxicity assessments. The increasing availability of large, more‐or‐less curated data sets of toxicity data form the basis for this development. Prominent examples are PubChem, ChemBL and ToxRefDB. The strong publication bias for toxic results, which does not really reflect the chemical universe, is a dramatic shortcoming of these databases. A major step forward was the European Registration, Evaluation, Authorisation and Restriction of Chemicals (REACH) legislation, which includes the publication of a robust summary of findings for each registration dossier by the European Chemical Agency (ECHA). These files, however, are not machine‐readable. A download of the data in December 2014 has already found about 10,000 chemicals with 800,000 associated studies (Luechtefeld et al., 2016a). Using natural language processing, large parts of the data were extracted and transferred to a database that could be processed. The database showed a predominance of non‐toxic substances: no Globally Harmonised System (GHS) classification was assigned to more than 20% of chemicals. This analysis might be biased, in part, by the predominance of high‐production‐volume chemicals registered until December 2014.

Although the data are public, ECHA has concerns about ownership and republication, and we have agreed to use the database only for our own research and collaborations and not to publish it until it is approved. ECHA, in turn, made a database publicly available, which included about 15,000 chemicals but appears to be redacted for about 70% of the information otherwise available on their website.

One impressive finding was the high number of repeat animal experiments. On average, every study was carried out three times, although this distribution was very skewed, with, for example, two chemicals tested in the rabbit Draize eye test more than 90 times (69 chemicals were tested more than 45 times) (Luechtefeld et al., 2016b). In particular, the acute and topical tests – often referred to as the toxicological six‐pack – had many repeat studies, allowing a formal analysis of the reproducibility of these tests. Notably, these six tests consumed 55% of all animals in toxicology in Europe (European Commission, 2011). No test carried out a single time can be better than its reproducibility. It will actually be worse, as besides reproducibility the relevance of a test includes aspects such as strain differences, species differences, relevance of dose regimen, relevance of age, human diversity including susceptible subpopulations, etc. However, reproducibility is probably the only one of these shortcomings that can be easily addressed.

2. Assessment of animal test reproducibility

We used the simple method of joint probability of agreement. Based on 350–700 chemicals per hazard, each tested multiple times, the probability that an Organisation for Economic Co‐operation and Development (OECD) guideline animal test would output the same result in a repeat test was 78–96% balanced accuracy (BAC; sensitivity 50–87%) (Luechtefeld et al., 2018a,b). On average, BAC was 81%, which reflected the high prevalence of negative test results. A non‐toxic substance will rarely suddenly become positive, and indeed sensitivity, which reflects whether a toxic finding can be reproduced, was on average only 69%.

These data confirm similar findings for smaller sets of chemicals. For example, a reproducibility of only 77% was found for skin sensitisation by Kolle et al. (2013). Hoffmann (2015) analysed the variability of the local lymph node assay (LLNA) test, using the NICEATM database with more than 60 repeat experiments, the false‐positive rate ranged from 14 to 20% (false‐negative rate 4–5%). With respect to the Draize eye test, Weil and Scala (1971) and Cormier et al. (1996) studied the variability of responses of nine substances tested in up to 24 laboratories showing enormous variability. This was also confirmed in the assessment by Adriaens et al. (2014) of the test's reproducibility.

Others have shown even more pronounced reproducibility issues for ecotoxicological tests (Hrovat et al., 2009). They analysed fish acute lethality (LC₅₀) values for 44 compounds (for which at least 10 data entries existed) that were extracted from the ECOTOX database, yielding 4,654 test records showing variability of test results exceeding several orders of magnitude. There is no indication that reproducibility is better for systemic toxicities (Smirnova et al., 2018): a concordance of 57% was found comparing 121 replicate rodent carcinogenicity assays (Gottmann et al., 2001). Our own analysis (Luechtefeld et al., 2016c) of 28‐day vs 90‐day no observed adverse effect levels (NOAELs) showed no correlation.

Some animal tests are carried out on two species, allowing interspecies variability to be addressed. We showed this earlier for skin sensitisation, for which guinea pigs and mice are commonly used. While both assays were about 90% reproducible, based on 655 chemicals, the two species were only 77% predictable of each other (Luechtefeld et al., 2016c). That is still better than mice and rats in cancer bioassays predicting each other in only 53% of cases (Basketter et al., 2012; Smirnova et al., 2018) or various species predicting each other's results in reproductive toxicity studies by about 60% (Hurtt et al., 2003; Bailey et al., 2005). Species concordance (310 chemicals) for non‐neoplastic pathology between mouse and rat was 68% (Wang and Gray, 2015). Wang and Gray (2015) also showed interspecies concordance for 37 substances tested in the US National Toxicology Program (NTP) of mouse with rats at 57–89% (average 75%) in the short‐term and 65–89% (average 80%) in long‐term studies. The mouse‐to‐rat organ prediction in long‐term studies aligned, on average, in 55% of cases; in short‐term studies with an average of 45%; and for rat to mouse, the averages are 27% and 49%, respectively.

These reproducibility estimates provided benchmarks for what any method calibrated (to avoid the term ‘validated’) against mere test results of single‐tested chemicals can achieve. Any better results represented either overfitting or carefully selected reference substances, representing more information than a single OECD guideline test.

3. Development of a predictive tool for chemical toxicity

After the 2016 study (Luechtefeld et al., 2016a), which was also accompanied by analyses of ocular, skin sensitisation and ocular toxicity (Luechtefeld et al., 2016b,c,d), the potential of such a database to develop a predictive tool for chemical toxicity was explored. Taking the example of skin sensitisation (Luechtefeld et al., 2016c), a rather simple approach asked how the closest chemical neighbour with data could substitute for the experimental result of a given chemical (k‐nearest neighbour with n = 1). Requesting a minimum 75% similarity (Tanimoto index), 2,288 of 3,024 chemicals had a neighbour with data, and the result was the same in 80%. With increasing minimum similarity to 85, 90, and 95%, the number of chemicals with neighbours fell to 1,738, 1,189, and 525 chemicals, respectively, i.e. coverage dropped from 70% to 15%. At the same time, accuracy of prediction increased to 84, 85 and 92%. This means that this simple read‐across was as good as mice and guinea pigs predicting each other in the respective guideline studies (already at 75% similarity) and outperformed the reproducibility of either assay at 95% similarity.

These findings encouraged the development of automated read‐across (Hartung, 2016). They also showed that a certainty can be assigned to a prediction based on the similarity of its neighbour. This is an important advantage not offered by any other current predictive tool. Whether these public data represent a legitimate access to use for registration purposes is an important question. This has fortunately been clarified by ECHA, if used in an aggregated manner for predicting other substances: ‘A registrant would need permission to use protected data to read‐across from a single substance to the target substance. But they would not need this to make a quantitative structure–activity relationship (QSAR) prediction’ (ECHA gives clarity on IP issues for QSAR predictions. Chemical Watch, 5 July 2017).

Our parallel work on read‐across (Patlewicz et al., 2014; Ball et al., 2016; Zhu et al., 2016) prompted the development of a read‐across‐based structure–activity relationship (RASAR). The RASAR models are not traditional QSARs, in which a highly curated, small training data set is used to predict a single property based on chemical descriptors (i.e. classifications per hazard). The method is based as a first step for creating a map of the chemical universe (forced layout graph). This is based on chemical similarity, in which similar chemicals are close and different ones are far from each other. Similarity is assessed in two steps, i.e. first a fingerprinting of the molecules (typically substructures present) and then a metric to express similarity. The most commonly used metric is the so‐called Tanimoto index, which expresses, how many of all substructures in the two molecules are shared. Tanimoto makes use of about 900 such substructures. There is a variety of other metrics that could be used, e.g. Dice, Cosine, Sokal, Kulczynski, McConnaughey, Asymmetric, Braun Blanket, etc. Our initial exploration suggests that no single metric alone performs significantly better than Tanimoto, but that a combination of such metrics could improve the approach.

Any chemical of interest can then in a second step be placed into this similarity map. To our surprise, extending the k‐nearest neighbour approach to more than one neighbour (i.e. informing the prediction by more surrounding chemicals) did not improve predictions to major extent. However, an approach in which the distance to the closest negative and to the closest positive neighbour were used, predictions improved. This means, however, setting two thresholds for a valid prediction, i.e. minimum similarity to the positive and minimum similarity to the closest negative neighbour. This creates a grey zone in which these distances are too similar to classify. Setting these thresholds directly affects sensitivity (% of toxic substances found), specificity (% of real positives among toxic calls), and coverage (percentage of chemicals for which a prediction is possible). We chose a sensitivity of at least 80% for each hazard and balanced the resulting specificity (above 50%) and coverage (aiming at two‐thirds of chemicals). Teaming up with Underwriters Laboratories (UL), a global safety certification company headquartered in Northbrook, Illinois, this was implemented as a web‐based prediction tool (now discontinued) called REACHacross™ (Luechtefeld and Hartung, 2017). This extended database included 58,000 known hazards associated with a chemical, up to 15,000 for each of the six hazards covered (skin sensitisation, eye irritation, skin irritation/corrosion, acute oral and dermal toxicity, and mutagenicity). This allowed a leave‐one‐out cross‐validation, i.e. for each and every chemical with a given hazard information a prediction was made and then compared with the actual result. A prediction was possible for 42,500 cases (73% coverage) with on average 84% sensitivity and 61% specificity. This puts the method roughly on par with the reproducibility of the respective animal tests as discussed above.

This ‘simple’ RASAR only made use of the information of neighbours on the hazard to be predicted. This means, it uses what we know from neighbours about eye irritation to predict the eye irritation of the chemical of interest. So, one model per hazard is developed. There is an enormous wealth in what else we know about the chemical itself and its neighbours, however. This is called transfer learning or data fusion.

We next developed a data fusion RASAR. The published model (Luechtefeld et al., 2018b) uses data on 100,000 chemical structures, calculates 5 billion similarities, and simultaneously makes 190,000 predictions for nine hazards of toxic properties of chemicals (the six endpoints above, plus acute inhalation as well as acute and chronic fish toxicity). The database represents a combination of reliable data sources (PubChem, ECHA, ICE, etc.). It still uses the most common Chemical Similarity (PubChem2d) with the Tanimoto (Jaccard) metric. The network features use proximity to positive and negative neighbours. Data fusion makes use of the other 74 toxicity, biological and chemophysical endpoints. This means that a 222‐dimensional vector is used for the prediction. Machine Learning (logistic regression, random forest) gives probabilistic hazard estimates. Computing Clusters (Apache Spark pipeline) allow this massive scale computing. In a similar cross‐validation as carried out for the simple RASAR, the 190,000 predictions of chemical classifications were 87% correct. Markedly, our model provided 100% coverage for nine hazards with high, balanced accuracies throughout.

4. Conclusions

For the six most used toxicity tests (which use 55% of animals in toxicology in Europe, almost 600,000 animals per year), animal repeat tests showed 81% (balanced) accuracy. The machine learning approach achieved 87% (balanced) accuracy, as shown for 4 to 48,000 chemicals with animal data‐based classifications. The comparison was even more impressive when looking for the identification of toxic substances (sensitivity), in which the repeat animal test succeeded in 69% of cases and the machine learning in 89%.

In the meantime, the model has been implemented with customisable user interfaces as the UL Cheminformatics Tool Kit and is available for beta‐testing. As a new feature, it now also predicts the different GHS classes as a measure of potency. Notably, this feature has not yet been validated. The tool kit allows installation behind a firewall, combination with proprietary data and customised user interfaces. Features such as running lists of chemicals, chemical design, one‐on‐one comparisons for alternative chemistry, and identification of candidate alternative chemicals are currently implemented.

Evaluation and validation with various data sets in agencies and industry is on the way. These efforts are integrated into the EUToxRisk flagship project (Daneshian et al., 2016) and attempts to further the regulatory acceptance of read‐across (Chesnut et al., 2018). The data fusion RASAR demonstrated the potential of big data and machine learning to reduce animal testing for acute and topical endpoints.

Besides REACH and other global legislation for new and existing chemicals, the RASAR approach lends itself to prioritisation, classification and labelling, emergency risk assessments, and risk assessment for new products allowing frontloading of risk assessments (i.e. Green Toxicology) (Maertens et al., 2014; Crawford et al., 2017; Maertens and Hartung, 2018). Another interesting use is for determining impurities, for example, in pharmaceuticals; these can be easily detected but the effort to synthesise large quantities and test them is usually not possible. These examples might suffice to illustrate possible uses, which we have only started to explore with the advent of artificial intelligence and big data in safety sciences.

Abbreviations

BAC

balanced accuracy

ECHA

European Chemical Agency

GHS

Globally Harmonised System

LC₅₀

lethal concentration, median

LLNA

local lymph node assay

NOAEL

no observed adverse effect level

NTP

National Toxicology Program

OECD

Organisation for Economic Co‐operation and Development

QSAR

quantitative structure–activity relationship

RASAR

read‐across‐based structure–activity relationship

REACH

Registration, Evaluation, Authorisation and Restriction of Chemicals

UL

Underwriters Laboratories

Suggested citation: Hartung T, 2019. Conference article on predicting toxicity of chemicals: software beats animal testing. EFSA Journal 2019;17(S1):e170710, 8 pp. 10.2903/j.efsa.2019.e170710