Large-scale Modeling of MuAlti-Species Acute Toxicity Endpoints using Consensus of Multi-Task Deep Learning Methods

Abstract

Computational methods to predict molecular properties regarding safety and toxicology represent alternative approaches to expedite drug development, screen environmental chemicals, and thus significantly reduce associated time and costs. There is a strong need and interest in the development of computational methods that yield reliable predictions of toxicity, and many approaches, including the recently introduced deep neural networks, have been leveraged towards this goal. Herein, we report on the collection, curation, and integration of data from the public datasets that were the source of the ChemIDplus database for systemic acute toxicity. These efforts generated the largest publicly available such dataset comprising > 80,000 compounds measured against a total of 59 acute systemic toxicity endpoints. This data was used for developing multiple single- and multi-task models utilizing Random Forest, deep neural networks, convolutional and graph convolutional neural network approaches. For the first time, we also reported the consensus models based on different multi-task approaches. To the best of our knowledge, prediction models for 36 out of the 59 endpoints have never been published before. Furthermore, our results demonstrated a significantly better performance of the consensus model obtained from three multi-task learning approaches that particularly predicted the 29 smaller tasks (less than 300 compounds) better than other models developed in the study. The curated dataset and the developed models have been made publicly available at https://github.com/ncats/ld50-multitask, https://predictor.ncats.io/ and https://cactus.nci.nih.gov/download/acute-toxicity-db (dataset only) to support regulatory and research applications.

Graphical Abstract

INTRODUCTION.

Early in silico assessment of small molecule toxicity is an indispensable step in drug discovery and development that helps reduce costs and labor, can inform on regulatory decision making, and has parallel applications in environmental chemical screening and prioritization^1–6. The advent of big data in chemistry and biology, complemented by advances in screening technologies, has enabled the development of large-scale toxicity prediction models^4,7–10. While the ChEMBL database¹¹ serves as a major public resource of compound bioactivity data, there are other open-access databases that provide information on the toxicity of small molecules, such as TOXNET¹² (www.nlm.nih.gov/toxnet/index.html) and DSSTox¹³. Dedicated resources with information related to specific toxicity endpoints are also becoming increasingly available^14–16. RTECS® (Registry of Toxic Effects of Chemical Substances) is currently made available as a proprietary database that provides in vivo data for more than 180,000 chemical substances with a major focus on acute toxicity¹⁷. ChemIDplus¹⁸ on the other hand is a publicly available database that contains more that 150,000 compounds having acute systemic toxicity outcome records (e.g., lethal dose, 50% or LD₅₀) in different species and multiple routes of administration.

A wide range of quantitative structure-activity relationship (QSAR) methods has been employed for computational toxicity prediction^19–29. Machine learning methods such as Random Forest and Support Vector Machines have served as popular tools for building cheminformatics models^30,31, and more recently, neural networks have emerged as robust methods that perform exceedingly well on large datasets and provide better extrapolation in comparison to traditional QSAR models^4,32–36. However, models trained on small datasets often result in poor predictive performance on unseen external data^37,38, and could potentially benefit from learning on biologically related endpoints. In this context, multi-task learning facilitates simultaneous modeling of multiple endpoints to develop better models, particularly when the endpoints are mechanistically correlated with one other^36,39–41. Erhan et al.⁴² were the earliest to report single-task and multi-task predictive models for a family of biological targets in 2006. Later, a series of unified QSAR models were developed to predict antimicrobial activity of drugs against multiple fungal and bacterial species^43–45. In 2008, Varnek et al.⁴⁶ applied multi-task learning for modeling 11 types of tissue-air partition coefficients along with another inductive knowledge transfer technique known as Feature Net. In Feature Net approach, additional tasks are used to build models, predictions from which are used as descriptors for modeling the main task. Here, in the case of acute systemic toxicity, there are multiple specific endpoints in ChemIDplus¹⁸ that have a limited number of data points (e.g., human, cat, and rabbit lethal doses), but the expectation is that there is a finite number of mechanisms by which chemicals cause lethality and that these would be fairly consistent across species. Compounds that have data in one endpoint would then inform predictions on structurally similar compounds for related endpoints.

In 2018, at the CATMOS Meeting, Zakharov et al. proposed multi-task deep learning approach to model toxicity across 9 different endpoints^47,48. Later, Sosnin et al.⁴⁹ extended this approach for a total of 29 toxicity endpoints using data from RTECS® to report that these models outperformed single-task models based on other machine learning methods. More recently, Zakharov et al.⁴ proposed a novel deep learning consensus architecture (DLCA) to model more than 1000 endpoints using bioactivity data available from publicly accessible resources such as ChEMBL and Tox21 (https://tripod.nih.gov/tox21). Different types of graph convolutional neural networks (GCNN) were proposed and validated on benchmark datasets that include several multi-task classification and regression tasks^50–52. Other studies showed that the implementation of transfer learning^53–55 led to improved model performance^40,41,56. In the toxicity domain, studies have used Tox21⁵⁷, ToxCast⁵⁸, SIDER⁵⁹, and the recently introduced ClinTox dataset for benchmarking different machine learning methods^19,50,57. While most of these datasets comprise multiple endpoints, consensus of multi-task approaches has not been extensively investigated. However, consensus modeling approaches have been reported to outperform simple QSAR models^60–62. Thus, in this study, we consider consensus approaches that combined predictions from different multi-task learning approaches to predict acute toxicity.

Considering the chemical space coverage of the dataset and the sparsity of the measurements against 59 different endpoints, we thought that the acute toxicity data from ChemIDplus¹⁸ could be an ideal case study, both to improve existing models using multi-task learning and to implement and test new multi-task algorithms. Although several groups^{24,25,49,63–65} previously reported QSAR models for acute toxicity endpoints, most of which are publicly accessible, much of the modeling data were not made publicly available and therefore the quality could not be assessed. Here, we formulated the primary goals of the study as follows: i) collect, curate, and integrate all available data for systemic acute toxicity into the largest publicly available multi-species toxicity dataset; ii) use this dataset for benchmarking and comparing state-of-the-art single- and multi-task machine learning methods; iii) use the most recent advances in multi-task modeling to improve existing, or develop novel, models for a total of 59 acute toxicity endpoints spanning multiple species and routes of administration. Of the 59 endpoints, prediction models for 36 endpoints have not been previously reported in literature to the best of our knowledge.

MATERIAL AND METHODS.

Dataset.

The quality of experimental data is a crucial part of building machine learning models. In this study, we used data which are publicly available from ChemIDplus¹⁸. A set of 165,182 measurements related to 456 endpoints, representing 25 dosing routes across 28 species and expressed as LD₅₀, lethal dose low (LD_Lo) and toxic dose low (TD_Lo) were extracted. The dataset consists of toxicity measurements in different units (mg/kg, mL/kg, gm/m³ and many others). In order to have a harmonized dataset, we considered the data from three measurement units: mg/kg, μg/kg, and ng/kg. This led us to a dataset of 159,968 measurements for 91,642 compounds tested against 437 endpoints.

Data curation.

The initial dataset containing 91,642 compounds was curated following a protocol previously developed by Fourches et al.^66–68. Briefly, salts and solvents were stripped from all compounds followed by removal of counterions, large organic compounds (Da >= 2,000), mixtures, and inorganic compounds. Specific chemotypes such as aromatic, nitro groups, sulfo groups, tautomers, and protonation state were standardized using the ChemAxon Standardizer software (https://chemaxon.com/)⁶⁹. If duplicates presented discordant potencies (i.e., > 0.2 −log units), both entries were excluded; if the reported potencies were similar, an average of the values was calculated, and one entry was retained in the dataset. Stereocenters were kept and enantiomers analyzed. After curation, 85,848 compounds and 255 endpoints were retrieved. In order to generate reliable prediction models, we removed the endpoints that had less than 100 reported measurements. This led us to a dataset of 80,081 unique compounds with 122,594 measurements against at least one of the 59 endpoints. Table S1 in the Supporting Information provides information on the number of measurements across each endpoint.

Molecular Descriptors.

Although the use of public descriptors in combination with commercial descriptors to model acute toxicity was previously reported⁴⁹, we intended to stick to descriptors available in the open source domain. In 2018, Zakharov⁴⁷ reported the superior performance of multi-task deep learning models for acute toxicity endpoints with Avalon fingerprints in comparison to Morgan fingerprints and RDKit descriptors. Therefore, we decided to use only Avalon fingerprints (1024 bits)^70,71 in this study and calculated them using the RDKit Fingerprints node⁷² available in the KNIME analytics platform⁷³.

Machine Learning Methods.

We used deep neural networks (DNNs) to build both multi-task (MT-DNN) and single-task (ST-DNN) models. In addition, we used Random Forest to build single-task (ST-RF) baseline models due to their widespread application and robust performance in cheminformatics and machine learning^74–77. These methods are briefly explained below.

Deep Neural Networks (DNN)

DNNs have been reported to outperform most other machine learning methods for the prediction of molecular properties^4,49,78,79. A DNN is an alteration of an artificial neural network (ANN) that consists of several sequential hidden layers. Each layer in a DNN is represented by a linear vector transformation Wx+b where W is a matrix of tunable weights and b is a bias vector, followed by a nonlinear transformation function (i.e., sigmoid). In our study, we developed multi-task DNN models utilizing the multi-layer feedforward neural networks implemented in Keras⁸⁰ using the Tensorflow backend⁸¹. The loss function was minimized using the Adam algorithm⁸². In order to further identify the best hyperparameters for DNN, we used the grid search function available from the scikit-learn⁸³ library. The grid search was performed for the following parameters: (i) number of epochs; (ii) batch size; (iii) activation function, (iv) learning rate of Adam optimizer, and (v) dense layer candidates, i.e., the number of neurons in each dense layer. The detailed list of hyperparameters optimized for the MT-DNN model can be found in Table S2 in the Supporting Information. While some parameters were fixed based on previous experience with the dataset, some were exhaustively searched to find the optimal performing hyperparameters. In the case of single-task DNN models, we used the best performing hyperparameters from the multi-task DNN since it would not be practical to evaluate an extensive list of hyperparameters over 59 different tasks individually. However, the learning rate for Adam optimizer was tuned for each task separately.

Random Forest

Random Forest (RF) is an ensemble of decision trees⁸⁴. In this study, the single-task regression models (ST-RF) were built using the RF implementation in scikit-learn⁸³. The number of trees was arbitrarily set to 100, since it has been shown that the optimal number of trees is usually 64 – 128, while further increasing the number of trees does not necessarily improve the model’s performance^75,85. Due to the robust nature of RF⁸⁶, no parameter optimization was performed.

Model Benchmarking

In addition to the single-task baseline models, we also benchmarked our DNN models with models reported in the literature. We first explored the ‘deep learning consensus architecture’ (DLCA) proposed by Zakharov et al.⁴. The approach averages the outputs of separate DNNs built using different descriptors inside a single neural net. This imposes a constraint on the learning algorithm to prevent propagation of corresponding errors, which leads to an improvement in the consensus results. In this study, we developed a DLCA model that combines descriptors-based and so-called descriptors-free models. The descriptors-based models were generated using three different types of fingerprints (Morgan, Avalon, and AtomPair), and RDKit descriptors. The descriptors-free model was created using SMILES notation and a convolutional neural net architecture based on 1D convolutional and GlobalMax pooling layers following by hidden dense and output layers (the architecture and training parameters are provided in Table S2 in the Supporting Information).^87,88).

Next, we used the recently published graph convolutional neural networks (GCNN)^52,89–93. In this study, we developed multi-task GCNN models by using a message-passing variant of GCNN as implemented in ChemProp⁵¹. These networks construct a learned molecular representation by operating on the graph structure of the molecule. Further, we also performed hyperparameter grid optimization and used the best settings to generate models for final validation. Optimization for the GCNN models was performed as proposed by Swanson et al.^51,94,95.

Consensus Models

In this study, we developed two different consensus models from the best performing individual multi-task models. The first consensus model is based on the multi-task DNN model with hyperparameter grid optimization (MT-DNN), and multi-task GCNN model with grid optimization (GCNN). This is referred to as ‘consensus A’ in the rest of the study. The second is based on MT-DNN, GCNN, and multi-task DLCA models, referred as ‘consensus B’ in the rest of the study.

Model Validation and Statistical Performance.

To estimate the performance of the models developed in this study, we applied a 5-fold cross-validation procedure⁹⁶. The dataset was randomly subdivided into five parts, where four parts were used as the training set for model building, and the remaining part was used as the test set for the assessment of predictive accuracy. As it was observed that the selection of hyperparameter plays a crucial role in the model performance⁹⁷, in-order to have a fair and unbiased comparison, the best hyperparameters were selected based on grid search performed on the first fold of the dataset and applied on the remaining four folds. Further, in addition to random split, we also applied a scaffold-based splitting procedure as proposed by Yang et al.⁵¹

The performance of each model for 5-fold CV procedure was assessed on the basis of root mean squared error (RMSE) (Eq. 1), and determination coefficient R² (Eq. 2),

$$\mathit{\text{RMSE}}=\sqrt{\frac{1}{n}{\sum }_i^n{\left({\widehat{Y}}_l-Y_i\right)}^2}$$ (1)

$$R^2=1-\frac{{\sum }_i^n({\left({\widehat{Y}}_l-Y_i\right)}^2}{{\sum }_i^n({\left(Y_i-\overline{Y}\right)}^2}$$ (2)

$\widehat{Y}$ is the predicted value for each particular compound; Y_i is the observed value for each particular compound; $\overline{Y}$ is the mean value over all compounds; n is the number of compounds.

The difference between the model performance was evaluated using the Wilcoxon paired singed-rank non-parametric statistical test. For the given two methods, the predicted performance (RMSE and R²) for each of the 59 tasks was compared pairwise to identify the method that significantly outperforms the other. We defined the statistical significance as p-value less than 0.05.

Calculation of the Applicability Domain.

Applicability domain (AD) is a crucial part of the QSAR methodology that, if used correctly, may significantly improve the prediction results^4,98. There are multiple ways to calculate the applicability of a QSAR model^99–103. In this study, we used two different approaches for estimation of the model’s AD. In the first approach, we estimated the Tanimoto similarity^104,105 between the test set compounds and nearest neighbor in the training set using Morgan fingerprints. For each fold, we filtered out those compounds that were below a certain similarity threshold and further calculated the RMSE (endpoint-wise) and the coverage of predictions as the percentage of compounds that fall within the model’s AD. In the second approach, since the DLCA model⁴ provides an integrated output from models based on different descriptors, we extracted the prediction output for each compound from individual descriptor models and calculated the standard deviation (SD) of prediction for each compound. Then, for each endpoint, we calculated the mean (μ) and standard deviation (σ) from the standard deviation of prediction for each compound. We then filtered out those compounds that were above the μ + 0.5 σ (t1), μ + σ (t2), μ + 2σ (t3) and μ + 3σ (t4), simultaneously and calculated the RMSE and coverage on the remaining.

RESULTS AND DISCUSSION

Data Overview.

After curation, 80,081 compounds remained in the dataset, with 122,594 measurements available for 59 endpoints. However, not all of these compounds were measured for all endpoints. The most frequently reported endpoints were for mouse, rat, and rabbit. For mouse, there were 12 different measurement types, i.e. combinations of dosing routes and acute systemic outcomes expressed as LD₅₀, LD_Lo, and/or TD_Lo were reported (70,442 unique measurements). The second most reported species was rat (14,948 unique measurements) followed by rabbit (3,447 unique measurements) with 11 and 9 different measurement types, respectively. Oral, intravenous, and subcutaneous were the 3 most frequently studied routes of administration. The sparsity of the data matrix (80081×59) was found to be >97% (Table S3 in the Supporting Information). Next, PCA plots were generated based on Avalon fingerprints and the median acute systemic toxicity values across different endpoints for each molecule (Figure 1). Overall, the compounds span a fair extent of chemical space and potency. As can be seen, the majority of the overlapping chemical structures do not have distinct toxicity profiles. This indicates that the modelability¹⁰⁶ of the dataset should be high, because ‘structurally similar molecules tend to exhibit similar properties’. This supports the idea of applying multi-task learning, in which the smaller tasks (endpoints with less than 300 measurements) are simultaneously learned with the larger tasks and the learner optimizes the performance across all tasks.

Figure 1.

Two-dimensional PCA plot for the complete dataset based on Avalon fingerprints. The color scale represents the median toxicity value of the compounds against different endpoints in −log(mol/kg), i.e., the higher the values, the more toxic the compounds.

Modeling Results.

We evaluated the performance of multi-task regression models for different acute systemic toxicity endpoints, across species and dosing routes. We built multi-task DNN (MT-DNN), single-task models (DNN and Random Forest), multi-task DLCA (DLCA) and GCNN models. In order to avoid any bias that might occur due to the splitting schemes employed, all models were evaluated in a five-fold cross-validation scheme^96,107. Figure 2 provides a comparison of the average performance (RMSE, R²) over 59 endpoints for different models generated in this study. The best results (average RMSE = 0.65; average R² = 0.57) were obtained from the consensus B models, which is a consensus of the predictions from multi-task DNN, GCNN, and DLCA models. The DLCA models alone provided an average RMSE of 0.68. In general, our MT-DNN model performed slightly better than the GCNN model. The single-task DNN models on the other hand, performed the worst amongst all models in all folds. Though the single-task models based on Random Forest provided better performance than single-task DNNs, their performance was inferior compared to the multi-task models. The superior performance of the Random Forest model could be due to the robustness of algorithm¹⁰⁸ as compared to the DNNs that require relatively large datasets in order to fit the hidden layers^109,110. A similar performance trend was observed with the R² values (Table S4 in the Supporting Information). Except for MT-DNN and GCNN, the difference in performance for any given pair of methods was found to be statistically significant (p < 0.05; Table S5 in the Supporting Information).

Figure 2.

Average performance (a) RMSE, (b) R² of all 59 endpoints for each approach over five-fold cross-validation based on training and test data generated using random splitting. The error bar represents the standard deviation of the average performance over five-folds.

With respect to specific endpoints, our best model (consensus B) predicted LD₅₀ values fairly well for several species and several routes of administration, for example mouse oral, intravenous, intraperitoneal; rabbit skin; rat intraperitoneal, skin and others. The mouse oral LD₅₀ had the best RMSE (RMSE = 0.43, R² = 0.50), and was one of the most frequently measured endpoints with 23,373 values in the final dataset. The rabbit subcutaneous LD_Lo endpoint had the highest R² value (R² = 0.76, RMSE = 0.61) for our consensus B model, although it had one of the lowest incidences, with only 241 measurements. It should be noted that LD_Lo was predicted with lower accuracy than LD₅₀ toxicity for all species and route of administration types, followed by TD_Lo. A possible reason could be that LD₅₀ endpoints have comparatively higher numbers of measurements since it is more often evaluated as compared to TD_Lo and LD_Lo. Moreover, TD_Lo and LD_Lo are non-standard toxicity measurements and thus are less reliable due to lack of harmonized protocols causing variability in experimental conditions. The detailed model performance statistics can be found in Table S6 in the Supporting Information.

In addition to ‘random split,’ we also performed ‘scaffold split’, which is challenging, but a more realistic evaluation of the predictive power of the models^4,51. ‘Scaffold split’ ensures that there is no molecular scaffold overlap between the train and test sets which indirectly mimics the evolution of new chemical space. As the ultimate goal of modeling is to predict properties of newly synthesized chemicals, performance assessment using ‘scaffold split’ can be considered a more realistic evaluation where new chemicals may not bear any resemblance to compounds in the training set¹¹¹. In concordance with this, our results indicate superior performance of ‘random split’ in comparison to ‘scaffold split’. For the scaffold-split, DLCA model showed the best prediction results, followed by MT-DNN and GCNN models which provided similar performance. Detailed model statistics are provided in Table S4 in the Supporting Information.

Comparison to Previous Studies.

Outside of our own presentation at CATMOS meeting,^47,48 only Sosnin et al.⁴⁹ reported multi-task models for a total of 29 toxicity endpoints. They provided a comparison of both multi-task and single-task models using a wide range of molecular descriptors. It was shown that the best performance was obtained by averaging of the predictions of the top-five individual multi-task models (RMSE = 0.68; on 29 endpoints). In our study, the consensus B model combining three multi-task approaches provided the best performance with RMSE = 0.65 and R² = 0.57 on 59 endpoints. Although we would like to benchmark the performance of our models against the results of Sosnin et al.,⁴⁹ direct comparison is impossible because of the different numbers of compounds and endpoints. For some overlapping endpoints, we have fewer compounds in our dataset because of a more rigorous data curation procedure applied in this study. Furthermore, the raw data for our study were obtained from the ChemIDPlus portal, and therefore could have different numbers of compounds and measurements compared to the latest version of RTECS® dataset available from commercial vendors. These two reasons outlined above may explain the discrepancy in the number of measurements across different endpoints in Table S1 (in the Supporting Information). Ideally, future comparisons will be possible using newly obtained data. Despite the challenges in directly comparing the results, we checked for toxicity endpoint overlap with Sosnin et al.,⁴⁹ and found that 23 were in common with the 59 endpoints addressed in this study. Of these 23 endpoints, we noticed that only 18 endpoints had a comparable number of measurements in both studies, considering a threshold of at least 300 measurements per endpoint (Table S1). We therefore drew parallels between both studies for these 18 relatively similar datasets. The best results from Sosnin et al.⁴⁹ were achieved by a consensus model (RMSE = 0.54, R² = 0.60). Our consensus B model provides an RMSE of 0.53 (R² = 0.61) on the same 18 endpoints (Table S7 in the Supporting Information). Furthermore, we provide curated data and prediction models for 36 acute toxicity endpoints that to the best of our knowledge have never been published before. These include different combinations of species (dog, chicken, rabbit etc.), exposure route (oral, skin, intramuscular, etc.) and dose metric (LD₅₀, LD_Lo, TD_Lo). While most of these represent non-standard endpoints in terms of internationally harmonized OECD guidelines, such studies are often performed and submitted to regulatory authorities as part of chemical toxicity evaluation packages. Ideally, the models presented here would substitute for future studies being performed using animals, saving considerable resources and providing reliable predictions using alternative approaches that have been trained on information from multiple species.

Multi-task Models versus Single-task Models.

It is clear from the results (Figure 3 and Table S6 in the Supporting Information) that our multi-task DNN models outperformed the single-task models on the smaller tasks (endpoints with fewer chemicals tested). This is expected according to the results from previous studies^{39,49,112–114} and due to the ability of multi-task methods to co-learn larger and smaller tasks¹¹⁵. Thus, multi-task models can learn from related tasks and thus tend to provide better performance on small (related) tasks compared to a single-task model trained using a smaller dataset. This emphasizes the advantage of using multi-task learning approaches for such understudied endpoints.

Figure 3.

Performance (a) RMSE, (b) R² of the best multi-task models obtained using different architectures and the single-task models for different endpoints ordered by the total number of measurements available in the dataset. (IM: intramuscular; IP: intraperitoneal; IV: intravenous; P: parenteral; S: skin; SC: subcutaneous; U: unreported; O: oral; mammal: mammal (species unspecified))

Applicability Domain Analysis.

Applicability domain (AD) of a QSAR model defines the limitations in its structural domain and response space. In this study, based on the two approaches (as presented in the ‘Materials and Methods’ section: ‘Calculation of the Applicability Domain’), the RMSE and the corresponding coverage (for the predictions from the DLCA model) were calculated and are presented in Figure 4 below with respect to the threshold values of AD (0.1–0.9) and SD (t1–t4) cut-offs. Figure S1 in the Supporting Information shows the R², AD cut-off, SD cut-off and the corresponding coverage.

Figure 4.

Distribution of the DLCA prediction results (RMSE) and coverage values over AD (0.1–0.9) and SD (t1–t4) cut-offs. The error bar represents the standard deviation of the average performance over five-folds.

Figure 4 shows an inverse correlation between the coverage and the accuracy of model prediction, meaning the higher the AD threshold, the better the accuracy of the model (lower RMSE) as expected. Based on the second approach, the higher the SD cut-off, the less accurate the model’s predictions (greater RMSE). The best results were obtained with AD = 0.9, which resulted in an RMSE value of 0.54 and 8% as the coverage of prediction. Considering both the coverage and the prediction accuracy, we found that t1 (mean + 0.5 SD) cut-off provides an optimal ratio between them, resulting in an RMSE value of 0.60 and coverage of 82%. Considering the Tanimoto similarity values, those predictions satisfying an AD threshold of 0.7 can be regarded as reasonable predictions (RMSE= 0.60; Coverage = 51%). Thus, both AD approaches could be used to select compounds with certain prediction confidence.

Online Service for Prediction of Acute Toxicity Profile of Chemical Compounds.

The MT-DNN model developed during the study are accessible via the NCATS Predictor (https://predictor.ncats.io/). Users can provide different molecular representations such as SMILES, SDF (structure data format) files or two-dimensional images of chemical structures as input. As an output, the online interface provides predictions for all 59 endpoints and reports the applicability domain assessment for each compound based on different models. The applicability domain calculation is based on the Tanimoto similarity to the nearest compound within the training set. A compound with a similarity value to the nearest neighbor falling in the region of (i) 1 to 0.7 is considered to be predicted with a high confidence; (ii) 0.7 to 0.5 is considered to be predicted with a medium confidence; (iii) less than 0.5 is considered to be predicted with a low confidence. This web service is provided to help researchers and regulators rapidly identify and prioritize compounds with toxic liabilities and gain additional insights based on the predicted profiles against the 59 multi-species acute-toxicity endpoints.

CONCLUSIONS.

Predicting molecular properties of small molecules is an essential step in modern drug discovery and environmental chemical assessment. Increasingly accurate computational methods for toxicity prediction are facilitated by data availability, novel algorithms, and computing power. Herein, we report on the collection, curation, and integration of all freely available data for systemic acute toxicity into the largest publicly available dataset (59 multi-species acute systemic toxicity endpoints and more than 8000 compounds). We used it for the development of deep-learning-based multi-task models and benchmarking them against state-of-the-art modeling techniques such as RF and recently proposed graph neural network architectures. We demonstrate that the MT-DNN approach offers a statistically significant advantage over single-task models, especially for endpoints with smaller number of compounds. Among multitask models, the DLCA model showed the best performance for both random and scaffold splitting procedures. Consensus predictors constructed from the results of MT-DNN, GCNN, and DLCA yielded the statistically highest predictive power. Both the curated acute toxicity dataset and the best performing models are made freely accessible to the research and regulatory community via the NCATS Predictor (https://predictor.ncats.io/), https://github.com/ncats/ld50-multitask as well as https://cactus.nci.nih.gov/download/acute-toxicity-db (dataset only) and can be readily used to predict and analyze acute toxicity of small molecules measured for different species and routes of administration.

Supplementary Material

Supplemental Figures

Figure S1. Distribution of the DLCA prediction results (R²) over AD cut-offs and coverage values. The error bar represents the standard deviation of the average performance over five-folds. Figure S2. Average performance (a) RMSE, (b) R² of all 59 endpoints for each approach over five-fold cross-validation based on training and test data generated using random splitting. The error bar represents the standard error mean of the average performance over five-folds.

Supplemental Tables

S1. Provides information on the number of measurements across each endpoint extracted from the ChemIDPlus portal¹⁸. It also summarizes the extent of data overlap between the endpoints modeled in this study in comparison the study published by Sosnin et al.⁴⁹ S2.The network architecture for MT-DNN, ST-DNN and DLCA and the list of hyperparameters optimized for the MT-DNN. S3. Provides information on the proportion of missing data across different endpoints. S4. Model statistics (5-fold cross-validation results) of multi-task regression models developed in this study. S5. Statistical analysis of prediction results using Wilcoxon paired singed-rank non-parametric statistical test. S6. Average RMSE of prediction across 5 folds for all endpoints by different models. S7. RMSE comparison of the overlapping endpoints for the models developed in this study and study published by Sosnin et al.⁴⁹

ABBREVIATIONS

RTECS®

Registry of Toxic Effects of Chemical Substances

QSAR

Quantitative Structure-Activity Relationship

GCNN

Graph Convolutional Neural Networks

LD₅₀

Lethal Dose, 50%

LD_Lo

Lethal Dose low

TD_Lo

Toxic Dose low

DNN

Deep Neural Network

MT-DNN

Multi-Task Deep Neural Network

ST-DNN

Single-Task Deep Neural Network

ST-RF

Single-Task Random Forest

RF

Random Forest

DLCA

Deep Learning Consensus Architecture

RMSE

Root Mean Squared Error

AD

Applicability domain

SD

Standard Deviation

PCA

Principal Component Analysis

CATMOS

Collaborative Modeling Project for Predicting Acute Oral Toxicity

OECD

Organization for Economic Co-operation and Development