DDI–CYP: Metabolism ensemble models for drug-drug interaction predictions

Abstract

Cytochrome P450 (P450)-mediated drug-drug interactions (DDIs) are responsible for most adverse drug interactions, and occur when 2 concurrently administered drugs inhibit, upregulate, or are substrates of the same target enzyme. A machine learning approach enables the detection of DDIs with rarely used drugs, as well as newly approved drugs. To facilitate this, we present a framework for predicting DDIs by first predicting P450 interactions for both drugs, generating a fingerprint based on the predictions in addition to the molecular structures of the drugs, and training a machine learning model to predict the overall interaction. After optimization, the model detected potential DDIs with 85% accuracy, representing an improvement over a DDI-only model (ie, a model trained on structure-based fingerprints without supporting P450 model predictions). We also present a corresponding adverse outcome pathway to allow for increased model explainability through visualizing each predicted P450 interaction, further enhancing its real-world applicability. Finally, we show the importance of the model applicability domain to DDI models by demonstrating how the performance of our model degrades as the inference set becomes dissimilar to the training data.

Significance Statement

Polypharmacotherapy (especially in older populations) results in more drugs prescribed concurrently, creating an increased risk of drug-drug interactions and adverse drug reactions. Computational tools to predict potential drug-drug interactions could accurately aid in reducing risk for the patient and be used in the early stages of drug design to avoid such undesirable molecular interactions.

Visual Abstract

Schematic of drug-drug interaction models with supporting machine learning models. The 3 dots are intended to denote that additional data and models were also included in pretraining.

1. Introduction

The major family of enzymes responsible for phase 1 metabolism is the cytochrome P450 (P450) superfamily.1, 2, 3, 4 These enzymes are ubiquitously expressed in different tissues, such as the lung and liver, and are a first line of defense against xenobiotics.5, 6, 7 Metabolism by the main drug-metabolizing enzymes, such as CYP2C9, CYP2D6, and CYP3A4,^8,9 has been modeled computationally. These models, which have included quantitative structure activity relationships, 3-dimensional pharmacophores, and docking, have used a wide range of inputs, such as molecule structure, to predict the affinity of a molecule for each P450 enzyme or its inhibition.9, 10, 11, 12, 13 Molecules can interact with each P450 enzyme depending on their shape, size, electrostatics, and molecular properties, which can result in what are termed drug–drug interactions (DDIs) when 2 or more molecules interact with the same enzyme. These DDIs may also occur with other phase 1 as well as with phase 2 enzymes^14,15 and transporters impacting drug development.¹⁶

More broadly, DDIs are defined as the alteration of a drug’s activity due to the coadministration of additional drugs.¹⁷ DDIs can be classified as pharmacodynamic or pharmacokinetic. Pharmacodynamic DDIs can be synergistic, additive, or antagonistic, and occur when the pharmacological effect of one drug is changed by another. On the other hand, pharmacokinetic DDIs occur when the absorption, distribution, metabolism, or excretion of one drug is affected by another drug. DDIs can range in severity, which is typically classified as minor, moderate, or major.¹⁸ That said, DDI classification is somewhat subjective, as it is typically assessed by clinical relevance according to a physician. Minor DDIs typically do not require any changes in treatment, but major DDIs can be life-threatening and require immediate medical attention. A positive correlation between severe or multiple DDIs and death among elderly patients has been found.¹⁹ One study found that most clinical DDIs do not require additional care, as they either do not manifest or do not cause significant issues.²⁰ Regardless, when DDIs are present in patients, they typically are severe.²¹

Polypharmacotherapy (the concurrent use of multiple medications) is becoming more prevalent to treat complex conditions, especially in elderly populations.^22,23 A 2019 study found that within a population of 237 retirement home residents, over 87% used 5 or more drugs concurrently.²² Additionally, over 43% used 10 or more drugs concurrently. As the number of drugs increases, the likelihood of potential DDIs increases dramatically, meaning that the prescription of concurrent drugs becomes increasingly complex.^24,25

In practice, massive DDI data bases, such as Micromedex Drug-Reax,²⁶ are used to evaluate drug combinations for potential DDIs.²⁷ However, these data bases are typically compiled from reports in the literature, meaning that a DDI must be observed first for it to be listed in the data base.^28,29 In silico methods have the advantage of being able to virtually evaluate all possible combinations of a set of drugs for DDIs, even if no information is available in the literature about their interactions.³⁰ This may be especially beneficial for applications of novel or infrequently used drugs. Additionally, “in silico” methods provide an opportunity to decrease the number of costly in vivo or in vitro DDI studies, which can save resources and time.³¹

We now describe using DDInter,²⁹ as we found its structure and comprehensiveness to lend itself well to our study, as we rely on the P450-related metabolism-specific DDI subdataset. DDInter is a comprehensive DDI data base that contains over 200,000 clinically relevant DDIs between 1833 Food and Drug Administration (FDA)–approved drugs. The data base was also curated by a team of clinical pharmacists. It is intended to be used as a tool for clinicians to detect and avoid potential unwanted DDIs when prescribing medications. The data base is divided into 6 different subsets by mechanism type, such as metabolism, excretion, antagonism, and more. There is also a seventh category that captures confirmed DDIs with unknown causes. Each DDI is annotated with a severity, which can be either minor, moderate, major, or unknown. We have now used this data base as the foundation of our DDI model, before supplementing it with additional mechanism-specific information.

2. Materials and methods

We have studied the effects of P450 inhibition predictions on the performance of various DDI classification models. This involved the use of 15 datasets: 1 describing DDIs, 6 describing P450 substrates, 6 describing P450 inhibition, and 2 describing the pregnane X receptor (PXR). Supplemental Table 1 describes each dataset used in this study and its source. Furthermore, we trained and evaluated a wide variety of different model configurations on each dataset, as well as the final model combination, which involved all 15 datasets (Fig. 1). Finally, we developed adverse outcome pathways (AOPs) for each DDI prediction, aiding interpretability and boosting the applicability of our DDI model system.

Fig. 1.

Schematic of drug-drug interaction models with supporting machine learning models. The 3 dots are intended to denote that additional data and models were also included in pretraining.

2.1. DDI dataset

A subset containing 56,368 DDIs was downloaded from DDInter.²⁹ This list of interactions involves a set of 1757 drugs, all of which are orally delivered. This subset was chosen to include all the metabolism-specific interactions found in the DDInter data base, meaning that the majority are P450-mediated interactions. The DDInter dataset maps a pair of drugs to an interaction severity, which can be either minor (no treatment change required), moderate (use caution), major (potentially life-threatening), or unknown (an interaction is present, but the severity is poorly described by the source), as defined by the federally recognized drug use data base, DRUGDEX. Additional DDIs are available from DDInter, but these are caused by a wide range of nonmetabolic mechanisms, hence our focus on the metabolism-related (P450-mediated) subset only.

Simplified Molecular Input Line Entry System (SMILES) structures for each drug were sourced from the National Institutes of Health’s Chemical Identifier Resolver data base³² or PubChem.³³ If the structure for a particular drug was unavailable in both resources, all interactions involving that drug were removed from the dataset. This primarily included common foods (eg, garlic). A total of 199 of the 1757 drugs were removed from the dataset during structure identification and standardization, corresponding to 5119 of the 56,368 interactions (9%).

2.2. P450 datasets

In addition to the main DDI dataset, we selected and curated 14 datasets that describe P450 inhibition, P450 substrates, and PXR.^11,34, 35, 36 The selected enzymes (CYP1A2, CYP2C9, CYP2B6, CYP2C19, CYP2D6, and CYP3A4) and associated datasets were chosen for their crucial role in drug metabolism, as P450 families 1–3 metabolize approximately 80% of drugs.^7,37 Six datasets comprise P450 inhibition (IC₅₀) or agonism (EC₅₀) data, and 6 comprise P450 substrate data, with each dataset corresponding to 1 P450 enzyme. Finally, the 2 PXR datasets are significant due to PXR’s role in upregulating CYP3A4.^38,39 Most of the molecules in these datasets do not appear in the DDI dataset, and vice versa. We also explored additional datasets and models for targets that upregulate P450 activity,⁴⁰ such as the constitutive androstane receptor, glucocorticoid receptor, retinoid X receptor, SF-1 protein, and vitamin D receptor. However, these datasets and associated models did not improve DDI prediction performance, so we excluded them from this study. Induction model statistics (5-fold cross-validation [CV]) can be found in Supplemental Table 2. It is important to note that retinoid X receptor, SF-1, and vitamin D receptor are not known to regulate the major P450 enzymes involved in drug metabolism, and the CV recall for constitutive androstane receptor and glucocorticoid receptor was relatively poor, which could be the reason these models did not improve DDI prediction performance.

2.3. Data curation

Both the DDI dataset and supporting datasets described in the P450 Datasets section were curated using our proprietary data curation software, eClean. This step used RDKit version 2023.09.4⁴¹ to canonicalize and neutralize the SMILES structures. Structures were desalted using a list of common salts. We also converted the standard value (IC₅₀ or EC₅₀) columns to negative log-molar units where applicable. Additionally, records with invalid values, units, or qualifiers were removed from the 14 supporting datasets. During this step, measurement values greater than 12 negative log-molar (ie, <1 pM) were removed on the grounds that such values are physically unreasonable. Nonpotent outliers were removed if the measurement values were less than the first quartile (Q) by 1.5 times the interquartile range (IQR; ie, <Q1–1.5 ∗ IQR) on the negative log-molar scale. Finally, structures with multiple measurement values were consolidated by first removing exact duplicates (which typically correspond to the same measurement pulled from multiple sources), then removing extreme outliers (1.5 ∗ IQR above or below Q2) from each group of duplicate structures, and finally averaging the remaining measurements.

To adapt the DDI dataset for use with binary classification models, we used a synthetic negative class generation method, where we assumed nonpresence in the training set implies noninteraction.⁴² Thus, we assigned the entire original DDI dataset (excluding unknown severity interactions) to the positive class. Any combination of drugs that did not explicitly appear in the DDI dataset was assigned to the negative class.

2.4. Baseline model, metrics, and molecular descriptors

To establish a baseline to quantify model performance, we opted to use a randomization technique to “scramble” the ground truth labels. By randomly permuting the labels, but not the structures, we effectively created a random model. This technique is sensitive to class imbalance, which can artificially improve the baseline statistics in similar ways to how a model may be affected. To avoid any potential issues from class imbalance, we balanced the DDI dataset by introducing the appropriate number of synthetic "no interaction" datapoints, as mentioned previously. When evaluating and selecting models, we discarded any model that performed worse than the baseline model. Additionally, we quantified any accuracy improvements by using the baseline model as the zero point (ie, if a model scored twice as far from the baseline as another model, it performed 100% better).

In all cases, we used a nested 5-fold CV with hyperparameter grid searching methods for model evaluation. Hyperparameter grids were set from previous modeling efforts and experimentally determined to be sufficient for the DDI study. When selecting model types and optimizing final models, we focused on maximizing recall, as we deemed false negatives to be significantly more harmful than false positives. Recall was defined as follows:

$$\text{Recall}=\frac{\text{TP}}{\text{TP}+\text{FN}}$$ (1)

where TP is the number of true positive predictions and FN is the number of false negative predictions. Thus, maximizing recall minimizes false negatives. We also captured receiver operating characteristic (ROC) curves and associated areas under the ROC curves (AUCs) to show a comprehensive picture of model performance. ROC curves capture binary model performance by plotting the model’s true positive rate against the false positive rate as the classification threshold varies. AUC values closer to 1 (larger area) correspond to better model performance.

Models were built using either functional connectivity circular fingerprints (FCFP6) or extended connectivity circular fingerprints (ECFP6), in either 1024- or 2048-bit length, as molecular descriptors.⁴³ All fingerprints were generated from standardized SMILES using RDKit. These descriptors capture molecular features in a bit vector based on atom connections for all structure subgraphs of a maximum diameter of 6. In this way, we can train 2-dimensional structure-aware DDI models, which allows us to perform inference on previously unseen drugs. For the 14 supporting models, we used the molecular fingerprints as generated. However, to capture interactions between pairs of drugs, we introduced an additional preprocessing step, which combines pairs of fingerprints by summing the fingerprint vectors bitwise (Supplemental Table 3). In this way, we encoded shared structural similarities between 2 drugs in a pair as input for the model, while still maintaining information about the individual drugs. Additionally, we eliminated bias due to the ordering of the molecular inputs. We also tried combining the fingerprints by appending one onto the end of the other, but we found that the summation technique was acceptable. We performed comparisons to determine the most effective molecular descriptors for the DDI task and found that the 1024-bit ECFP6 produces the best performance, although all fingerprinting methods produced similar results (Supplemental Fig. 1).

2.5. Model generation

All machine learning models were built using our Assay Central software⁴⁴ with Python 3.12, scikit-learn 1.3.0, Pandas 2.1.4, and Numpy 1.26.3 on a 24-core AMD Ryzen Threadripper 3960× running Ubuntu 20.04 with 128 GB RAM. Data curation and preprocessing were performed on a 2023 MacBook Pro (M2 Pro CPU, 32 GB RAM) with Python 3.11.7, Numpy 1.24.3, and Pandas 2.1.4.

Initially, we trained and optimized a binary classification model on the full DDI dataset. We tested 4 fingerprint types (1024-bit FCFP6, 2048-bit FCFP6, 1024-bit ECFP6, and 2048-bit ECFP6) and classification model types (AdaBoost, random forest, naïve Bayes, logistic regression, XGBoost, k-nearest neighbors [KNN], support vector classification [SVC], and deep learning [DL]; grid search parameters are in Supplemental Tables 4–11, class separation data are in Supplemental Figs. 2–16). Based on the 5-fold CV recall score for each combination of model and fingerprint, we selected the core DDI model configuration 1024-bit ECFP6, fingerprint sum, and DL. Additionally, we trained and optimized separate classification models for each of the 6 inhibition datasets and the 2 PXR datasets. Classification models were also trained on the P450 substrate datasets, as these do not contain a continuous prediction value. Each model was trained using a predetermined hyperparameter grid (Supplemental Tables 4–11) that was empirically evaluated to perform well in similar quantitative structure-activity relationship tasks in previous case studies.⁴⁵

To enable the DDI model to leverage the support models’ predictions, we modified the input fingerprints before training the overall DDI + P450 model, as shown in Fig. 1. Specifically, during DDI inference, each pair of drugs was fed into the n supporting models, resulting in a prediction vector of length 2n (step 1). Then, the prediction vector was concatenated with the fingerprint-sum ECFP6 fingerprints (step 2). We tried several different methods of combining fingerprints and prediction vectors, such as double and reverse concatenation (Supplemental Table 12), but we found that the method did not significantly affect the results. Next, we trained a new DDI model on the resulting augmented fingerprints (step 3) and evaluated the output (step 4) via CV performance metrics (Supplemental Fig. 1).

2.6. AOPs and molecular feature importance analysis

To increase the interpretability and usability of our P450-supported DDI model, we developed an AOP to describe the possible predicted P450 interactions that cause DDIs. Specifically, if 2 drugs are predicted to be inhibitors or substrates for the same P450 target, it is suggested that an interaction will occur. This P450 prediction mechanism allows us to verify the accuracy of our DDI model by cross-checking the P450 inhibition and substrate predictions with known P450 metabolism targets for an interacting pair of drugs. This can be performed by tracking the P450 model inference results during DDI model execution and then looking for instances where a P450 model predicts inhibition for both drugs in a pair.

Additionally, we performed a feature importance analysis to identify the molecular features that contribute the most to model prediction. This analysis further explains the overall DDI prediction and AOP by revealing the exact molecular features that the inhibition models draw from.

2.7. Real-world test set

To test our model’s generalizability to real-world uses, we tested it against a set of 30 drugs from the FDA drug list (Supplemental Table 13). All the drugs selected were approved in either 2022 or 2023, and none are present in the DDInter data base. We also selected 30 additional drugs from the DDInter data base. Our real-world test case simulated the use of a novel drug alongside an existing drug, as we hypothesize that this case is more common than using 2 novel drugs concurrently. Thus, for each of the novel drugs, we predicted interactions with each of the existing drugs using our tuned model for a total of 900 predictions. Then, we leveraged the DrugBank⁴⁶ drug interaction data base to form a ground truth set to validate our model predictions. However, confirming the existence of interactions involving novel drugs was made more difficult by the limited clinical data. Thus, we could not assume that absence from the DrugBank data base implied noninteraction (as we did with DDInter to create negative class data for our training set). This limited us to only measuring true positives from our model. Only 10 P450-mediated DDIs out of the 900 predictions in our test set had confirmed DDIs according to DrugBank.

3. Results

3.1. Supporting models: P450 inhibition and PXR (upregulation)

Models trained on the individual P450 inhibition and PXR datasets were optimized and evaluated separately to produce the final supporting model ensemble used to enhance the efficiency of the overall DDI model. We evaluated several classification model types (AdaBoost, DL, KNN, random forest, SVC, and XGBoost) as well as various molecular fingerprinting algorithms (FCFP6 and ECFP6 as 1024-bit and 2048-bit versions). This parameter selection process is illustrated in Supplemental Fig. 1. The training results for the best fingerprint type, 1024-bit ECFP6, are shown in Table 1. For each P450 target, we selected the best-performing model for use in our final DDI + P450 model. From Table 1, we can see that the P450 inhibition models had good AUC scores around 0.8, with only the CYP2B6 and CYP3A4 models dropping slightly below. CYP2B6 had a significantly smaller dataset (n = 228), and CYP3A4 is known to be a large and complex target.

Table 1.

Five-fold cross validation results from training the P450 inhibition supporting models

Dataset/Model	Accuracy	F1	AUC	Cohen κ	MCC	Precision	Recall	Tn/Fp/Fn/Tp
CYP1A2/DL (n = 2449)	0.867	0.638	0.885	0.561	0.579	0.800	0.532	1835/71/254/289
CYP2C9/SVC (n = 7625)	0.767	0.553	0.804	0.402	0.413	0.867	0.647	4768/1192/588/1077
CYP2B6/SVC (n = 228)	0.777	0.558	0.782	0.412	0.425	0.602	0.559	145/26/25/32
CYP2C19/SVC (n = 2208)	0.903	0.946	0.824	0.435	0.438	0.954	0.939	99/91/124/1894
CYP2D6/SVC (n = 8198)	0.842	0.671	0.867	0.568	0.569	0.644	0.701	5583/731/563/1321
CYP3A4/SVC (n = 4127)	0.725	0.548	0.785	0.363	0.381	0.465	0.685	2307/821/315/684

Fn, false negative; Fp, false positive; MCC, Matthews correlation coefficient; Tn, true negative; Tp, true positive.

Like the P450 inhibition models, the PXR models had good AUCs around 0.8 (Table 2). We found that the SVC models performed the best on both PXR datasets in terms of AUC. However, the Matthews correlation coefficient for both models was quite low, especially for the model built on the smaller dataset (n = 237).

Table 2.

Five-fold cross validation results from PXR model training

Dataset/Model	Accuracy	F1	AUC	Cohen κ	MCC	Precision	Recall	Tn/Fp/Fn/Tp
PXR-2/SVC (n = 6447)	0.784	0.588	0.817	0.442	0.444	0.552	0.629	4055/808/587/997
PXR-4/SVC (n = 237)	0.755	0.842	0.790	0.294	0.322	0.789	0.906	23/42/16/156

Fn, false negative; Fp, false positive; MCC, Matthews correlation coefficient; Tn, true negative; Tp, true positive.

3.2. Supporting models: substrate

The P450 substrate models were trained in a similar manner to the inhibition datasets. For each P450 target, the model with the best recall was selected for use in the DDI + P450 model. These models had slightly lower AUCs but were still greater than 0.7 in nearly all cases (Table 3).

Table 3.

Five-fold cross validation results from the P450 substrate classification model training

Dataset/Model	Accuracy	F1	AUC	Cohen κ	MCC	Precision	Recall	Tn/Fp/Fn/Tp
CYP1A2/KNN (n = 639)	0.695	0.637	0.751	0.374	0.375	0.642	0.635	272/96/99/172
CYP2C9/SVC (n = 639)	0.720	0.543	0.723	0.349	0.359	0.644	0.476	353/61/118/107
CYP2B6/SVC (n = 639)	0.725	0.469	0.722	0.285	0.288	0.428	0.520	385/104/72/78
CYP2C19/SVC (n = 639)	0.728	0.591	0.762	0.388	0.390	0.609	0.578	339/82/92/126
CYP2D6/SVC (n = 639)	0.714	0.646	0.781	0.407	0.410	0.681	0.619	289/80/103/167
CYP3A4/RF (n = 639)	0.750	0.828	0.779	0.367	0.368	0.845	0.813	93/71/89/386

Fn, false negative; Fp, false positive; MCC, Matthews correlation coefficient; RF, random forest; Tn, true negative; Tp, true positive.

3.3. DDI model

Using the DDI dataset in isolation, we built and evaluated simple classification models to predict interaction presence. After finding the best-performing model configuration (DL classification model and 1024-bit ECFP6 fingerprints), a comparison was performed between these models and the baseline model (as described in the Materials and Methods section above). Specifically, we found that these models performed better (21%) than the baseline in terms of recall (Table 4). Furthermore, the models exhibited an increase (44%) in accuracy compared with the baseline.

Table 4.

Five-fold CV training results for the DDI model with and without supporting P450 predictions. The P450 models’ CV statistics are shown in Table 3. Drug pairs absent from the training set were assumed to be noninteracting. Baseline model results are included for illustration purposes

Model	Accuracy	F1	AUC	Recall	Tn/Fp/Fn/Tp
DDI/DL (n = 72,030)	0.843	0.835	0.916	0.816	30,928/4604/6724/29,774
DDI + support/DL (n = 72,030)	0.855	0.846	0.932	0.821	31624/3908/6516/29,982
Baseline/DL (n = 72,030)	0.499	0.515	0.500	0.524	16,830/18,702/17,364/19,134

Fn, false negative; Fp, false positive; Tn, true negative; Tp, true positive.

Table 4 also shows the CV metrics for the DDI + P450 model (which uses P450 inhibition and substrate predictions as well as molecular fingerprints as input, as described in the Materials and Methods). While the improvements over the original DDI model are small, they show a distinct increase in performance across all metrics. Specifically, we see a small (2%) increase in recall over the DDI model with respect to the baseline, and an almost 4% increase in accuracy. While small, these improvements suggest that predictions of the supporting model ensemble slightly enhance the CV performance of these DDI models.

3.4. AOPs and molecular feature importance analysis

Additionally, we developed AOPs for the P450-supported DDI model to quantify the decisions of the model based on enzyme inhibition and induction. AOPs aim to describe the P450 interactions behind the model’s predictions.⁴⁷ We developed a graphical representation of the predicted P450-drug interactions to quickly identify the presence and cause of a potential interaction (Figs. 2 and 3). Each colored square in the 6 subplots at the top of the figure represents the model’s prediction for a single P450 enzyme. The type of model (inhibition or substrate) and the drugs predicted are indicated next to the squares. Black represents negative predictions, gray represents inconclusive predictions, and green represents positive predictions. The top row of squares indicates whether each set of P450 predictions suggests a likely DDI. Finally, the right-most square indicates the overall interaction prediction of the 2 drugs based on the overall DDI model. For example, for CYP1A2 in Fig. 2, drug 1 (fluconazole) is predicted to be a substrate while drug 2 (pioglitazone) is not, and both inhibition predictions are uncertain. This leads to an inconclusive DDI prediction for CYP1A2, as indicated by the gray square at the top of the subplot. However, the 2 drugs are predicted to interact via CYP2C9, CYP2C19, CYP2B6, and CYP2D6, resulting in an overall positive DDI prediction, as shown by the right-most green indicator.

Fig. 2.

Example AOP graphic for fluconazole (drug1) and pioglitazone (drug2). This drug pair has been shown to interact according to DrugBank. pred. int., predicted interaction.

Fig. 3.

Example AOP graphic for rabeprazole (drug1) and budesonide (drug2). This drug pair does not have any listed P450-mediated interactions in DrugBank, although they interact via a different mechanism. pred. int., predicted interaction.

3.5. Feature importance analysis

Understanding model reasoning is crucial to providing trustworthy and verifiable results. To achieve this goal, we demonstrate the process of determining important molecular features by performing a feature importance analysis on fluconazole, which is 1 of the 2 interacting drugs from the previous section. Specifically, to support our AOPs, we targeted the P450 models, as they are key to understanding the predictions of the overall DDI model.

For SVC models using a linear kernel (CYP2C19), we can simply inspect the coefficients of the trained P450 inhibition model. Larger coefficients correspond to more important molecular features, which we can then map to the molecular structure of the drug (Table 5). On the other hand, the DL model (CYP1A2) and the nonlinear SVC models (CYP2C9, CYP2B6, CYP2D6, and CYP3A4) require the usage of permutation importance to determine the most important molecular features.⁴⁸ Permutation importance determines the importance of input features by randomly permuting each one and measuring the resulting quality of the model. Interestingly, Table 5 shows that many of the same features are used by each model, suggesting some similarities between the models. This analysis can also be used to validate DDI predictions in a clinical setting by validating the results of the underlying P450 models, further improving model interpretability.

Table 5.

Feature importance analysis of the P450 inhibition models. The top 3 features in order of importance for each model are highlighted on the example drug fluconazole

Model	First	Second	Third
CYP1A2 inhibition
CYP2C9 inhibition
CYP2B6 inhibition
CYP2C19 inhibition
CYP2D6 inhibition
CYP3A4 inhibition

3.6. FDA novel drug test set

The results of evaluating our model on a test set of 30 novel FDA-approved drugs and a further 30 existing drugs present in the DDInter data base were mixed. Of the 10 known interactions assembled from the test set, our best model, DL classification, detected 2 with a recall score of 0.2. While this may be significantly lower than expected based on the 5-fold CV scores during the training process, it is important to consider a similarity analysis of the novel drugs. Patterson et al⁴⁹ found that, in general, 2 compounds have similar activity if they have a Tanimoto similarity of >0.85. Thus, if a given previously unseen drug has a Tanimoto similarity of >0.85 to drugs in the model training set, it is likely that the drug is within the applicability domain (AD) of the model, and an accurate prediction can be made. We, therefore, analyzed the novel drugs’ Tanimoto similarity using a maximum similarity method. For each novel drug, we found the most similar drug in the DDInter data base to produce the maximum similarity score. This metric was also calculated for each pair of DDInter drugs to calculate a baseline for comparing similarity scores. We found the average similarity between drugs in the DDInter dataset to be 0.809, whereas the average similarity between drugs in the FDA test set and the DDInter dataset was 0.761. This implies that the model may perform slightly worse when predicting interactions involving drugs from the FDA test set. However, both datasets’ average Tanimoto similarity was below the defined threshold of 0.85.

4. Discussion

A wide range of in silico DDI modeling approaches has been explored in the literature. Ferdousi et al⁵⁰ described a similarity-based DDI model using the Russell-Rao similarity method, correctly identifying 72% of known DDIs in their dataset. Furthermore, the authors developed a specialized feature vector system for drug representation based on biological elements of each drug. Similarly, Li et al⁵¹ used a Bayesian network-based approach to model interactions by mapping each drug pair to a set of 6 features, such as chemical similarity or side effect similarity, to predict interacting drug combinations. This achieved an AUC of 0.95 in a dataset of DDIs, and an AUC of 0.71 in a generalization dataset of previously unseen DDIs. Such similarity-based approaches ignore drug molecular structure altogether,⁵² leading to the development of graph-based DDI prediction models. Typically, the drugs and other entities (ie, receptors) are represented as nodes, and the edges are interactions or other relationships. Thus, various graph algorithms, such as random walk and restart, can be applied to derive DDIs from the graph.⁵³ Yan et al⁵⁴ developed a node-based drug network to predict DDIs involving fully unknown drugs by calculating initial interaction information based on the network. The authors then used a recursive least squares algorithm to calculate the interactions. Cheng et al⁵⁵ evaluated 5 different machine learning models (naïve Bayes, decision tree, KNN, logistic regression, and SVC) in DDI prediction tasks. Using data gathered from multiple sources such as DrugBank⁵⁶ and MetaADEDB,⁵⁷ the authors then used drug descriptors based on several molecular similarity measures, such as structural similarity and genomic similarity. Ultimately, they found that SVC was the most effective model, producing a 5-fold CV AUC of 0.67. Zhang et al⁵⁸ took a slightly different approach to similarity metrics by applying network-based neighbor similarity. Based on calculated similarity, the authors used a neighbor recommender method commonly used in movie recommender systems to calculate user ratings for unseen movies. Specifically, if 2 drugs are considered similar, it is likely that they have similar interactions when exposed to the same drugs. Thus, unknown interactions for a given drug can be calculated by averaging the interaction severities on the same set of drugs. This method resulted in a maximum 5-fold CV recall score of 0.76 when trained on drug substructure data.⁵⁹

DL has also been applied to modeling DDIs. Lee et al⁶⁰ proposed a feed-forward network model with autoencoders to predict interactions based on several similarity metrics. They found that introducing molecular information other than structural similarity profiles produced up to a 2% increase in accuracy and over a 4% increase in recall. Hou et al⁶¹ trained deep neural networks with features generated from drug SMILES. The authors encoded the training data using one-hot encoding, where each bit represents 1 of 80 DDI types across 4 major mechanisms. The deep neural networks performed better on a holdout test set than the SVC model, with AUCs of 0.903 and 0.942, respectively. In summary, there is a wide range of different machine learning algorithms, such as network models, that leverage many types of data, such as molecular structure and even gene ontology, which have been used on public data to predict DDIs. Since each research group trains and evaluates its respective models on different data bases using various techniques, it is impossible to quantitatively compare approaches and benchmark them. Many other studies have used drug interaction and structure data from sources such as DrugBank,^50,56 which is a comprehensive drug–drug target information data base, and SIDER,^51,54,62 a data base of adverse drug side effects. Additionally, the OFFSIDES data base has been used,^54,63 which represents a collection of ∼0.5 million off-label drug side effects, and TWOSIDES^54,63 is another DDI data base.

Previous DDI modeling efforts have typically focused on applying different machine learning algorithms, such as Bayesian networks.⁵¹ Many models use molecular similarity, such as Russell-Rao similarity, to make DDI predictions.⁵⁰ On the other hand, our work focuses on leveraging knowledge of drug-P450 interactions to enhance the accuracy of a DDI model. Instead of predicting DDIs using similarity between drugs, our models learn how drug structure and P450 inhibition impact the potential for DDIs.64, 65, 66, 67, 68, 69 Additionally, the unique structure of our models provides some degree of interpretability through P450 interaction modeling, which is likely vital in a clinical setting.

As shown in our studies, using P450 machine learning models to help predict DDIs marginally improves the model across several performance metrics. This may be due to a few different factors. First, others have shown70, 71, 72 that DDI typically happens between an inhibitor and a substrate for an enzyme. Predicting this type of interaction could be a weak point in our model configuration, as our substrate datasets limited us to training classification models instead of regression models. This prevented us from capturing weak versus strong substrate interaction behavior. Second, P450 induction can also cause DDIs, as the serum concentration can be decreased to the point where the drug(s) no longer have the desired effect.^28,72 However, DDIs caused by induction are not common in practice.⁷³ This means that our PXR models may have little contribution to the model performance and could be introducing noise when generalizing to unseen test sets, such as the FDA novel drugs set. Third, it is possible that the datasets used in this study cover dissimilar molecular spaces, meaning that the AD of the resulting model is highly constrained, leading to poor performance in generalization tasks.

Through the limited FDA novel drug interaction test set, we showed that DDI model generalization to real-world circumstances is not straightforward. As with any model, predictions involving input data from outside the model’s AD can result in inaccurate and unusable results. Only 2 out of 10 predictions in the novel drugs dataset were correct, despite the high recall (0.8) computed with CV. The Tanimoto similarity between the novel drugs and the DDInter drugs was also low, and the model was unable to make accurate predictions.

Conclusively showing that 2 drugs have no interaction is typically not the focus of drug toxicity studies,⁶⁴ partially due to the risk involved in making the claim and the complexity, as other types of interactions could also occur.⁷⁴ Instead, for most clinical applications, showing the presence of a DDI is generally reported, whereas the absence of a DDI is not, so there is a publication bias that exists. Molecules that have undergone a DDI risk assessment using in vitro data and clinical exposures (ICHM12)⁷⁵ can be used to exclude clinical investigation of DDIs and, as such, could be additionally added to the no interaction dataset. Thus, training classification models can be challenging due to the lack of an established negative dataset. In our study, we assumed that absence from the training set implied noninteraction, but this obviously is not always the case.²⁹ Alternatively, it is possible that one-class classification models can be used to achieve the desired results without needing to generate or collect data for the negative class.⁷⁶

By only modeling P450 enzymes, we assume that all interactions in DDInter’s metabolism-mediated DDI dataset are caused by P450 enzymes. However, P450 enzymes are only one of many types of human drug-metabolizing enzymes that could be involved in DDIs.^14,77 Our DDI model could likely detect non–P450-mediated DDIs because it learned relevant structural attributes, but such detection would not be augmented by P450 predictions. Furthermore, we only modeled the P450 enzymes with enough available data and those that are known to contribute significantly to metabolism. These P450 predictions would not be expected to augment DDI predictions when mediated by unrepresented P450 enzymes.

In future work, we could address these various shortcomings of our model. For example, we could include dose level and subsequent exposure, which might impact the accuracy of predictions. This could be via applying other machine learning methods, such as Gaussian mixture models,⁷⁸ large language models,⁷⁹ or positive unlabeled learning techniques, wherein the classifier is given data that belong to either “positive” or “unlabeled” classes and tasked with classifying them as “positive” or “negative” instances.⁸⁰ Such approaches would allow us to determine if the accuracy of the DDI model can be improved by removing potentially incorrect training data due to the negative class generation method. While it may be possible that the number of false negatives added to the dataset is low, it is well established that the incorporation of better training data nearly always results in better accuracy. We could also leverage transporter models^45,81, 82, 83 by incorporating them along with P450 models to predict DDIs. Additionally, we could validate the model in clinically relevant scenarios, such as a larger collection of recorded DDIs from molecules that were not included in our DDI prediction models and that were likely outside the AD. Finally, the AOP system presented in this paper could be expanded to include information about prediction uncertainty to paint a clearer picture of the risks involved in the drug combination. While uncertainty thresholding could be included in the models, it may be useful to expose the uncertainty figures to human operators who may be better suited to make the final risk assessment based on the patient.

In summary, P450 enzymes are key in predicting DDIs, as inhibition and induction of the enzymes can vary the drug concentration in the human body.⁸⁴ We have demonstrated that the incorporation of this existing knowledge can slightly improve the accuracy of DDI models using a machine learning approach that includes input for P450 inhibition, induction, and substrate predictions. We have also evaluated model performance using recently approved FDA drugs, which demonstrates the challenges with novel molecules outside the model’s AD. Future work still needs to demonstrate the real-world prospective prediction applications and potential of such P450-enhanced DDI prediction models. This may require further increasing the accuracy of our P450 models, adding additional datasets of other enzymes and transporters, and adopting alternative machine learning techniques to expand beyond the approaches described herein. DDI predictions could also be extended to when more than 2 drugs are used simultaneously. This might be feasible by taking a pairwise approach and combining the results, but interpreting the data could be complex. This indicates there are plentiful opportunities for further research on this topic.

Conflict of interest

Sean Ekins is the owner, and Jason T. Wong, Joshua S. Harris, Thomas R. Lane, and Fabio Urbina are employees of Collaborations Pharmaceuticals, Inc.