Robust prediction of drug combination side effects in realistic settings

Abstract

Side effects caused by drug combinations pose a major challenge in healthcare. Knowledge of these side effects is limited because often they are not detected in clinical trials, which typically involve a restricted number of participants and tested drug combinations. We introduce DCSE (Drug Combinations Side Effects), a novel machine learning method for predicting polypharmacy side effects. DCSE learns latent signatures for drugs, drug pairs, and side effects to predict the probability that a side effect occurs in a given drug combination. We first evaluate its performance in the commonly adopted experimental settings in the literature. However, these rely on balanced testing datasets and sampled negative examples, which do not capture the highly imbalanced and structured set of unknown side effects encountered in practice. Therefore, a key contribution of this paper is the introduction of more realistic experimental settings under prospective evaluations. Here, we attempt to predict side effects reported between 2009 and 2014 after training only on data available prior to that period. These evaluations include warm-start scenarios, in which some side effects are already known for a drug pair, and cold-start scenarios, in which the model predicts side effects for previously uncharacterized drug pairs. Our results indicate that DCSE consistently outperforms state-of-the-art methods, demonstrating its robustness and efficacy in real-world applications.

Author summary

Understanding the risks of combining medications is a central problem in healthcare. Clinical trials rarely capture the full range of side effects, and many side effects are identified only after drugs reach the market, leaving many possible drug combinations insufficiently tested. Computational approaches can help fill this gap by predicting which combinations are likely to cause specific side effects. Our approach works by learning signatures for both drugs and side effects, which can be thought of as representations that describe how drugs act and what is required to trigger a side effect. A key challenge is how to combine the signatures of two drugs into one representation for the pair. We show that simple linear models fail to capture the complexity of drug interactions, so we use a deep neural network to combine them in a non-linear way. Our tests, which use reports collected at different points in time to mirror how information about drug pairs becomes available, show that our approach is useful for monitoring risks after drugs have been approved and for exploring new uses of pairs of existing drugs.

Introduction

Many human diseases arise from highly complex biological processes that resist the influence of a single drug. Polypharmacy, the concurrent use of multiple medications, is a common strategy to overcome such limitation [1,2]. However, a major drawback of polypharmacy therapies is the increased risk of side effects triggered by drug combinations [3,4] that, in the U.S. alone, are affecting over 15% of the population and costing over 170 billion annually for treatment [5]. Healthcare providers often lack awareness of the clinical risks associated with most drug combinations. Incomplete knowledge about these side effects stems from the limited sample size of clinical trial participants, that restricts the ability to detect less common side effects [2], along with the infeasibility of experimentally testing all possible drug combinations during clinical trials.

In the past decade, machine learning has become increasingly prominent across drug-related prediction tasks, from drug-target interaction prediction to broader biomedical data analysis [6–8]. In the context of polypharmacy, public resources describing interactions between pairs of drugs, also known as Drug-Drug interactions (DDIs), have made it possible to train computational models for predicting side effects caused by DDIs [9,10]. Early methods were not specific to individual side effects, instead focusing primarily on predicting the probability that two drugs would interact, causing any side effect [11–14]. More recently, machine learning methods have shown promising results in identifying the specific side effects caused by drug pairs. Among these are graph-based methods, such as Decagon, the pioneering work by Zitnik et al. [5,15,16], that models DDI data as a multi-relational knowledge graph, where each side effect is treated as a distinct edge type. A graph convolutional network is then trained to perform multi-relational link prediction between drugs. Traditional graph-based methods, such as DeepWalk [17], and tree-based models like XGBoost [18], have also been widely adopted as baselines for evaluating polypharmacy side effect prediction methods. A different approach, DeepDDI, presented by Ryu et al. [19], represents each drug with a structural similarity profile and concatenates them to form a drug pair input to a deep feedforward neural network that predicts specific side effects. Nyamabo et al. [20,21] introduced GMPNN, a message-passing neural network that learns representations of molecular substructures from drug graphs and combines them to predict side effects for drug pairs. They further demonstrate the effectiveness of GMPNN in predicting interactions involving drugs with limited or no prior information (cold-start scenarios). Wang et al. [22] introduced KnowDDI, a graph neural network method that combines a DDI graph with an external biomedical knowledge graph and learns drug-pair-specific knowledge subgraphs to predict and interpret DDI types. Zhu et al. [23] proposed SSF-DDI, which extracts sequence features with a convolutional network and substructure features from molecular graphs before fusing them to predict drug interactions. Zhou and Uddin [24] later proposed FG-DDI, which incorporates functional-group information into molecular message passing to make the learned drug representations more chemically informed. Other recent methods have broadened the strategies used for DDI prediction through multi-view contrastive learning, large language model representations combined with biomedical knowledge graphs, spatial-aware capsule aggregation, and relation-aware graph transformers [25–28]. Despite these advances, the evaluation of DDI prediction methods often remains disconnected from the conditions under which new associations are identified in practice. Standard benchmarks commonly rely on randomized splits and artificially balanced test sets, which can obscure the strong imbalance and temporal changes present in real-world DDI data. Recent work has similarly emphasized that DDI prediction benchmarks can give misleading estimates of performance when distribution changes between training and testing are ignored [29].

In this paper, we present DCSE (Drug Combinations Side Effects, pronounced “dixie”), a method for predicting the probability of side effects caused by DDIs. Our approach learns signatures (latent feature vectors) for drugs and side effects. The two drug signatures are then combined nonlinearly using a deep neural network to obtain a signature for the drug pair. DCSE estimates the probability of a side effect by comparing the drug-pair signature with the side-effect signature through a dot product, adjusting this score with learned bias terms for the drug pair and side effect, and passing the result through a sigmoid function. The resulting value can be interpreted as the probability of the side effect being associated with the drug pair. We first show the performance of DCSE in the standard experimental settings adopted in the literature. We then conduct experiments in a more realistic setting, aiming to predict the probability of associations that were unknown at the time the training set was created. In our prospective experiments, we distinguish two scenarios: warm-start, where we predict side effects for drug pairs for which some side effects are already known; and cold-start, where we predict side effects for drug pairs for which no side effects are known. Our results demonstrate that DCSE consistently delivers robust performance across both standard and more realistic scenarios, outperforming state-of-the-art methods.

Results

The DCSE model

We recently showed [30] that drugs and side effects can be represented as vectors (signatures) in a low-dimensional space, such that the frequency class (as defined by the World Health Organization (WHO) – Council for International Organizations of Medical Sciences (CIOMS) [31]) of the side effect for a single drug can be estimated using the dot product of their corresponding vectors. Fig 1A shows a representation of this idea for the drug “metformin” and the side effect “diarrhea”. The signatures can be interpreted as representing the states of components in a biological system: the elements of a drug signature reflect the intensity of the drug’s action on these components, while the elements of a side effect signature indicate which components need to be affected to trigger a specific side effect.

Fig 1. The approaches by Galeano et al. (2020) and DCSE.

(a) Galeano et al. [30] showed that the frequency of diarrhoea in patients on metformin can be obtained by the dot product of the signature for metformin and the signature for diarrhoea. (b) In DCSE, the learned signatures for zolmitriptan and adapalene are combined using an MLP to obtain the combination signature. The probability of an irregular menstrual cycle being caused by this pair is obtained by the logistic function of the product of the drug combination signature and the side effect signature.

The idea of representing the state of a system using signatures could potentially be applied to the problem of predicting the probability of side effects caused by drug pairs, provided we could learn signatures representing the intensity of the action of a drug pair on the system components. Can we combine the signatures of two drugs to derive a signature for the pair? Initially, we explored learning a linear combination of the signatures of individual drugs. As expected, we saw that a linear model cannot effectively capture the complexity of combining drug signatures (see S1 Text). We consider this empirical proof supporting the well-established understanding that the effects of drug combinations are highly non-linear [32].

Consequently, we opted to use a deep neural network to nonlinearly combine the signatures of the drugs. Fig 1B illustrates this approach, that we call DCSE: the learned drug signatures for zolmitriptan and adapalene are combined through a Multi-Layer Perceptron (MLP) to generate the signature for the pair. The probability of the side effect “irregular menstrual cycle” being caused by the zolmitriptan-adapalene pair is then calculated as the logistic function of the dot product between their corresponding signatures.

An overview of the complete DCSE architecture is shown in Fig 2, and a formal description of the system can be found in Methods. The input comprises three one-hot encoding vectors, representing the identity of the two drugs, drug i and drug j, and the side effect l. Above the input layers are the embedding layers, fully connected layers that linearly project the one-hot encodings to dense vectors, which are the drug (${\mathbf{w}}_{\mathbf{i}}$ and ${\mathbf{w}}_{\mathbf{j}}$) and side effect (${\mathbf{h}}_{\mathbf{l}}$) signatures. To ensure consistent representations, the weights in the layer W are shared between the two drug inputs, so that a drug has the same signature, regardless of its position in the pair. The signatures for the two drugs are concatenated and passed through a fully-connected MLP architecture, which we refer to as the combination layers. The output of the combination layers, ${\mathbf{\text{c}}}_{i,j}$, is the signature of the combination of drugs i and j.

Fig 2. Architecture of DCSE.

The model learns signatures for drugs and side effects, $w_i$, $w_j$, and $h_l$, respectively, using an embedding layer. The drug signatures are then combined through an MLP to obtain the combination signature, $c_{i,j}$. The final prediction ${\hat{y}}_{i,j,l}$ is obtained by the dot product between the signatures of the combination and the side effect and then subtracted by the product of the drug combination and side effect biases, $p_{i,j}$ and $v_l$, respectively.

When learning from association data, it is essential to account for inherent biases, such as prescription bias — where some drugs are prescribed more frequently — and the higher prevalence of certain side effects in the population [32,33]. DCSE addresses this by explicitly modeling biases for individual drugs, combinations, and side effects. Our approach learns a vector of drug biases, where each element is the bias of a drug, and a vector of side effect biases, where each element is the bias of a side effect. Biases for drug pairs ($p_{i,j}$) are learned by nonlinearly combining the biases of the individual drugs using a small MLP (Fig B in S1 Text). Our experiments demonstrate that these biases capture systematic tendencies in the data, such as the “popularity” of specific drug pairs and side effects (see S1 Text).

The predicted probability of side effect l for the combination of drugs i and j is calculated by first computing the dot product between the signature of the drug pair and the signature of the side effect. To this value, a bias term is then subtracted (see Methods), that is calculated by multiplying the bias for the drug pair $p_{i,j}$ and the bias of the side effect $v_l$. This is then transformed using a sigmoid function, which maps it to the probability of the side effect being associated with the drug pair.

All the weights in this model are learned using the Twosides [9] database of polypharmacy side effects, which provides statistically significant associations between side effects and drug pairs (see Methods). All parameters are learned simultaneously, including the single drug and side effect signatures, the weights of the deep neural network, and the biases.

Evaluation

We first evaluate DCSE using widely adopted experimental settings in the DDI prediction literature. Next, we present a discussion on the limitations inherent in these settings and propose more realistic evaluation procedures. Our novel evaluations are carried out prospectively: we attempt to predict side effects reported between 2009 and 2014 after training only on data available prior to this period. Within this framework, we also distinguish between scenarios where drug pairs in testing are encountered during training (referred to as warm-start) and scenarios where testing involves drug pairs that are entirely new (referred to as cold-start).

Conventional experimental settings.

We selected two evaluation procedures commonly adopted in the literature: the first proposed by Zitnik et al. [5] and the second by Nyamabo et al. [20]. The Zitnik et al. procedure begins by preprocessing the Twosides dataset to ensure a minimum amount of information for each side effect. Specifically, only side effects occurring in at least 500 drug combinations are retained, resulting in 645 drugs, 964 side effects, and 4,577,977 positive associations between side effects and drug pairs. For each side effect, associations are randomly divided into training (80%), validation (10%), and testing (10%) sets. Negative samples are selected for training by substituting one of the drugs in a positive association. In the training set, the replacement drug is selected according to a sampling distribution defined empirically by Mikolov et al. [34]. In the testing set, the drugs are sampled randomly from a uniform distribution. Both training and testing phases are conducted under a balanced setting, i.e., with a ratio of positives to negatives of 1:1.

The Nyamabo et al. setup is based on the procedure by Zitnik et al., but with a few variations: the percentages for the splits are adjusted to 60% for training, 20% for validation, and 20% for testing; negative samples are selected using a different sampling distribution, defined by Wang et al. [35], for both training and testing; and the entire process is repeated three times, resulting in three stratified randomized folds. Again, both training and testing phases are conducted under a balanced setting.

We compared the performance of DCSE against DeepWalk [17], XGBoost [18], DeepDDI [19], Decagon [5], and GMPNN [20] (these algorithms are described in the Methods section). These baseline methods and state-of-the-art competitors were selected because they (1) are the representative methods of the most commonly adopted experimental setups in the literature; (2) encompass a wide range of machine learning methods; (3) integrate various sources of information as input features for the drugs; and/or (4) work for both warm and cold-start scenarios. DeepWalk and XGBoost do not support multi-label classification and required training one model independently for each side effect, as already implemented by Zitnik et al. [5]. In the case of DeepDDI, since in this setting different drug pairs have different numbers of side effects, we also trained one model independently for each side effect.

The results of these evaluations are presented in Table 1. We can observe that DCSE’s performance is consistent across the two evaluation procedures. Our results show that the signatures learned by DCSE are descriptive and effective, outperforming methods with more complex architectures, such as Decagon, which relies on graph convolutional networks, and GMPNN, which uses gated message passing neural networks. It is also important to note that the superior performance of end-to-end methods (DCSE, Decagon, and GMPNN) compared to the feature-based approaches (DeepWalk, XGBoost, and DeepdDI). This suggests that for the complex task of predicting side effects for drug combinations, methods greatly benefit from learning directly from the association data. The reduced performance of DeepWalk, XGBoost, and DeepDDI could also be due to the fact that training one model per side effect prevents information from being shared across side effects. As a result, these methods cannot fully capture the interdependence between drugs and side effects.

Table 1. Comparative benchmark in conventional experimental evaluation. For the Nyamabo et al. setting, the mean and standard deviation across the three folds are shown. The best performance is shown in bold. † Results taken from Zitnik et al. [5].

Method	Zitnik et al. Evaluation			Nyamabo et al. Evaluation
	AUROC	AUPRC	Method	AUROC	AUPRC
DeepWalk	71.51	70.9	DeepWalk	70.13 $\pm$ 0.1	69.02 $\pm$ 0.11
XGBoost	80.12	81.15	XGBoost	78.42 $\pm$ 0.13	78.84 $\pm$ 0.14
DeepDDI	62.37	61.04	DeepDDI	60.44 $\pm$ 0.07	59.81 $\pm$ 0.06
Decagon†	87.23	83.2	GMPNN	90.07 $\pm$ 0.12	87.24 $\pm$ 0.12
DCSE	91.71	87.24	DCSE	92.44 $\pm$ 0.03	89.67 $\pm$ 0.02

Our proposed prospective experimental settings.

A concern with evaluating system performance in the above experimental settings is the reliance on a balanced test set, which does not accurately reflect the true class distribution. For instance, in the Twosides dataset, among the 83,657,414 possible associations between 63,473 drug pairs with at least one side effect association and 1,318 side effects, only 4,651,131 are known associations (positives), while the remaining 79,006,283 are unknown (negatives). Therefore, negatives account for 94.44% of all possible associations, a distribution that is far from balanced. We also notice that not all negative associations are equally difficult to predict. One can argue that drug pairs for which no side effects are currently known are more likely to be acting on completely different biological systems and are therefore very unlikely to cause any side effects. Thus, a negative association between drug pairs with no known associations should be generally easier to predict. For the 645 drugs in Twosides, taking all possible combinations two at a time results in 207,690 possible drug pairs. Of these, 144,217 have no known side effects at all, and this amounts to a total of 190,078,006 negative associations that are easier to predict. In contrast, for the 79,006,283 associations from the 63,473 drug pairs that have some known associations, predicting the absence of a specific side effect is generally more difficult. In conventional experimental settings, the procedure for sampling negative examples does not control for this distinction, making it highly likely that easier pairs dominate the negative samples.

To address these issues, we introduce a different experimental setting for evaluating performance on this problem. First, we do not balance the testing datasets. Second, we focus only on the more challenging negative associations by taking the 79,006,283 unknown observations between the drug pairs and side effects in Twosides.

We perform our experiments in a prospective setting, where the goal is to predict new side effects reported in an updated version (https://nsides.io/) of Twosides, which includes reports from 2009 to 2014, while training on the original version with reports up to 2009. We refer to these datasets as Twosides-2014 and Twosides-2009, respectively. These tests provide a more realistic assessment of the method’s performance in real-life scenarios, as they account for the chronological order in which side effects are discovered. This aspect is important because it has been noted that associations between side effects are often related [36]. An inherent limitation of evaluating DDI prediction with pharmacovigilance data is that unknown associations are used as negative labels, although some of them may correspond to side effects that have not yet been reported. The absence of a report does not guarantee that the association cannot occur. By respecting the chronological order of the data, our prospective evaluation partly addresses this issue by counting associations absent from Twosides-2009 as positives when they appear in Twosides-2014. Associations that remain unreported in Twosides-2014 may still include true side effects, so some high-scoring predictions assigned to the negative class during evaluation could represent plausible associations awaiting future evidence.

We also distinguish between two distinct scenarios: one in which some side effects are already available for the drug combinations (warm-start), and another in which no prior data exists (cold-start). These two scenarios reflect different real-world situations. Warm-start experiments are relevant for post-marketing pharmacovigilance, which involves the identification, evaluation, and mitigation of drug-related issues after a drug or drug combination has been released to the market [37]. Cold-start experiments are relevant for drug repurposing, where identifying potential side effects can help accelerate the approval process for new drug combinations [1,38].

Finally, we assess DCSE’s performance by benchmarking it against DeepWalk, XGBoost, DeepDDI, and GMPNN. We also compare against more recent state-of-the-art methods, KnowDDI, SSF-DDI, and FG-DDI. For DeepWalk and XGBoost, we again follow the procedure described by Zitnik et al. [5], and train one model per side effect. For DeepDDI, in this setting, we can frame the problem as a multi-label classification problem and follow the original procedure described by Ryu et al. [19]. Decagon was excluded from this evaluation due to technical constraints in the source code (see Methods). GMPNN, KnowDDI, SSF-DDI, and FG-DDI were adapted to train with the full Twosides-2009 dataset, since their released implementations rely on filtered versions of this dataset.

Warm-start evaluation.

Our warm-start evaluation is shown in Fig 3. We train a model using Twosides-2009 (the middle matrix in the figure) and use Twosides-2014 (the matrix on the left) as the test set. Twosides-2014 contains 3,642,690 new associations not present in the training data. All entries labeled as negatives in Twosides-2009 are used in testing, and evaluation is performed by ranking these entries and identifying which of them were confirmed as positives in Twosides-2014 (green cells). The gray cells correspond to the positive associations during training and, hence, are not considered during testing. This evaluation protocol conceptually resembles matrix completion, where predictions are made over a fixed set of unknown entries in a partially observed matrix, in contrast to conventional settings where both positives and negatives differ between training and testing.

Fig 3. Prospective evaluation scenarios.

The center of the figure shows our training dataset, Twosides-2009, with drug combinations on the rows and side effects on the columns. This entire dataset is used to train the model for both warm-start and cold-start scenarios. In warm-start (left-side), the model is tested on new associations from Twosides-2014 involving the same drug combinations from Twosides-2009. Cold-start (right-side), testing is done on new drug combinations introduced in Twosides-2014. Note that while pairs $d_A,d_E$ and $d_C,d_E$ were not present during training, the single drugs $d_A$, $d_E$, and $d_C$ were present but in combination with other drugs.

The results of this evaluation are presented in the “Warm-start” columns in Table 2. The first thing to note here is the sharp drop in AUPRC across all methods, including DCSE. This reflects the shift to an unbalanced testing set, with a $~1:20$ positive-to-negative ratio, as AUPRC is highly sensitive to class imbalance [39]. In this setting, DCSE improves over competing methods by 8.08–22.47 percentage points in AUROC and 7.59–15.76 percentage points in AUPRC. The corresponding ROC and PR curves are shown in Fig C in S1 Text.

Table 2. Performance results in our prospective evaluations. Best performance is shown in bold.

	Prospective
	Warm-start		Cold-start
	AUROC	AUPRC	AUROC	AUPRC
XGBoost	71.62	12.21	56.89	4.34
DeepWalk	74.81	14.62	63.42	4.69
FG-DDI	64.07	7.09	55.40	3.91
KnowDDI	64.74	6.84	55.54	3.61
SSF-DDI	65.12	6.45	60.78	4.35
GMPNN	67.25	7.29	61.02	4.56
DeepDDI	78.46	14.15	88.47	23.04
DCSE	86.54	22.21	89.70	27.32

DCSE’s performance indicates that the signatures learned for drug combinations effectively capture their interplay with the different side effects and can generalize well to predict new associations. For DeepDDI, we observe a notable improvement in performance compared to the setting where separate models were trained for each side effect. This gain can be attributed to the unified multi-label training setup, which allows the model to exploit dependencies between side effects. DeepWalk and XGBoost underperform here because they are trained separately for each side effect, so they cannot exploit the same dependencies.

KnowDDI, SSF-DDI, FG-DDI, and GMPNN show substantially lower performance here. It is important to note that these results differ considerably from those reported in their original publications. In their experimental settings, these methods report AUROC values of up to 0.99, with similarly high AUPRC values [20,22–24]. Under these procedures, negative samples are generated by corrupting one of the drugs in a positive association [5,20]. This corruption can produce drug pairs with no known associations with any side effect, which generally makes the resulting negative associations easier to distinguish from positive associations. We further verified that this occurs in the negative samples generated for these methods, as more than 60% of the corrupted drug pairs have no positive associations. In our warm-start setting, the task is more challenging because the model must identify new side effects among the set of drug pairs with previously known associations. This may explain why these methods struggle to separate the new positive associations from the remaining unknown associations in Twosides-2014.

Cold-start evaluation.

The cold-start setting amounts to a de-novo prediction of side effects for new drug combinations. The testing set corresponds to the new drug pairs in Twosides-2014 that are not present in Twosides 2009, while ensuring that the drugs forming these pairs appeared during training. The composition of the test set can be observed in the “Cold-start” matrix of Fig 3, where the pair composed of drugs $d_A$ and $d_E$ is absent from the training data, but both drugs are present individually in pairs involving other drugs in the training set (Twosides-2009). For our negative associations, we use all unknown drug pair-side-effect observations corresponding to these new pairs.

The results of this evaluation are presented in the “Cold-start” columns in Table 2. In this case, we have a $~1:33$ positive-to-negative ratio. It is important to note that DCSE can make these predictions because it learns signatures for individual drugs. As shown in Fig 2, each drug is passed through a shared embedding module that produces its signature. The individual drug signatures are then passed through the combination layers to generate a signature for the drug combination. This allows DCSE to produce signatures even for pairs not seen during training, as long as both drug signatures are available. Here, DCSE improves over competing methods by 1.23–34.30 percentage points in AUROC and 4.28–23.71 percentage points in AUPRC. The corresponding ROC and PR curves are shown in Fig D in S1 Text.

The higher performance observed in the cold-start setting should be interpreted in light of the distinction between the warm- and cold-start settings. In the warm-start setting, the new positive associations from Twosides-2014 correspond to entries that were treated as negatives during training in Twosides-2009, as shown by the green cells in Fig 3. This connects to the more challenging negative associations discussed above, where the model must rank newly discovered side effects above other unknown side effects for drug pairs that already had known associations. In the cold-start setting, neither the positive nor the negative associations for the evaluated drug pairs were present during training, although the individual drugs appeared in other combinations. This explains why DCSE and DeepDDI perform substantially better than DeepWalk and XGBoost in this setting. Both methods can use information learned from other combinations involving the same individual drugs, allowing them to transfer signals to drug pairs that were not observed during training. DCSE further improves over DeepDDI since it learns relationships between drug combinations and side effects directly from the association data in a shared latent space, making the side-effect-specific comparison part of the prediction mechanism. DeepDDI can share information across labels, but it maps fixed drug-pair features to output probabilities, so the side-effect structure is less explicitly modeled. In contrast, DeepWalk and XGBoost train independent models for each side effect and have less capacity to transfer information about how individual drugs behave across different combinations.

While GMPNN, KnowDDI, SSF-DDI, and FG-DDI train a single model for all side effects, similarly to DCSE and DeepDDI, their performance falls even lower than in the warm-start setting. This suggests that sharing information across side effects is not sufficient when the model has primarily learned to separate positives from easier negatives, ultimately making it harder for these methods to generalize to more realistic testing scenarios.

Performance across MedDRA side-effect classes.

To understand whether DCSE’s performance varies across anatomical and physiological side-effect phenotypes, we evaluate predictions according to the Medical Dictionary for Regulatory Activities (MedDRA) terminology. Similarly to previous work [30], we group side effects by System Organ Class (SOC) and compute AUROC and AUPRC within each class. To avoid drawing conclusions from SOCs with too few side effects in our prospective test sets, we include only SOCs with at least 20 represented side effects. This results in 24 SOCs in the warm-start setting and 20 SOCs in the cold-start setting. The complete SOC-level AUROC and AUPRC results are shown in Fig E and Fig F in S1 Text.

In the warm-start setting, DCSE shows consistently high AUROC across SOCs, ranging from 0.812 to 0.898, with the highest AUROC observed for infections and infestations, eye disorders, and endocrine disorders. In the cold-start setting, AUROC ranges from 0.781 to 0.937, with the highest values observed for blood and lymphatic system disorders, skin and subcutaneous tissue disorders, and gastrointestinal disorders. AUPRC varies more strongly across SOCs, reflecting differences in class prevalence, with the highest values observed for general disorders and administration site conditions and blood and lymphatic system disorders. Overall, this suggests that DCSE maintains strong discriminative performance across diverse side-effect classes, while the larger differences in AUPRC mainly reflect differences in the number of positive associations available for each SOC.

External evidence for previously uncharacterized drug pairs.

To further evaluate the applicability of DCSE in the cold-start setting, we inspect the highest scoring predictions for drug pairs that are absent from Twosides-2009 to determine whether they have independent evidence of interaction in external resources. Specifically, we focus on the top 50 DCSE predictions in the cold-start setting, which include 20 unique drug pairs. We then selected DrugBank and Drugs.com as external sources of drug interaction information. DrugBank reports curated DDI descriptions, typically phrased as changes in pharmacological activity, exposure, efficacy, toxicity, or side-effect risk (https://dev.drugbank.com/guides/drug_interactions/overview). Drugs.com reports pair-level clinical interaction warnings, with severity labels and adverse-effect descriptions derived from Cerner Multum interaction content (https://www.drugs.com/drug_interactions.html). It’s important to note that these resources are used only after model training and evaluation and play no role in learning DCSE signatures. Among the 20 unique pairs, 19 have supporting evidence in DrugBank, while Drugs.com reports interaction warnings for 8 pairs. The full breakdown of these pairs and their external evidence is provided in S1 Table. This suggests that the highest-ranking DCSE predictions recover interaction signals that are also supported by independent pharmacological and clinical resources. This high proportion is encouraging evidence that DCSE could prioritize cold-start drug combinations that warrant further pharmacological or clinical inspection.

Next, we want to connect these top-ranked predictions to a concrete clinical setting. We examine whether DCSE prioritizes plausible side effects for a cold-start drug pair that is also commonly coprescribed. Among our top predictions, we investigate fluoxetine and naratriptan. Fluoxetine is a selective serotonin reuptake inhibitor, while naratriptan is a triptan used for migraine treatment. Coprescription of triptans with selective serotonin reuptake inhibitors or serotonin-norepinephrine reuptake inhibitors is also observed in clinical practice. In a registry study of 47,968 triptan users, Orlova et al. found that 19,017 patients were coprescribed triptans and antidepressants, with yearly coprescription proportions ranging from 21% to 29% after 2002 [40]. Furthermore, Drugs.com classifies fluoxetine and naratriptan as a major interaction and reports increased risk of serotonin syndrome for this combination [41]. DrugBank also reports that the risk or severity of serotonin syndrome can be increased when naratriptan is combined with fluoxetine. Although serotonin syndrome itself is absent from the Twosides dataset, DCSE ranks several symptoms compatible with serotonin syndrome highly for this pair. These are nausea, anxiety, insomnia, dizziness, tremor, hyperhidrosis, increased heart rate, convulsion, and coma [42], all of which are above the 99.88th percentile of our predicted scores. Furthermore, most of them (except hyperhidrosis and coma) are confirmed positives for this pair in Twosides-2014. DCSE’s ability to prioritize these symptoms suggests that it can help identify drug combinations that are more likely to elicit clinically relevant side effects, which could be useful for clinical decision-making and drug development.

Discussion

We presented DCSE, a novel method for the prediction of polypharmacy side effects. DCSE learns signatures for drugs, combinations, and side effects to infer an association between a pair of drugs and a given side effect. The work by Galeano et al. showed that drugs and side effects can be represented as vectors in a low-dimensional space (signatures), such that the frequency of a side effect for a given drug is given by the dot product between their corresponding signatures. This work extends a similar reasoning to the case of drug combinations. We show that the probability of a side effect for a given drug combination can be approximated by the logistic of the dot product between their signatures. We also showed that combining individual drug signatures is a highly nonlinear problem, and we developed an approach to learn signatures for pairs of drugs. We examined the role of the bias terms in DCSE and found a strong correlation with the popularity of drug pairs and side effects, as measured by their frequency in the training data. This suggests that the biases capture general prevalence patterns in the data, allowing the learned signatures to focus on association patterns that are not explained by frequency alone (see S1 Text for details). DCSE learns signatures for drug combinations solely from interaction data; thus, our work shows that state-of-the-art performance can be achieved by only using information about known interactions between pairs of drugs. We believe that these results can be further improved by integrating external information, such as chemical and substructural information about drugs.

We have shown the performance improvement of DCSE over representative state-of-the-art methods in the most widely adopted experimental settings in the literature. We have then discussed the limitations of these settings, and we have proposed a novel evaluation setting where we used data from Twosides-2009 to make predictions of polypharmacy side effects introduced in Twosides-2014; we did not balance the testing datasets; and we focused only on the more challenging negative associations. As done before by other authors [20], our evaluation distinguished between warm and cold-start scenarios, where drug pairs in testing were either encountered during training or were completely new. DCSE consistently achieved higher performance than its state-of-the-art competitors across these scenarios. The results for the competitors using our evaluation settings highlight the importance of our testing approach. In the evaluations used in their original publications, these methods reported very high performance, with AUROC values ranging from 0.90 to 0.99, and similarly high AUPRC values [20,22–24]. Under these settings, DCSE obtained an AUROC of approximately 0.92 and a similarly high AUPRC, showing competitive performance, albeit lower than that reported by methods such as FG-DDI. However, when evaluated in our more realistic scenario, the performance of these competitors falls substantially, while DCSE maintains strong performance. We contend that our evaluation setting offers a more authentic evaluation, closely resembling how side effects are identified in the patient population and more closely reflecting the class distribution observed in real-life scenarios. Moreover, testing the challenging negative associations is particularly relevant in pharmacovigilance, where the main question is often which side effects should be prioritized for expert review [43]. For this reason, ranking a true side effect above other candidate side effects for a drug pair for which some side effects are already known is more informative than distinguishing it from negative examples involving unrelated drug pairs. DCSE’s high performance indicates that the signatures for the drugs and their combinations can effectively capture the interplay with the different side effects and generalize well to new, meaningful associations. Moreover, the external evidence observed for the top cold-start predictions suggests that DCSE can identify clinically meaningful signals for drug combinations with incomplete side effect profiles. This supports its potential use for prioritizing side effects that warrant follow-up in pharmacovigilance.

Galeano et al. showed that their learned signatures offer interpretability in terms of biological information about drug activity at the anatomical and molecular level [30]. We attempted to derive a similar biological interpretation for the signatures learned by DCSE. Our first step was to ensure the stability of these signatures across different model runs [44,45]. However, we encountered challenges with the reproducibility of the signatures learned by DCSE. Nevertheless, DCSE’s predictive performance remains consistent across various random initializations, as indicated by the low standard deviations shown in Table 1.

It is important to acknowledge certain limitations of our approach. Firstly, DCSE is unable to generate de-novo predictions for drug combinations formed by drugs that did not appear at all during training. This limits the immediate deployment of DCSE to combinations formed from drugs already represented in the association data. However, this still leaves a large space for post-marketing pharmacovigilance and drug repurposing. In our settings, the 645 represented drugs define 207,690 possible drug pairs, which is far larger than the number of combinations that can be systematically evaluated in clinical trials. Therefore, even within this restriction, DCSE can help prioritize side effects for a broad set of drug combinations with existing post-marketing evidence. Secondly, DCSE is effectively a collaborative filtering approach, and therefore, it relies on the availability of very diverse association data to learn meaningful signatures. For this reason, it is unlikely to work on datasets such as DrugBank that count only 86 adverse events, that are very few compared to the 1,318 side effects in Twosides. We believe that both these limitations can be addressed by integrating external information, such as chemical and substructural information about drugs, and will tend to become less relevant with the ever-increasing size of available datasets. A further limitation is that Twosides is derived from spontaneous adverse-event reports, which are known to be affected by reporting bias and unmeasured confounding, such as indication bias, concomitant medications, patient demographics, medical history, and reasons for prescribing a drug [9]. Consequently, DCSE predictions should be interpreted as prioritization signals that require external pharmacological or clinical validation before any causal interpretation is made. Beyond these limitations, the architecture of DCSE is naturally suited for extension to combinations involving more than two drugs. This would simply require adding additional drug inputs to the combination layers, so that the signatures of all drugs in the combination are used to learn a higher-order combination signature. While the MLP used by DCSE provides a natural starting point for this extension, other approaches, such as attention-based modules, could also be explored for combining multiple drug signatures.

Materials and methods

Polypharmacy side effects data

Our experiments were carried out using polypharmacy side effects association data from the Twosides database [9], which contains information about 63,473 drug combinations and 1,318 side effects. The total number of individual drugs that make up the combinations is 645. In total, there are 4,543,686 associations between side effects and drug pairs. Twosides is the result of a data-driven approach that identifies significant associations between drugs and side effects from the Adverse Event Reporting System, which collects reports from doctors, patients, and drug companies.

We used two versions of this dataset: Twosides-2009, which includes reports up to and including 2009; and Twosides-2014, which contains reports up to and including 2014. To facilitate further research, we provide our Twosides-2009 and Twosides-2014 datasets for prospective evaluations in https://doi.org/10.6084/m9.figshare.30355195. These datasets were pre-processed to include the new associations for the same drugs and side effects present in Twosides-2009.

Formal definition of DCSE

Given a dataset containing m drugs and n side effects, let the input to the model be represented by three vectors, ${\mathbf{\text{d}}}_i$, ${\mathbf{\text{d}}}_j$, and ${\mathbf{\text{r}}}_l$, with a one-hot encoding that determines the identity of the two drugs and the side effect, respectively (see Fig 2). Next is the embedding layer, a fully connected layer that projects the sparse encoding to a dense vector. To ensure consistent representations for all drugs, we use the same layer $W\in {\mathrm{ℝ}}^{m\times k}$ for both drug inputs, where k is the signature size. For the side effect input, the embedding layer is denoted as $H\in {\mathrm{ℝ}}^{n\times k}$. Thus, the signatures for drugs and side effects are obtained as follows:

$$\begin{array}{c}\begin{array}{cc}{\mathbf{\text{w}}}_i& =W^T{\mathbf{\text{d}}}_i\\ {\mathbf{\text{h}}}_l& =H^T{\mathbf{\text{r}}}_l\end{array}\end{array}$$ (1)

where $\mathbf{\text{d}}\in {\mathrm{ℝ}}^{m\times 1},\mathbf{\text{r}}\in {\mathrm{ℝ}}^{n\times 1}$ and $\mathbf{\text{w}},\mathbf{\text{h}}\in {\mathrm{ℝ}}^{k\times 1}$.

The output of the combination layers represents the combination signature and is obtained as follows:

$$\begin{array}{c}\begin{array}{c}{\mathbf{\text{c}}}_{i,j}={\varphi }_{out}\left({\varphi }_X(...{\varphi }_2\left({\varphi }_1({\mathbf{\text{w}}}_i\oplus {\mathbf{\text{w}}}_j)\right)...)\right).\end{array}\end{array}$$ (2)

Here ⊕ represents concatenation, ${\varphi }_{out}$ and ${\varphi }_x$ respectively denote the mapping function for the output layer and the $x_{th}$ hidden layer, where there are X hidden layers in total, and ${\mathbf{\text{c}}}_{i,j}\in {\mathrm{ℝ}}^{k\times 1}$ denotes the combination signature of drugs i and j. It’s important to note that following Galeano et al.’s work [30], w, h, and c are kept non-negative to favor interpretability of the signatures. This constraint is implemented by applying a rectified linear unit (ReLU) activation to the outputs of the drug and side-effect embedding layers, and to the output of the combination MLP that produces $c_{i,j}$.

Similarly to how DCSE learns the combination signature, we learn a bias for the combination of drugs by nonlinearly combining the individual biases of the drugs with a small MLP. Specifically, DCSE learns a vector $\mathbf{\text{u}}\in {\mathrm{ℝ}}^{m\times 1}$ to represent the drug biases and a vector $\mathbf{\text{v}}\in {\mathrm{ℝ}}^{n\times 1}$ to represent the side effect biases. The biases $u_i,u_j\in \mathrm{ℝ}$ for drugs i and j, respectively, the side effect bias $v_l\in \mathrm{ℝ}$, and the combination bias $p_{i,j}$ are obtained as follows:

$$\begin{array}{c}\begin{array}{cc}{u}_i& ={\mathbf{\text{u}}}^T{\mathbf{\text{d}}}_i\\ v_l& ={\mathbf{\text{v}}}^T{\mathbf{\text{r}}}_l\\ p_{i,j}& ={\varphi }_{out}\left({\varphi }_B(...{\varphi }_2\left({\varphi }_1(u_i\oplus u_j)\right)...)\right),\end{array}\end{array}$$ (3)

where B hidden layers are employed in total for the bias combination MLP. Finally, the predicted score ${\hat{y}}_{i,j,l}$ is obtained by

$$\begin{array}{c}\begin{array}{c}{\hat{y}}_{i,j,l}=\sigma ({\mathbf{\text{c}}}_{i,j}^{\text{ T}}{\mathbf{\text{h}}}_l-p_{i,j}{v}_l)\end{array}\end{array}$$ (4)

The term ${\mathbf{\text{c}}}_{i,j}^T{\mathbf{\text{h}}}_l$ represents the dot product between the combination and side effect signatures, while $p_{i,j}{v}_l$ denotes the bias term. Here, $\sigma$ represents the sigmoid activation function and maps the output value ${\hat{y}}_{i,j,l}$ to a probability between 0 and 1. Note that since both ci,j and $h_l$ are non-negative, their product is also non-negative. To capture the full range of the sigmoid function, the bias term is subtracted from this dot product.

Implementation details

All experiments were conducted using an NVIDIA QUADRO RTX 5000 GPU. The model was implemented in PyTorch, and the hyperparameters were tuned using a validation set, following the same procedure described in Section Our proposed prospective experimental settings. The optimal architecture for the combination MLP was a 3-layer neural network, with each hidden layer containing 1,024 neurons, and a signature size for the drugs, combinations, and side effects of 512. For the bias MLP, the architecture is composed of two hidden layers of 4 neurons each. To ensure the model’s invariance regarding the order of drugs, each input drug is presented in both positions as distinct samples. For the prospective experiments, DCSE is trained with binary cross-entropy loss using Adam. We do not perform negative sampling in the prospective setting; instead, the model is trained on the full Twosides-2009 drug-pair/side-effect matrix, where known associations are labeled as positives and unobserved associations are labeled as negatives. Training is performed for a fixed number of epochs, without early stopping, and random seeds are fixed for reproducibility.

The code for the implementation is available at: https://github.com/paccanarolab/DCSE.

Details about baselines and competitors

• DeepWalk [17]: similar to DCSE, this method learns k-dimensional neural features for each drug based on a biased random walk procedure exploring network neighborhoods of nodes. To implement it for our experiments, drug combinations are represented by concatenating learned drug feature representations and used as input to a logistic regression classifier. For each side effect we train a separate classifier. DeepWalk was used as a baseline method in [5].

• XGBoost [18]: to implement this method for our experiments, we follow the procedure detailed by Zitnik et al. [5]. We first construct a feature vector for each drug based on the PCA representation of the drug-target protein interaction matrix and the PCA representation of the side effect profile of each individual drug. The drug pairs are then represented by the concatenation of the corresponding drug feature vectors and used as input to the XGBoost classifier that then attempts to predict the exact side effect for a given drug pair. Similarly to DeepWalk, a separate model is trained for each side effect.

• DeepDDI [19]: in this approach, drugs are represented by a structural similarity profile that contains pairwise structural similarity scores obtained from the comparison between the input drug and all the FDA-approved drugs. The features for each drug are concatenated to represent the drug pair and then fed to a deep neural network to predict the side effect associations. In the experiments following the conventional settings (Section Conventional experimental settings), DeepDDI was trained separately for each type of side effect with a single neuron as output, indicating the probability of the side effect occurring for the input drug pair. Later in our proposed experimental settings (Section Our proposed prospective experimental settings), DeepDDI was adjusted – and framed as a multi-label classification problem – to output a probability for all possible side effects given one drug pair, i.e., having the same number of output neurons as side effect types in the dataset.

• Decagon [5]: this approach first constructs a multimodal graph of protein-protein interactions, drug-protein target interactions, and polypharmacy side effects, which are represented as drug-drug interactions, where each edge has multiple types corresponding to all the different side effects. Decagon consists of a graph convolutional neural network for multi-relational link prediction in multimodal networks. As we were unable to run the provided code, we compared our method to Decagon by replicating the same experimental conditions (Section Conventional experimental settings). We were unable to run Decagon’s code base, and therefore, it was excluded from the different evaluation settings as it would require substantial modifications to the original implementation. To still compare fairly against this highly representative method, we include a comparison of DCSE against them using their proposed evaluation methodology.

• GMPNN [20]: uses a Gated Message Passing Neural Network (GMPNN), which learns chemical substructures with different sizes and shapes from the molecular graph representations of drugs for DDI prediction for a pair of drugs. The final prediction is based on the interactions between pairs of their learned substructures. For our prospective experiments, we have adjusted the code to support training with the entire Twosides-2009 and score on our Twosides-2014 datasets.

• KnowDDI [22]: this method augments a DDI graph with an external biomedical knowledge graph and learns drug-pair-specific knowledge subgraphs to predict DDI types. The subgraphs are used by a graph neural network to capture the local biomedical context around each drug pair. Since this method was originally trained for a reduced subset of Twosides, we adapted its implementation to train on the full Twosides-2009 and evaluate the resulting model on our Twosides-2014 warm-start and cold-start datasets.

• SSF-DDI [23]: combines drug sequence information with molecular substructure information to represent each drug before predicting the interaction type for a drug pair. Sequence features are extracted with a convolutional neural network, while substructure features are obtained from molecular graph representations. In our prospective evaluation, the method was retrained with the full Twosides-2009 and then tested on the corresponding Twosides-2014 associations.

• FG-DDI [24]: is a functional group-aware graph neural network designed to incorporate medicinal-chemistry information into molecular message passing. Functional group enrichment scores are used to modulate both intra-molecular and inter-molecular interactions between drugs. We trained FG-DDI using the entire Twosides-2009 and applied the trained model to score the Twosides-2014 prospective datasets used in our evaluations.

Supporting information

S1 Text. Supplementary material.

This file includes additional experiments, definitions, and figures.

(PDF)

S1 Table. External interaction evidence for the top DCSE cold-start predictions.

This table reports Drugs.com and DrugBank evidence for the 20 unique drug pairs represented among the top 50 DCSE predictions in the cold-start prospective setting.

(PDF)

Data Availability

All data used in this study were derived from the publicly available Twosides database, which contains statistically significant associations between drug pairs and side effects. Two curated versions of this dataset were employed: Twosides-2009, comprising reports up to and including 2009, and Twosides-2014, which extends coverage to 2014. To facilitate reproducibility and prospective evaluation, both Twosides-2009 and Twosides-2014 datasets used during this study are openly available on Figshare at https://doi.org/10.6084/m9.figshare.30355195. The source code implementing the DCSE model is publicly accessible in the GitHub repository https://github.com/paccanarolab/DCSE.

Funding Statement

AP was supported by Biotechnology and Biological Sciences Research Council (https://bbsrc.ukri.org/) grant number BB/K004131/1, BB/F00964X/1, and BB/ M025047/1; Medical Research Council (https://mrc.ukri.org) grant number MR/ T001070/1; Consejo Nacional de Ciencia y Tecnología Paraguay (https://www.conacyt.gov.py/) grant number PINV01-719, PINV01-108; National Science Foundation Advances in Bio Informatics (https://www.nsf.gov/) grant number 1660648; Fundacão de Amparo a Pesquisa do Estado do Rio de Janeiro (https://www.faperj.br) grant number E-26/ 201.079/2021 (260380), E-26/204.352/2024 (299859) and E-26/210.037/2026 (319256); Conselho Nacional de Desenvolvimento Científico e Tecnológico (https://www.cnpq.br) grant number 311181/2022-8 and 313268/2026-6; and Fundacão Getulio Vargas. RJ was supported by the College Studentship from Royal Holloway, University of London; Fundacão Getulio Vargas; and Consejo Nacional de Ciencia y Tecnología Paraguay (https://www.conacyt.gov.py/) grants number PINV01-719, PINV01-108. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.