SynBa: improved estimation of drug combination synergies with uncertainty quantification

Abstract

Motivation

There exists a range of different quantification frameworks to estimate the synergistic effect of drug combinations. The diversity and disagreement in estimates make it challenging to determine which combinations from a large drug screening should be proceeded with. Furthermore, the lack of accurate uncertainty quantification for those estimates precludes the choice of optimal drug combinations based on the most favourable synergistic effect.

Results

In this work, we propose SynBa, a flexible Bayesian approach to estimate the uncertainty of the synergistic efficacy and potency of drug combinations, so that actionable decisions can be derived from the model outputs. The actionability is enabled by incorporating the Hill equation into SynBa, so that the parameters representing the potency and the efficacy can be preserved. Existing knowledge may be conveniently inserted due to the flexibility of the prior, as shown by the empirical Beta prior defined for the normalized maximal inhibition. Through experiments on large combination screenings and comparison against benchmark methods, we show that SynBa provides improved accuracy of dose–response predictions and better-calibrated uncertainty estimation for the parameters and the predictions.

Availability and implementation

The code for SynBa is available at https://github.com/HaotingZhang1/SynBa. The datasets are publicly available (DOI of DREAM: 10.7303/syn4231880; DOI of the NCI-ALMANAC subset: 10.5281/zenodo.4135059).

1 Introduction

With the increased use of small molecule drugs in monotherapy and drug combination treatments, off-target toxicity and resistance to treatments are becoming clear challenges. The use of combinations of drugs offers a possible solution to reduce toxicity minimizing doses and bypassing resistance with alternative targeting. Thanks to the development of high-throughput approaches to screen drug sensitivity in cell lines, dose–response data for a large number of combinations have been made available. Examples include the AstraZeneca-Sanger DREAM challenge (DREAM) (Menden et al. 2019), NCI-ALMANAC (Holbeck et al. 2017), DrugComb (Zagidullin et al. 2019), and the screening data from the Wellcome Sanger Institute (Jaaks et al. 2022). The availability of these datasets makes it possible to predict the effect of drug combinations from modelling drug-sensitivity data, leveraging information from biological features of the cell lines and chemical characteristics of the drugs (Bulusu et al. 2016).

To understand how drugs can work synergistically when combined, models with a quantification framework are required. Traditional quantification frameworks usually involve a null surface, based on a number of set assumptions. The Bliss model is based on the Multiplicative Survival Principle (Bliss 1939), whereas the Loewe model is based on the Dose Equivalence Principle (Loewe 1953). In the past decade, parametric methods have started to emerge as alternatives to the above. These include MuSyC (Meyer et al. 2019; Wooten et al. 2021), BRAID (Twarog et al. 2016), and the Effective Dose model (Zimmer et al. 2016). All these frameworks are based on different assumptions and parameterizations. As a result of this, the outputs from these models often disagree. Although there have been efforts to unify the frameworks, this is still an open problem for the field.

To define a common framework for drug combination, we need a less ambiguous definition of ‘synergy’ (Tang et al. 2015). When a combination is said to be ‘synergistic’, it is unclear whether it implies that the combination is desirable in terms of its ‘potency’ or its ‘efficacy’. Potency is the amount of dosage required for a drug to produce a specified effect, whereas efficacy is the degree of the beneficial effect produced by the drug (Meyer et al. 2019). A strong synergistic potency implies the toxicity may be reduced when the drugs are combined, which is crucial for avoiding overdose, whereas a strong synergistic efficacy implies that the combination increases the maximal possible effect. Both aspects are relevant for progressing with pre-clinical and subsequent clinical investigation. An ideal combination would be potent and effective. However, in clinical research, there are situations where a drug partner is requested only for enhancement of potency and not to increase efficacy. Until recently, in quantification frameworks including Bliss, Loewe, BRAID, and the Effective Dose Model, potency and efficacy are entangled within the concept of ‘synergy’ as defined in the traditional quantification frameworks. To tackle this problem, Meyer et al. (2019) developed MuSyC, a framework that decouples potency and efficacy following the principles of the generalized Hill equation. Although the MuSyC approach is effective for modelling both potency and efficacy, the model does not fully explore the challenge of model-based estimation of uncertainty. There are various sources of uncertainty associated with drug-sensitivity modelling. First, the biology of dose–response relationships is still unknown despite existing efforts. This leads to uncertainty associated with insufficient scientific knowledge. Second, there are systematic and random errors arising from the experimental procedures. Third, biological variation exists among cell lines of the same disease, resulting in a further source of noise. Finally, uncertainty also stems from limited information that can be extracted due to the small size of available data (‘epistemic uncertainty’).

Given these multiple sources of uncertainty, it is often impossible to reach an accurate estimate of the quantities of interest, e.g. potency of a monotherapy or the synergistic effect (in terms of either potency or efficacy) of a combination. Since the ‘best’ single deterministic estimates might not be reached, here we focus on accurately quantifying uncertainty in the model as deviation from the true estimate given the noise (e.g. the credible range of model parameter estimates/predictions).

Most existing frameworks for drug combinations either do not compute the uncertainty or estimate it by standard error (Zimmer et al. 2016) or parametric bootstrap (Wooten et al. 2021), each of which contains unrealistic assumptions. The standard error is only accurate when a large number of samples is available, which is not the case for most pharmacology datasets. Similarly, for the parametric bootstrap, an accurate uncertainty estimation relies on a sufficiently large number of observations to be reasonably close to the truth. In either case, the uncertainty estimation will often be inaccurate due to the small data size. For this reason, here we approach uncertainty estimation with a Bayesian framework, which incorporates the uncertainty by treating all parameters of interest as probabilistic quantities. This enables us to continuously model the uncertainty in our estimates as the number of measurements grows, without becoming over-confident. Moreover, the estimated quantities of interest are obtained simultaneously with their uncertainties, making this approach computationally efficient.

In the current literature, there are examples that use a probabilistic model to incorporate uncertainty in their outputs. Hand-GP (Shapovalova et al. 2021) is a non-parametric model based on the combination of the Hand model with Gaussian processes, providing more believable uncertainty estimation than MuSyC in some cases. However, Hand-GP does not incorporate the 1D Hill equation that imposed biological constraints useful for providing interpretable model outputs. For example, monotonicity of the monotherapy fitted curve is not enforced in Hand-GP, meaning it is possible that the model produces a dose–response surface that is unlikely to occur in the in vitro setting (Tansey et al. 2022). More importantly, due to the non-parametric structure of Hand-GP, the parameters describing potency and efficacy are lost. The bayesynergy model (Rønneberg et al. 2021) is a Bayesian framework that models synergistic interaction effects using Gaussian Processes, which provide uncertainty quantification. Although flexible, its formulation is based on the Bliss independence assumption that is biased against drug combinations with a moderate level of efficacy (Wooten et al. 2021). The bayesynergy model also does not separate out synergistic potency and efficacy.

Here, we design a flexible Bayesian framework to infer synergistic effects of drug combination (SynBa) where (i) the classic Hill equation is preserved to produce estimates of efficacy and potency (ii) the existing biological knowledge or insight from historical data may be conveniently added through the prior distribution over parameters in the model. In SynBa, we will use MuSyC as a baseline framework to decouple synergistic potency and efficacy and add probabilistic inference to provide outputs and their associated uncertainty. This is to design a framework estimating the most favourable scores and efficiently provide optimal candidates to the drug discovery pipeline with actionable decision-making criteria derived from the model outputs. SynBa is also the first synergy framework that simultaneously provides principled uncertainty estimation, preserves the Hill equation, and decouples the synergistic efficacy and potency.

With our approach, when a combination is predicted to be synergistic, a level of confidence will be quantified for this prediction, together with a level of improvement in efficacy or in potency. The associated uncertainty can guide decision on further laboratory experiments to proceed to the next phase. This will provide ‘actionable metrics’ for the subsequent stages of the drug discovery pipeline, which is an unmet need.

2 Methods

Our proposed method can be used for both analysing existing dose–response data and predicting unseen dose–response data for a given monotherapy $\mathcal{D}=(X,Y)=\{(x_i,y_i)\}$ or a given combination $\mathcal{D}=(X,Y)=\{({\mathbf{x}}_i,y_i)\}$. The covariate can be a scalar x_i (in monotherapy) or a vector ${\mathbf{x}}_i$ (in combination) corresponding to the drug dosages. The response y_i can be defined as cell growth or inhibition of growth, depending on how the data are collected. In this study, we focus on inhibitory datasets, where a large dosage typically results in growth inhibition. In this case, y_i is defined as the percentage of growth-inhibited cells. Nevertheless, our method can be easily modified to accommodate the opposite setting where the drug response is enhancing growth with respect to the dosages.

Figure 1A illustrates a typical dose–response matrix for a combination from a screening, where the first row and column contain monotherapy data and the remaining entries contain combination data. The core aim of our method is to infer the synergistic potency and efficacy given such a matrix (or a vector in the case of monotherapy). To accomplish this, we designed SynBa, a Bayesian framework for the inference of Synergistic effects of drug combinations. SynBa is defined by a prior distribution $p(\mathit{\theta })$ for the parameters $\mathit{\theta }$ and a likelihood function $p(Y|\mathit{\theta },X)$ for the drug responses Y. The likelihood function describes the probability of the responses given the dosages X and the parameters $\mathit{\theta }$. The prior distribution encodes the existing belief or knowledge about the parameters. In monotherapy, we have $\mathit{\theta }=\{E_0,E_1,C,H,\sigma \}$, whereas in combination, $\mathit{\theta }=\{E_0,E_1,E_2,E_3,C_1,C_2,H_1,H_2,\alpha ,\sigma \}$. The likelihood function and parameters encode the shape of the dose–response curve (defined in Boxes 1 and 2 and described in more detail below).

Figure 1.

Description of the datasets for the proposed framework. (A) A typical dose–response matrix, where the top-left entry is the base value, the first row contains the monotherapy responses for the first agent, the first column contains the monotherapy responses for the second agent. The remaining entries are the responses when a combination of the two drugs is applied. (B) A dose–response matrix where not all responses are available for training. The shaded cells represent the test data.

In addition, our method provides a way of predicting unseen dose–response data for both monotherapies and combinations. For example, in the case of Fig. 1B, the quantity of interest would be the posterior predictive distribution of the response, i.e. where $\tilde{\mathbf{x}}$ is the untested dosage of interest, $\tilde{y}$ is the predicted response, and $p(\mathit{\theta }|X,Y)$ is the posterior distribution over the parameters $\mathit{\theta }$ given training data (X, Y).

$$p(\tilde{y}|X,Y,\tilde{\mathbf{x}})={\int }_{\mathit{\theta }}p(\tilde{y}|\mathit{\theta },\tilde{\mathbf{x}})p(\mathit{\theta }|X,Y)\mathrm{d}\mathit{\theta },$$ (1)

2.1 Overview of SynBa: Monotherapies

We begin by defining SynBa for monotherapy screens, where the dosage x is a scalar. The likelihood function for the response y is based on the Hill equation (Hill 1910), which has been the classic choice to model pharmacology data: where x is the dosage of the drug, E(x) is the corresponding measured response, and H controls the slope of the curve. The interpretation of C depends on the problem and the dataset of interest. In this study, as the focus is on inhibitory datasets, C represents the dosage required to inhibit the given biological process or biological component by 50%, known as ${\text{IC}}_{50}$. C quantifies the potency of the monotherapy in the study.

$$E(x)=E(0)+\frac{E(\infty )-E(0)}{1+{\left(\frac{C}{x}\right)}^H},$$ (2)

The base level of the monotherapy is denoted by $E_0:=E(0)$, which is the response when no drug is applied. The efficacy is quantified by $E_1:=E(\infty )$, or denoted as $E_{\text{inf}}$, which is the maximal inhibition when a sufficiently large dosage x is applied.

Box 1 defines the prior distribution for the parameters and the likelihood model for the response given the dosages, which are also illustrated in Fig. 2.

Box 1. Overview of SynBa for the inference of monotherapy dose–response data.

The joint prior distribution for the response and the parameters of the curve given the dosages is

$$p(Y,E_0,E_1,C,H,\sigma |x)=p(Y|E_0,E_1,C,H,\sigma ,x)p(E_1|E_0)p(E_0)p(C)p(H)p(\sigma ),$$ (3)

where the likelihood model for the responses Y given the dosage x is defined as

$$p(y|E_0,E_1,C,H,\sigma ,x)=\mathcal{N}(E_0+\frac{E_1-E_0}{1+{\left(\frac{C}{x}\right)}^H},{\sigma }^2)$$ (4)

independently for all $y\in Y$, where $E_0:=E(0),\hspace{0.17em}{E}_1:=E(\infty )$, and σ is the standard deviation of the noise level of y, and the priors are

$$\begin{array}{cc}\hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}p(E_0)=\mathcal{N}(B,0.03B),\hspace{1em}p\left(\frac{E_1}{E_0}\right)=\mathcal{U}(0,1)\hspace{0.17em}\text{or} \text{Beta}(0.46,0.58),\hfill \\ \hfill & p(\text{log}(C))=\mathcal{U}(\text{log}(\delta ),\hspace{0.17em}\text{log}(M)),\hspace{1em}p(H)=\text{lognormal}(0,1),\hspace{1em}p(\sigma )=\text{lognormal}(0,1),\hfill \end{array}$$ (5)

where $0=x_1<x_2\le x_3\le \cdots \le x_n$ are the dosages and $\delta \in (0,x_2)$ is a small non-zero value to avoid $\text{log}\hspace{0.17em}C$ being undefined.

Figure 2.

Two options (A) and (B) for the monotherapy prior model. The prior for $\text{log}\hspace{0.17em}C$ is plotted underneath the x-axis, which is uniform in both options. The priors for E₀ (which is Gaussian in both options) and the normalized E₁ are plotted along the y-axis. Given E₀, the normalized E₁ is uniform in Option (A) and follows a Beta(0.46, 0.58) distribution in Option (B). The 300 blue curves are random samples from the expected prior responses $\mathbb{E}[Y|\mathit{\theta }]$ where $\mathit{\theta }$ are sampled from the prior distributions defined in Box 1. The seven red points illustrate an example set of monotherapy dose–response data $\mathcal{D}$. The black curve is a sample from the expected posterior responses $\mathbb{E}[Y|\mathit{\theta },\mathcal{D}]$ after the model is fitted to the data $\mathcal{D}$, while the orange bell-shaped curves illustrate the i.i.d. Gaussian noise for the responses.

To account for observational noise in the data, we define a noise model for y centred around E(x). For each fixed experiment setting (i.e. a fixed cell line treated by a fixed monotherapy or combination in the same laboratory environment), y is modelled as having Gaussian noise that is conditionally independent given the dosage: $y\sim \mathcal{N}(E(x),{\sigma }^2)$. The assumption of conditional independence is valid because in screenings such as DREAM, the measurements for each monotherapy or combination are performed independently in different plates, instead of being performed sequentially. Therefore, given a set of measurements for a monotherapy or a combination, the noise level of their corresponding responses is independent. The i.i.d. noise is illustrated as orange bell-shaped curves in Fig. 2. As a result, the likelihood function for y is defined as Equation (4).

The datasets are normalized by measuring cell inhibition when no drug is added (at dose zero). However, due to the noise in the biological process, the inhibition at dose zero would not be the same if the experiment is repeated multiple times. Thus, the normalization procedure itself contains uncertainty. Therefore, we define E₀ to be probabilistic instead of a fixed initial value. A Gaussian prior with mean B and variance $0.03B$ is given for E₀, where B is the normalized inhibition of a dose zero, e.g. B = 100 in DREAM. The variance of this prior is defined to be $0.03B$ so that it is flexible enough to allow for errors, but not too conservative.

The prior for the normalized maximal response (i.e. $\widetilde{E_1}:=\frac{E_1}{E_0}$, with a range of $[0,1]$) may be defined in various ways depending on whether to insert existing knowledge or historical information. One option is to remain uninformative and impose a uniform prior, as shown in Fig. 2A.

Alternatively, we may make use of the existing information from the monotherapy data available. According to the single agent datasets in DREAM, the empirical distribution of $E_{\text{inf}}$ has a high density on both extremes of the range, with 17.9% of them ˂0.05% and 15.1% equal to 1 (after normalizing to the interval $[0,1]$). Using maximum likelihood estimation to fit a $\text{Beta}(a,b)$ distribution to these $E_{\text{inf}}$ values, we obtain a = 0.46 and b = 0.58. To account for this information, we define $\text{Beta}(0.46,0.58)$ as the second option of the prior for the normalized maximal response, as illustrated in Fig. 2B. This prior is consistent with the biological behaviour that a drug administered with a sufficiently large dose will either kill the majority of the targeted cells if effective, or very few of them if not effective. The choice of this empirical prior shows how existing knowledge or information on the dynamic/kinetic of drugs can be conveniently added to SynBa through its priors.

A uniform prior is imposed for $\text{log}\hspace{0.17em}C$ (the logarithm of ${\text{IC}}_{50}$), with C bounded by δ and M. The values of δ and M depend on the dataset and the unit of the dosages. In this work, we define δ to be smaller than any non-zero dosage in the dataset (i.e. $0<\delta <\min \left\{x\right|x>0,(x,y)\in \mathcal{D}\}$) and M to be larger than any dosage in the dataset (i.e. $M>\max \left\{x\right|(x,y)\in \mathcal{D}\}$). For example, $\delta ={10}^{-10}$ and $M={10}^6$ is a viable choice for DREAM, whereas in NCI-ALMANAC, we may have $\delta ={10}^{-15}$ and M = 10. The idea is to be uninformative about $\text{log}\hspace{0.17em}C$ a priori, since the ideal dosage range for the experiments is unknown and often unsuitable, either too small or too large. It is common that ${\text{IC}}_{50}$ exceeds the maximum dosage. In other cases, ${\text{IC}}_{50}$ may lie between zero dosage and the smallest non-zero dosage, due to the tested dosages being too large. Our method includes these possibilities a priori.

H and σ are both given a $\text{lognormal}(0,1)$ prior because they are both non-negative and assumed to be moderately small. In addition, previous literature indicated that H is approximately lognormal (Wooten et al. 2021). A $\text{lognormal}(0,1)$ prior ensures that $P(H<5)\approx 0.95$ and $P(\sigma <5)\approx 0.95$ a priori.

The blue curves in Fig. 2 are 300 random samples from the expected prior responses $\mathbb{E}[Y|\mathit{\theta }]$, where $\mathit{\theta }$ are sampled from the prior distributions. It can be observed that the curves cover a wide range of possibilities a priori. Yet, they are not excessively general, as all of them follow the Hill equation.

Note that the priors we have chosen are motivated by general knowledge and previous literature. When more knowledge exists for a specific combination, the prior can be conveniently adjusted to accommodate this, since the inference framework is agnostic to the choice of prior.

Figure 3 is an illustrative example of SynBa trained on a monotherapy (the compound MTOR_1 treated on the cell line MDA-MB-231) with six measurements, taken from the DREAM dataset. The first row shows the resulting model with a uniform prior for the normalized $E_{\text{inf}}$ (i.e. $\widetilde{E_1}$), whereas the second row shows the model with the $\text{Beta}(0.46,0.58)$ prior for $\widetilde{E_1}$. We start with two measurements and add two additional responses each time. It can be observed that the posterior distribution for ${\text{IC}}_{50}$ narrows down quickly for both models because the observed responses span across the range between 40 and 100, which provides sufficient information to estimate ${\text{IC}}_{50}$ with low uncertainty. The posterior for $E_{\text{inf}}$, on the other hand, is more uncertain, due to the dosage range being too small to observe the convergence of the responses. In this case, the two models provide a visibly different posterior distribution for $E_{\text{inf}}$, due to the different priors. As shown in Fig. 3F, the maximum a posteriori probability estimate for $E_{\text{inf}}$ is 0 when the Beta prior is imposed, which results from the inserted prior knowledge that the response is likely to converge to 0 if an effective (but non-zero) response is already observed. This example shows how the prior design affects the inference of the parameter uncertainty. Nevertheless, if we look at the posterior predictive distribution for the responses (with samples illustrated by light blue curves), the two models reach a similar conclusion. We will show in Section 3 that the predictive performance of the model is insensitive to the choices of prior.

Figure 3.

An example illustration of SynBa on a set of monotherapy data with six measurements. Each blue curve is a sample from $\mathbb{E}[Y|\mathit{\theta }]$ where $\mathit{\theta }\sim p(\mathit{\theta }|\mathcal{D})$. The distribution for ${\text{IC}}_{50}$ is shown in red, whereas the distribution for $E_{\text{inf}}$ is shown in green. (A)–(C) The prior distribution for $\widetilde{E_1}$ is $\mathcal{U}(0,1)$. (D)–(F) The prior distribution for $\widetilde{E_1}$ is $\text{Beta}(0.46,0.58)$.

2.2 Overview of SynBa: Combinations

Extending SynBa to combinations of two drugs, the goal is now to model the dose–response surface $E(x_1,x_2)=f(x_1,x_2)$ where x₁ and x₂ are the dosages for the two drugs, whereas $E(x_1,x_2)$ is the response, and f is some class of function to be defined.

To define our likelihood model for the responses Y, we take inspiration from MuSyC but maintain some flexibility on their model assumptions. The effect of a drug in a system is usually described by the Hill equation that describes the state of equilibrium of a reversible process between an unaffected population and an affected one (the principle of ‘detailed balance’). To obey to this equation and its effects, in our model, we incorporate the assumptions of the principle of detailed balance, of the proliferation rate of unaffected population and of the saturation of the maximum effect of the drug in the affected population. In MuSyC, these assumptions are defined in a nested structure in which levels are called Tiers (see Supplementary Table S5 in Meyer et al. 2019). In defining SynBa for combinations, we adopt the same model assumptions as Tier 4 of the levels specified by MuSyC. This category encodes the most complex class of models that still maintains the assumption of detailed balance. It is worth noting that, conversely to MuSyC, our model posterior covers all four tiers simultaneously. This is because Tier 4 subsumes all lower tiers and thus for how our model is defined, they will not be eliminated from the posterior distribution, unless the evidence from the data is strongly against them. The concepts of Tiers as described in MuSyC would require a post-learning model selection. The use of a Bayesian approach avoids such a selection procedure whilst still maintaining biologically viable assumptions.

In contrast to MuSyC, in SynBa, we choose to maintain the detailed balance assumption to also avoid over-parameterization, which is likely to occur due to the limited data size for each drug combination set. For example, the majority of the combinations in the NCI-ALMANAC dataset has a data size of 15 (excluding the base level at dose zero), only 3 more than the number of parameters in MuSyC. In a real-world scenario, the data size may often be even smaller. Furthermore, with the detailed balance assumption, matrix multiplication and inversion are avoided, which lowers the computational cost.

Box 2 defines the prior distribution for the parameters and the likelihood model for the response given the dosages. The definitions of the priors are a natural extension of the monotherapy model, with the same arguments being followed. The only new parameter is α, which follows a lognormal prior with median 1 because α is non-negative and equals 1 when the combination is neither synergistic nor antagonistic in terms of potency.

Box 2. Overview of SynBa for the inference of combination dose–response data.

We define the following joint distribution for the response and the parameters of the surface given the dosages $\mathbf{x}=(x_1,x_2)$,

$$\begin{array}{cc}\hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}p(Y,E_0,E_1,E_2,E_3,C_1,C_2,H_1,H_2,\sigma ,\alpha |\mathbf{x})\hfill \\ =\hfill & \hspace{0.17em}p(Y|E_0,E_1,E_2,E_3,C_1,C_2,H_1,H_2,\sigma ,\alpha ,\mathbf{x})p(E_1|E_0)p(E_2|E_0)p(E_3|E_0)p(E_0)p(C_1)p(C_2)p(H_1)p(H_2)p(\sigma )p(\alpha ),\hfill \end{array}$$ (6)

where the likelihood model for responses Y given the dosages is defined as

$$p(y|E_0,E_1,E_2,E_3,C_1,C_2,H_1,H_2,\sigma ,\alpha ,\mathbf{x})=\mathcal{N}(\frac{C_1^{H_1}{C}_2^{H_2}{E}_0+x_1^{H_1}{C}_2^{H_2}{E}_1+C_1^{H_1}{x}_2^{H_2}{E}_2+\alpha x_1^{H_1}{x}_2^{H_2}{E}_3}{C_1^{H_1}{C}_2^{H_2}+x_1^{H_1}{C}_2^{H_2}+C_1^{H_1}{x}_2^{H_2}+\alpha x_1^{H_1}{x}_2^{H_2}},{\sigma }^2)$$ (7)

for all $y\in Y$, where $E_0:=E(0,0),\hspace{0.17em}{E}_1:=E(\infty ,0),\hspace{0.17em}{E}_2:=E(0,\infty ),\hspace{0.17em}{E}_3:=E(\infty ,\infty )$, C₁ and H₁ are the monotherapeutic parameters associated for Drug 1, C₂ and H₂ are the monotherapeutic parameters associated for Drug 2, and α is an association parameter that controls how the two drugs are affected by the presence of each other.

The priors are

$$\begin{array}{c}p(E_0)=\mathcal{N}(B,0.03B),\hspace{1em}p\left(\frac{E_i}{E_0}\right)=\mathcal{U}(0,1)\hspace{0.17em}\text{or} \text{Beta}(0.46,0.58)\hspace{1em}\text{for} i=1,2,3,\\ p(\text{log}(C_i))=\mathcal{U}(\text{log}(\delta ),\hspace{0.17em}\text{log}(M))\hspace{0.17em}\hspace{0.17em}\text{and}\hspace{0.17em}\hspace{0.17em}p(H_i)=\text{lognormal}(0,1)\hspace{1em}\text{for} i=1,2,\text{and}\\ p(\alpha )=\text{lognormal}(0,1),\hspace{1em}p(\sigma )=\text{lognormal}(0,1),\end{array}$$ (8)

where $0=x_{i,1}\le x_{i,2}\le \cdots \le x_{i,n}$ are the dosages for drug i, and δ is a small non-zero value to avoid $\text{log}\hspace{0.17em}C$ being undefined.

2.2.1 Inference of the synergy

After inferring the posterior for the parameters and their associated uncertainty, we focus on distinguishing the effect of efficacy and potency in drug combinations. MuSyC has defined metrics for both synergistic efficacy and synergistic potency, which is a promising step in decoupling potency and efficacy. However, the uncertainty for these two quantities has not been quantified systematically. Our model output includes not only quantification for the synergistic efficacy and the synergistic potency, but also a separate uncertainty estimation for each.

For the synergistic efficacy, one simple yet informative quantity is $\Delta \text{HSA}=\min (E_1,E_2)-E_3$ which is the change in the maximal effect between the combination and the more effective single drug of the two (Greco 1995). A positive score indicates synergistic efficacy. As E₁, E₂, and E₃ are probabilistic, the resulting $\Delta \text{HSA}$ score is also probabilistic. A metric such as can then be defined to estimate how confident we are about the ‘synergistic efficacy’ of the combination, based on the dataset $\mathcal{D}$. It is possible to have a synergistic combination that is highly uncertain, which would indicate that more data are required to reach confidence in the estimation.

$$P(\Delta \text{HSA}>0|\mathcal{D})$$ (9)

For the synergistic potency, α contains the required information. $\alpha >1$ would indicate synergistic potency (Wooten et al. 2021), which means the potency of the two drugs has reduced due to being combined. Consequently, we define to estimate how likely a combination satisfies ‘synergistic potency’.

$$P(\alpha >1|\mathcal{D})$$ (10)

2.3 Case studies

To illustrate how the uncertainty estimation from our method can be explained and further used for decision-making, we take two combinations from the DREAM dataset as examples.

Figure 4A shows the dose–response matrix for the combination of ADAM17 and AKT acted on the cell line BT-20. The first column is the monotherapy dose–response measurements for AKT (as the dosage for ADAM17 is zero), whilst the first row is the monotherapy measurements for ADAM17 (as the dosage for AKT is zero). It can be observed that the responses for AKT start to decrease at a higher rate when the dosage increases, but the dosage range is too small to understand its potency (${\text{IC}}_{50}$) and efficacy ($E_{\text{inf}}$). The efficacy cannot be determined because the response has not shown any sign of convergence, whereas the potency cannot be determined because it relies on understanding the maximal response, which is itself uncertain. However, if a deterministic Hill equation is fitted to the monotherapy data, it will only provide a point estimate for the ${\text{IC}}_{50}$ and $E_{\text{inf}}$, without acknowledging the above caveats. On the contrary, our method provides an uncertainty estimation for both quantities. As shown in Fig. 4B–C, the posterior distribution of ${\text{IC}}_{50}$ and $E_{\text{inf}}$ has large variances, which correspond to large uncertainty. In particular, as shown in Fig. 4C, the posterior of $E_{\text{inf}}$ for ADAM17 has a multimodal shape, which is sensible because it is unclear whether the dosage range is too small (which corresponds to the peak at 0), or the drug is simply ineffective regardless of the dosage (which corresponds to the peak at around 90).

Figure 4.

The data and the inference outputs of the combination of AKT and ADAM17 applied on the cell line BT-20. (A) The original dose–response matrix. (B)–(C) The monotherapy model outputs. Each blue curve is a sample from $\mathbb{E}[Y|\mathit{\theta }]$ where $\mathit{\theta }$ is a sample from the posterior of the respective monotherapy model. The posterior distribution for ${\text{IC}}_{50}$ and $E_{\text{inf}}$ is shown in red and green, respectively. (D) The contour plot for the joint posterior distribution of the synergistic efficacy ($\Delta \text{HSA}$) and the synergistic potency ($\text{log}(\alpha )$). The distribution is smoothed from the empirical posterior with a kernel density estimation for visualization purpose. (E) and (F) The histogram of the empirical posterior distribution for the synergistic efficacy ($\Delta \text{HSA}$) and the synergistic potency ($\text{log}(\alpha )$). The areas on the right-hand side of the red vertical lines are the probability that the combination is synergistic in terms of efficacy (E) and potency (F).

Moving to the inference of the full combination matrix, most existing synergy methods have no means to showcase the uncertainty. Our method, on the contrary, provides the uncertainty around the synergistic potency and the synergistic efficacy, as shown in Fig. 4D–F. According to the model output, the combination is moderately likely to be synergistically potent (with a probability of 85.2%), but it is difficult to conclude its synergistic efficacy (with a probability of 62.9% to be synergistic efficacious). This is reasonable because the excessively small dosage range makes it difficult to conclude anything about efficacy with high certainty, but with the 25 available measurements on the plates where the two drugs have interacted, information can be extracted on whether combining the two drugs may lower the level of toxicity required to reach the same beneficial effect.

This combination is an example where the model implies some potential in the synergy of the combination, but the level of uncertainty in the synergy is still high, which may require more measurements at larger dosages to be narrowed down.

We now consider the combination of AKT and EGFR acted on the cell line MDA-MB-468. Figure 5 shows its dose–response matrix and the inference result from our model for the monotherapies and the combination, respectively. All parameters and metrics of interest have low variances, representing low uncertainties. As shown in Fig. 5B and C, the dosages have suitable ranges and approximately follow the sigmoidal shapes of the expected dose–response fit, in particular for AKT. They contain sufficient information for the possibilities for ${\text{IC}}_{50}$ and $E_{\text{inf}}$ to be narrowed down. Similarly, the combination data are well-behaved. Figure 5C–E shows that the probabilities of this combination being synergistic in terms of potency and efficacy are both close to 100%. These are signs that this combination is worth being taken to subsequent steps in the drug development pipeline.

Figure 5.

The data and the inference outputs of the combination of EGFR and AKT applied on the cell line MDA-MB-468. (A)–(F) The same as the caption of Fig. 4.

The two examples above show that concrete decisions can be made based on the posterior distributions (e.g. for ${\text{IC}}_{50},\hspace{0.17em}{E}_{\text{inf}},\hspace{0.17em}\Delta \text{HSA}$, and α) from our model, and more importantly, the uncertainties associated with these distributions.

2.4 Training details

The models are trained by Stan, a state-of-the-art platform for statistical modelling and high-performance statistical computation, particularly for Bayesian computation (Stan Development Team 2023). The user specifies the prior model of the parameters and the likelihood model of the data, while Stan performs either full Bayesian statistical inference with Markov chain Monte Carlo (MCMC) sampling, or approximate Bayesian inference with variational inference. In this study, we opt for MCMC due to the importance of the reliability of the output, which is ensured by the asymptotic exactness of the MCMC inference. Despite choosing the slower option of the two, SynBa is still computationally efficient. Running on four CPUs of the Intel Xeon Platinum 8276 CPU Processor, the median time taken to fit SynBa (via MCMC with 1000 iterations and 4 chains, including 500 iterations in the warm-up phase) to a six-by-six dose–response matrix in DREAM is 10.2 s, which is comparable to MuSyC with bootstrap.

With this training pipeline, we can avoid the overhead that occurs during the usage of non-linear optimization packages in deterministic parametric methods such as MuSyC, BRAID, and the Effective Dose model. A different choice of the numerical algorithm (and its hyperparameters) results in a different result for those methods. On the contrary, in SynBa, the same exact result can be found asymptotically via MCMC with Stan.

For the implementation of the other benchmark methods including MuSyC, BRAID, and the Effective Dose model, the Python package synergy (Wooten and Albert 2021) is used.

3 Results

3.1 Prediction of drug combination responses

In this subsection, we show that in addition to providing uncertainty estimations, SynBa is competitive in predicting unseen responses within a dose–response matrix, and is less prone to overfitting compared with the existing methods.

The datasets of interest are DREAM (Menden et al. 2019) and NCI-ALMANAC (Holbeck et al. 2017), two of the most widely used publicly available combination screenings. In DREAM, we focus on all examples in the training set of Challenge 1 that have passed the Quality Assurance and that only contain non-negative responses and one set of replicates, which are 1631 sets of combinations in total. In NCI-ALMANAC, we focus on the subset defined in Julkunen et al. (2020), which is a subset of the data consisting of 50 unique FDA-approved drugs and 36 120 combinations. We remove examples that contain negative measurements, which results in 28 854 remaining combinations.

We leave out 20% of the non-zero dosage combinations for prediction. The models are trained using the remaining 80% dosage combinations and then evaluated on the left-out points. For the dose–response matrices in DREAM, we leave out 7 of the 35 points with non-zero dosages from the six-by-six matrix (see Fig. 1A) using a specific leave-out strategy. For each of the two monotherapy slices, one point is left out for testing. For the five-by-five combination grid (i.e. the orange cells in Fig. 1A), five points are randomly left out for testing. Figure 1B shows an example of such a train-test split. The measurements are left out in this manner so that each monotherapy contains one measurement for testing.

For the dose–response matrices in NCI-ALMANAC, it is not possible to leave out points separately for monotherapies and for combinations because the data size is too small. For most combinations, there are only three points for each monotherapy and nine points for the interactions. Thus, we directly leave out 3 of the 15 points randomly for prediction.

3.1.1 Evaluation metrics

To evaluate the predictive performance of the models, test likelihood and the root-mean-square error (RMSE) are computed using the left-out points. The former focuses on the goodness of the full predictive distribution, whereas the latter focuses solely on the goodness of the point estimates for the responses.

The computation of the test likelihood for SynBa follows Equation (1) for both monotherapy and combination. However, the right-hand side of the equation cannot be computed in a closed form, so Monte Carlo estimation is required using the expression where $\tilde{y}$ is the predicted response of the left-out dosage, $\mathcal{D}$ is the training data containing the known dosages and the known responses, and ${\mathit{\theta }}_m\sim p(\mathit{\theta }|\mathcal{D})$ are the MCMC samples from the posterior distribution for the parameters.

$$p(\tilde{y}|\mathcal{D},\tilde{\mathbf{x}})\approx \frac{1}{M}\sum _{m=1}^{M}p(\tilde{y}|{\mathit{\theta }}_m,\tilde{\mathbf{x}}),$$ (11)

For MuSyC, BRAID, and the Effective Dose model, the parametric bootstrap pipeline described in Wooten et al. (2021) is followed. Each bootstrapped dataset provides a fitted curve. The density for $\tilde{y}$ is then estimated by averaging its density computed on the models learnt from the bootstrapped datasets.

The computation of RMSE is more straightforward. For each combination, its RMSE for the test responses $\{y_1,\dots ,y_N\}$ is where ${\widehat{y}}_i$ is the point estimate for the response that corresponds to dosage ${\mathbf{x}}_i$. For SynBa, we define ${\widehat{y}}_i$ to be the posterior predictive mean $\mathbb{E}[y_i|\mathcal{D},{\mathbf{x}}_i]$, which is estimated by Monte Carlo sampling from the posterior.

$$\text{RMSE}=\sqrt{\frac{{\sum }_{i=1}^N{(y_i-{\widehat{y}}_i)}^2}{N}},$$ (12)

3.1.2 Quantitative results

We compare our prediction results against MuSyC, BRAID, and the Effective Dose model, which are three of the most widely used synergy models. For SynBa, we implement both the uniform prior and the empirical Beta prior for the normalized $E_{\text{inf}}$, which we denote as SynBa-U and SynBa-B, respectively. Table 1 show the mean and the median of the test log-likelihood and the test RMSE for MuSyC, BRAID, Effective Dose model, and SynBa. Our method outperforms all three other methods in all metrics except for the mean test log-likelihood on DREAM. In particular, our method performs well on RMSE, a metric that only considers the quality of point estimates and ignores uncertainty. The upper diagonal panels in Fig. 6 show the scatter plots directly comparing the test RMSE values between methods (visualized with the blue colour). Our method is the most competitive, as evidenced by having more points above the diagonal y = x line. These show that our method is not trading off predictive accuracy for uncertainty estimation. By following a principled Bayesian workflow, our model is strong in both prediction and uncertainty estimation.

Table 1.

The mean and the median (Mdn) of the test log-likelihood (LL) and the test RMSE for MuSyC, BRAID, the Effective Dose model (ED), and SynBa, computed on a subset of DREAM and NCI-ALMANAC, along with their standard errors.

	DREAM, LL		DREAM, RMSE		NCI-ALMANAC, LL		NCI-ALMANAC, RMSE
	mean (± se)	Mdn (± se)	mean (± se)	Mdn (± se)	mean (± se)	Mdn (± se)	mean (± se)	Mdn (± se)
MuSyC	$-3.50\pm 0.35$	$-3.09\pm 0.02$	6.11 ± 0.10	5.01 ± 0.08	$-3.86\pm 0.01$	$-3.79\pm 0.01$	$14.87\pm 0.10$	10.17 ± 0.03
BRAID	$-4.12\pm 0.69$	$-3.06\pm 0.02$	$5.71\pm 0.09$	$4.88\pm 0.08$	$-3.80\pm 0.04$	$-3.48\pm 0.01$	$9.57\pm 0.06$	$7.16\pm 0.03$
ED	$-3.37\pm 0.09$	$-3.22\pm 0.02$	$6.46\pm 0.09$	$5.66\pm 0.09$	$-3.68\pm 0.02$	$-3.48\pm 0.01$	$8.47\pm 0.05$	$6.77\pm 0.09$
SynBa-U	$-3.59\pm 0.42$	$-3.01\pm 0.02$	$5.20\pm 0.07$	$4.56\pm 0.08$	$-3.42\pm 0.02$	$-3.24\pm 0.01$	$6.72\pm 0.04$	$5.45\pm 0.05$
SynBa-B	$-3.84\pm 0.66$	$-3.01\pm 0.02$	$5.15\pm 0.07$	$4.55\pm 0.08$	$-3.39\pm 0.01$	$-3.23\pm 0.01$	$6.66\pm 0.04$	$5.43\pm 0.04$

The standard error of the mean is computed by the standard deviation of the metrics across examples divided by the square root of the number of examples. The standard error of the median is estimated by non-parametric bootstrap. SynBa with a uniform prior for the normalized $E_{\text{inf}}$ is denoted by SynBa-U. SynBa with the $\text{Beta}(0.46,0.58)$ prior for the normalized $E_{\text{inf}}$ is denoted by SynBa-B. Bold values indicate the best performance for the metric.

Figure 6.

Upper diagonal panels (with blue points): Scatter plot of the test RMSE values obtained from different methods on DREAM with a train-test split ratio of $80\%:20\%$. Lower diagonal panels (with red points): Scatter plot of the test RMSE values obtained from different methods on DREAM with a train-test split ratio of $40\%:60\%$. Diagonal panels: Histograms of the test RMSE values and their corresponding kernel density estimates, where the train-test split is 4:1 for the blue ones and 2:3 for the red ones.

It is worth noting that at least one of MuSyC, BRAID, or the Effective Dose model fail to find a solution for 4.8% of the examples in DREAM and 38.4% of them in NCI-ALMANAC, despite an effort in tuning the bounds, initial values, and hyperparameters involved in the optimization. Most likely this is because these methods rely on external optimization packages with no guaranteed convergence, which can become a problem when overparameterization becomes severe due to small data sizes. SynBa does not incur this problem since its priors ensure conservative outputs when data size is too small.

To investigate whether SynBa is prone to overfitting and how it compares to the other three methods, we perform the same prediction experiment on DREAM, but with a train-test split ratio of $40\%:60\%$ instead. As shown in the lower diagonal panels in Fig. 6, the test RMSE values (visualized with the red colour) increase significantly for MuSyC, BRAID, and the Effective Dose model. For SynBa, however, the test RMSE values have increased on average, but not by much. It can be seen that the mean value and the spread increase more significantly for the other three methods compared with SynBa. It can also be observed that the predictive performance of SynBa is not sensitive to the choice of prior, with the two priors producing very close RMSE values to each other.

In addition, we compare SynBa with bayesynergy and Hand-GP, the two probabilistic synergy models. As shown in Supplementary Table S1, the test RMSE for bayesynergy and Hand-GP is significantly higher on both datasets. This could be because drug interactions are modelled non-parametrically in both methods, meaning a wide range of functions is represented and it is difficult to narrow down the probable dose–response curves or surfaces when the data size is small or noisy.

3.2 Uncertainty calibration

For a model $\mathcal{M}$ with learnt cumulative distribution $F_{\mathcal{M}}$ with well-calibrated uncertainty, it would approximately follow the identity that $F_Y({\mathbf{x}}_i)\approx F_{\mathcal{M}}({\mathbf{x}}_i)$ for every data point $\{({\mathbf{x}}_i,y_i({\mathbf{x}}_i))|i=1,\dots ,N\}$ in the dataset, where $y_i({\mathbf{x}}_i)$ is a sample from the unknown true cumulative distribution $F_Y({\mathbf{x}}_i)$. Equivalently, assuming the measurements $y_i({\mathbf{x}}_i)$ for a combination are conditionally independent given the dosages ${\mathbf{x}}_i$, their cumulative probabilities $F(y_i):=\mathbb{P}(y_i({\mathbf{x}}_i)<F_{\mathcal{M}}({\mathbf{x}}_i))$ would be approximately uniformly distributed between 0 and 1, if $\mathcal{M}$ is well-calibrated.

In this study, for each combination in DREAM, we split the 35 measurements (excluding the base value) with a 80%:20% ratio in the same way as the prediction evaluation in the previous subsection. We then evaluate the quality of the uncertainty calibration with the Kolmogorov–Smirnov (K–S) uniformity test (Massey 1951) for the empirical cumulative probabilities across all test data points. If the model is well-calibrated, then the cumulative probabilities for the test data points will be approximately uniform for each combination. Otherwise, they will show a non-uniform pattern and the resulting P-value for the K–S test will be statistically significant. This procedure is performed across every combination in DREAM, resulting in a P-value for each combination. The histogram of the P-values for SynBa is in Supplementary Fig. S1A, showing that 6.07% of the combinations have not passed the uniformity test, and thus are not well-calibrated. As a comparison, the same procedure is performed using MuSyC. Supplementary Fig. S1B shows that 25.1% of the combinations are not well-calibrated when modelled by MuSyC, which is roughly four times as high as the number for SynBa. This shows the estimated uncertainty from SynBa is more reliable and closer to the unknown ground truth on average.

4 Discussion

Machine learning methods have been developed for preclinical modelling and the prediction of drug combinations, thanks to the availability of large screenings (Julkunen et al. 2020; Wang et al. 2021), which are beneficial for discovering and explaining drug combinations. However, a few factors prevent most from being applied to real-world drug discovery projects. One issue is that performance measures rely on synergy scores, which do not have a gold standard and contain a non-trivial amount of uncertainty, as discussed in Section 1. The Spearman correlation of the replicate experiments in DREAM (Menden et al. 2019) and O’Neil et al. (2016) are 0.56 and 0.63,respectively, which show that quantifying a combination with a single synergy score would result in high variance. However, uncertainty measurement is not included in synergy score estimation. This could be one of the reasons that 20% of drug combinations are poorly predicted by all methods in the DREAM challenge. Measuring the uncertainties associated with the estimated scores is important for the subsequent decision-making process based on the model outputs. In real-world scenarios, scientists are often facing the decision to choose among a large set of drug combinations that score similarly in terms of synergy. Without quantification of how certain (or uncertain) the estimated scores are, they will have to rely on background knowledge compromising innovation in their choices. SynBa provides a way to implement a ranking strategy in the decision process of a drug discovery pipeline, which is a real-world unmet need.

SynBa has the limitation that it only models the combination of two drugs, while there exist methods such as the Effective Dose Model that consider higher-order combinations. While this is a limitation, our initial aim is to provide a method that is reliable and not over-parameterized, to meet with practical needs.

5 Conclusions

We have developed a new framework for quantifying dose–response relationships for monotherapies and combinations that provides a full uncertainty estimation for all parameters that are associated with the monotherapies and the combinations, including information about efficacy, potency, and synergy. These uncertainty information would be helpful to the biologists to make further decisions about progressing to the next stages of the drug discovery pipeline, or whether more experiments are required to lower the level of the uncertainty and better understand the drug mechanism of action.

We have also shown that SynBa is competitive in predicting unseen responses within a given dose-response matrix, and outperforms MuSyC, BRAID, and the Effective Dose model on DREAM and NCI-ALMANAC. In addition, the prediction performance is not sensitive to the choice of the priors.

In summary, our framework is capable of providing a reliable uncertainty estimation for the potency (e.g. ${\text{IC}}_{50}$) and the efficacy (e.g. $E_{\text{inf}}$) of a monotherapy, or the synergistic potency and efficacy of a combination, in a decoupled manner, and reliably predicting unseen responses within a dose–response matrix. The parameter uncertainties can be interpreted and used as guidance for further experiments and subsequent decision making.

Supplementary data

Supplementary data are available at Bioinformatics online.

Conflict of interest

None declared.

Funding

H.Z. acknowledges the receipt of studentship award from the Health Data Research UK—The Alan Turing Institute Wellcome PhD Programme in Health Data Science [Grant Ref.: 218529/Z/19/Z] and the Wellcome Cambridge Trust Scholarship. M.R. is supported by a Wellcome Trust award [204832/B/16/Z].