A statistical framework for detecting therapy-induced resistance from drug screens

Abstract

Resistance to therapy remains a significant challenge in cancer treatment, often due to the presence of a stem-like cell population that drives tumor recurrence post-treatment. Moreover, many anticancer therapies induce plasticity, converting initially drug-sensitive cells to a more resistant state, e.g. through epigenetic processes and de-differentiation programs. Understanding the balance between therapeutic anti-tumor effects and induced resistance is critical for identifying treatment strategies. In this study, we present a robust statistical framework leveraging multi-type branching process models to characterize the evolutionary dynamics of tumor cell populations. This approach enables the detection and quantification of therapy-induced resistance using high-throughput drug screening data involving total cell counts, without requiring information on subpopulation counts. The framework is validated using both simulated (in silico) and recent experimental (in vitro) datasets, demonstrating its ability to generate meaningful predictions.

Introduction

Cellular plasticity, the ability to reversibly transition between phenotypic states, is a fundamental cell feature. In the context of cancer treatment, cellular plasticity often manifests as a transition from a therapy-sensitive state to a therapy-resistant state, a phenomenon known as therapy-induced resistance^1,2. This phenomenon has been observed not only in pharmaceutical treatments but also in other modalities such as radiotherapy^1,3,4, reducing the efficacy of a broad class of anti-cancer therapies. A critical phenotypic state transition related to therapy-induced resistance is the de-differentiation of cancer non-stem-like cells (CNSCs). This process involves mature cancer cell phenotypes (CNSCs) reverting to immature, cancer stem-like phenotypes (CSCs), which have unlimited proliferation potential and are inherently drug-resistant⁵. Understanding drug-induced transitions from CNSCs to CSCs can help shed light on tumor heterogeneity and lead to improved treatment plans⁶.

In a recent study⁷, ciclopirox olamine (CPX-O), originally an antifungal agent, was identified for its ability to induce the production of gastric CSCs from gastric CNSCs. The study utilized the SORE-GFP reporter system coupled with fluorescence-activated cell sorting (FACS) to distinguish between gastric CSCs and CNSCs. A high-throughput screening (HTS) technique was employed to assess the effects of various potential medications on a homogeneous population of gastric CNSC to explore drug-induced plasticity. One key assumption of this study is the effective separation of gastric CSCs and CNSCs using the FACS technique and the SORE-GFP reporter system, which may limit the scope of the findings. Additionally, the study noted CPX’s significant cytotoxic impact on gastric CNSCs. This complicates assessment of the level of drug-induced plasticity, since increased abundance of CSCs relative to CNSCs can both be explained by drug-induced plasticity and a selective advantage over CNSCs. Consequently, the empirical inference of drug-induced plasticity presents significant challenges. As an alternative approach, we propose using a mathematical model to understand how the drug affects the proliferation of both CNSCs and CSCs, and to tease apart the drug’s effect on cell proliferation vs. transitions from CNSCs to CSCs.

Mathematical modeling has increasingly played a crucial role in understanding and treating cancer^8,9, offering advantages in modeling complex biological dynamics through manageable mathematical models. For instance, recent studies have applied the multi-type branching process model to depict diverse tumor populations undergoing phenotypic switching^10–13. This approach abstracts the cell division process into ‘branching events’, allowing each cell to generate descendants across any subtype described in the model. As a result, the model can naturally describe processes such as self-proliferation, differentiation, and de-differentiation of CSCs and CNSCs. Mathematical modeling has also been used recently to investigate optimal dosing protocols under drug-induced resistance, which often involve low-dose metronomic or intermittent dosing strategies as opposed to more conventional maximum tolerated dose strategies^14–19. However, to apply these treatment insights to specific cancer types, we need to establish methods for detecting and quantifying drug-induced resistance from experimental data involving these cancer types.

Several recent studies^20,21 have proposed frameworks for deconvoluting subpopulation structure directly from HTS bulk cell count data, avoiding the need for a reporter system coupled with FACS to separate distinct subpopulations. Instead, these studies distinguish subpopulations based on their distinct responses to a given drug. For instance, these works have successfully identified Imatinib-sensitive and -resistant Ba/F3 cells from a population mixture by examining heterogeneous drug response curves at various concentration levels. However, these frameworks assume that each subpopulation can only generate its own subtype, thereby limiting their ability to capture the differentiation and de-differentiation dynamics of CSCs and CNSCs.

In this study, we have developed a novel statistical framework that integrates the multi-type branching process with a drug response model to simulate heterogeneous cell growth dynamics under drug influence. This framework serves as a mathematical tool for investigating drug-induced plasticity between CNSCs and CSCs directly from HTS bulk cell count data. A recent study²² identified drug-induced persistence in two colorectal cancer cell lines using total cell count data, based on simple alternative models of drug-induced persistence. Here, we develop a more general stochastic framework which has wider applicability across different potential forms of drug-induced resistance and incorporates the variability in the population dynamics to provide more robust inference. Given our focus on drug-induced plasticity between CNSCs and CSCs and the inherent resistance of CSCs, we will use the term drug-induced plasticity and the term drug-induced resistance interchangeably.

The paper is organized as follows. In the “Results” section, we present our newly proposed statistical model and validate its performance using both in silico and in vitro datasets. The in vitro datasets, sourced from⁷ and²³, demonstrate that our framework can identify drug-induced resistance using only HTS bulk cell count data. The “Discussion” section explores the advantages and limitations of our approach and outlines potential directions for future research. Finally, the “Methods” section provides a comprehensive derivation and implementation of our statistical framework.

Results

In this section, we present validation results for our newly proposed statistical framework. We provide a concise overview of the framework here, with a more detailed discussion on implementation available in Section “Methods”.

Statistical framework based on asymmetrical birth multi-type branching process

We propose a statistical modeling framework built on an asymmetrical birth multi-type branching process²⁴, which captures the underlying cell division dynamics across K subpopulations (See Section “Asymmetrical birth multi-type branching process model” for details). In this model, each cell undergoes one of three possible stochastic events, each governed by its own rate (illustrated in Fig. 1).

Fig. 1. Three distinct events in the asymmetrical birth multi-type branching process.

Each type 1 cell can symmetrically divide into two type 1 cell at rate α₁, die at rate β₁, and asymmetrically divide into one type 1 cell and one type 2 cell at rate ν₁₂.

To incorporate drug effects on cell behavior, we use the Hill equation, a well-known model in biochemistry for describing dose-response relationships. The Hill function is parameterized by b, E, and m, and is defined as:

$$H\left(d;b,E\right)=b+\frac{1-b}{1+{\left(d/E\right)}^m},$$

where d represents the drug concentration. In our framework, we use a simplified form of this model by fixing the Hill coefficient m = 1. We use this formulation to model both cytotoxic effects and drug-induced plasticity by applying separate Hill functions to the corresponding rates. Further details on how drug effects are incorporated into the branching process model are provided in Section “Drug-effect Model”.

By combining the branching process with the drug-response model, we derive a statistical model for HTS bulk cell count dataset with N_R replicates:

$$\left\{{\mathbf{x}}_{\mathcal{T},d,r};d\in \mathcal{D}=\left\{d_1,\cdots ,d_{N_d}\right\},r=1,\cdots ,N_R\right\},$$

where $\mathcal{T}$ denotes the set of time points and $\mathcal{D}$ the set of drug concentrations. The statistical model is given by:

$${\mathbf{x}}_{\mathcal{T},d,r}\approx \mu \left(\mathcal{T},d;\theta \right)+N^{\left(r\right)}\left(\mathbf{0},\mathbf{V}\left(\mathcal{T},d;\theta \right)\right)+N^{\left(r\right)}\left(\mathbf{0},c^2\mathbf{I}\right),$$

where $\mu \left(\mathcal{T},d;\theta \right)$ and $\mathbf{V}\left(\mathcal{T},d;\theta \right)$ denote the mean and covariance derived from a central limit theorem approximation of the branching process. The term N^(r)(0, V) represents the r-th independent and identically distributed (i.i.d.) copy of a multivariate normal distribution, and the final term accounts for i.i.d. observation noise at each time point, with variance c².

In the subsequent ‘Results’ section, we apply maximum likelihood estimation to infer both the model parameters θ and the noise standard deviation c using both in silico and in vitro datasets. Section “Statistical Model” provides additional modeling details, including a simpler variant based on a law of large numbers (LLN) approximation.

Deconvolution of cellular dynamics and drug effects from in silico datasets

To evaluate our framework’s performance in analyzing in silico data, we utilized the Gillespie algorithm²⁵ to generate computer-simulated data based on the asymmetrical birth multi-type branching process model. Each simulation started with an initial total cell count of n = 1000, and we conducted N_R = 20 replicates across various concentration levels $\mathcal{D}$ and time points $\mathcal{T}$, where

$$\mathcal{D}=\left\{0,0.0313,0.0625,0.125,0.25,0.375,0.5,1.25,2.5,3.75,5\right\},$$

$$\mathcal{T}=\left\{0,3,6,9,12,15,18,21,24,27,30,33,36\right\}.$$

Finally, Gaussian observation noise was added to obtain the in silico dataset.

In these in silico experiments, we derive two types of estimation results: 1. point estimation (PE) obtained for each experiment through the MLE process, 2. confidence intervals (CIs) obtained using the bootstrapping technique, specifically for selected in silico experiments. We refer to Supplementary Note 5 for further details of the computational implementation of in silico experiments.

Given our focus on a specific case involving two subpopulations, as described in Section “Simplified model of drug effect on CSCs and CNSCs mixture”, we fix K = 2. It is worth noting that one can treat K as a variable and employ model selection techniques to estimate the number of subpopulations exhibiting distinct phenotypic drug responses. Similar approaches were explored in²⁰ using a simpler statistical framework, but a detailed exploration of this topic is outside the scope of the current study.

Illustrative example

We begin with a single illustrative example. Table 1 shows the true parameter set used to generate the in silico dataset and the corresponding PEs obtained using our statistical framework. In Fig. 2, we display both the PEs and CIs for this example, focusing on two key metrics: the stable proportionπ(d) between phenotypes under each concentration level d, and the GR₅₀ dose for the CNSC growth rate (cytotoxic effect) and de-differentiation rate (plasticity effect), respectively. The GR₅₀ dose is the concentration level at which the drug achieves half its maximal observed effect on the growth rate²⁶. It is a useful summary metric of the drug effect which incorporates both primary drug effect parameters E and b, as is further discussed in Supplementary Note 5.

Table 1.

True parameter set θ^* and point estimates $\widehat{\mathbf{\theta }}$ in Fig. 2

	αr	βr	νrs	αs	βs	bs,β	Es,β	bs,ν	Es,ν
θ*	0.5407	0.5055	0.2929	0.2280	0.2280	0.8536	0.7073	1.0827	1.2285
$\widehat{\mathbf{\theta }}$	0.5407	0.5036	0.2988	0.2412	0.2419	0.8612	0.6468	1.0773	1.2336

Fig. 2. Estimation of stable proportion and GR₅₀ based on the in silico data generated by the true parameter set from Table 1.

The pie chart illustrates the PE of the stable proportion between CSCs and CNSCs under a no-drug environment. The upper-right plot shows 100 bootstrapped PEs as scatter points and the corresponding 90% CIs for the stable proportion of CSCs at each drug concentration level, with the red line representing the true stable proportion of CSCs in the same plot. The boxplots illustrate the CIs of GR₅₀ of both the drug-induced plasticity effect and cytotoxic effect, with the dashed colored lines representing the corresponding true GR₅₀ values. Specifically, the red dashed line and red boxplot correspond to cytotoxic effect, while the blue dashed line and blue boxplot correspond to drug-induced plasticity effect.

Figure 2 illustrates that our newly proposed framework accurately estimates the stable proportion in the absence of drug (pie charts in top panel). Additionally, the varying stable proportions of CSCs at different drug concentrations are captured by the CIs constructed from the bootstrapping estimations (line plot in top panel). The CIs also encompass the correct GR₅₀ values for both the cytotoxic and plasticity effects (bottom panel).

In this example, the true GR₅₀ values for the two drug effects fall between two concentration levels applied in the experiment. In our previous work²¹, we used a similar statistical framework to investigate a tumor with two subpopulations, where each was affected by an anti-cancer drug to a varying degree and there were no transitions between the subpopulations. There, we observed that close GR₅₀ values for the cytotoxic effects on each subpopulation often resulted in poor estimations of the GR₅₀ values. In the current study, where we consider two distinct types of drug effects, the difference in GR₅₀ is no longer critical for distinguishing these effects.

Estimation across a wide range of true parameter sets

To assess estimation performance across a wide range of biologically realistic parameter sets, we examined the relative errors (RE) from 100 datasets, each generated using a different true parameter set. The RE between an estimator $\widehat{x}$ and the true value x^* was calculated as

$$Er\left(\widehat{x};x^{*}\right)=∣\frac{\widehat{x}-x^{*}}{x^{*}}∣.$$ 1

The results for all datasets are summarized using boxplots in Fig. 3. Note that we recorded the RE for 1 − b_s,β and b_s,ν − 1 rather than b_s,β and b_s,ν directly. These new measurements of REs ensure symmetry when measuring the maximum drug effect between cytotoxic and plasticity effects, since b_s,β ∈ (0.5, 1) and b_s,ν ∈ (1, 1.5). Additionally, we included a horizontal line at RE = 0.2 as a threshold to indicate reasonably well estimated parameters.

Fig. 3. Relative error of the parameter estimation.

The boxplot represents 100 independent experiments, each based on randomly selected true parameter sets. The gray horizontal line is a threshold when RE is equal to 0.2.

Figure 3 shows that most point estimates are reasonable, with over three fourths of estimates having a RE below the specified threshold. This indicates that our newly proposed framework can successfully disentangle the cell growth dynamics and drug effect dynamics. We note that the plasticity effect parameter b_s,ν is generally estimated less accurately than the cytotoxic effect parameter b_s,β. This may stem from the fact that in all parameter sets, the true plasticity effect is smaller than the cytotoxic effect, i.e. b_s,ν − 1 < 1 − b_s,β, making it more challenging to detect the plasticity effect. In addition, we note that the relative error metric is more sensitive when the true value is small, since even small absolute errors $\mid \widehat{x}-x^{*}\mid$ can lead to large relative errors $\mid \left(\widehat{x}-x^{*}\right)/x^{*}\mid$.

Although our estimation framework performs well overall, there are some examples where it fails to recover the true values from the data. One explanation for low quality in the estimation is a poor experimental design, i.e., poor selection of concentration levels and data collection time points, resulting in reduced parameter identifiability. For instance, when estimating E, if the tested concentration levels do not cover the true value, i.e. $\max \left(\mathcal{D}\right)<E$, it will be challenging for our framework to identify the true value. Even when E is within the tested concentration levels, observation noise and the stochastic nature of the data generation can still distort the estimation, particularly when E is close to the maximum tested concentration level. A detailed analysis of how experimental design impacts parameter identifiability will be explored in future work.

Analysis of failed estimations

As previously mentioned, one potential explanation for poor estimation under our framework is a poor experimental design. To better understand potential challenges in parameter identifiability, we analyzed the true parameter sets and the simulated data from the 100 experiments described earlier. We specifically examined the drug effects on two theoretical metrics of long-run behavior: stable proportion (π(d)) and long-run growth rate (λ₁(d)) (see Section “Long-run behavior”). For each metric, we computed its theoretical values across different concentration levels and determined the maximum discrepancy among them to quantify the ranges of drug effects observed in the experiments.

In Fig. 4, scatter plots illustrate these quantities plotted against the average RE over all estimated parameters for each experiment. In the right panel’s lower right corner, four examples show a maximum change in stable proportion below 0.1, coinciding with a notably high average estimation error. In the left panel, these same four examples exhibit low maximum changes in long-run growth rates. This observation indicates that inaccurate estimations tend to occur when observed drug effects are minimal. This insight is further supported by Fig. 5, where we visualized two example datasets with high vs. low RE, and observed that the dataset with less variation in growth pattern across different concentration levels has higher average RE. To summarize, poor estimation in terms of RE can in most cases be traced to a small observed drug effect, which can either arise due to a poor experimental design (drug effects do not manifest due to poor selection of concentration levels or data collection points) or a small true drug effect, in which case the RE metric is more sensitive to estimation inaccuracy as mentioned in the previous section.

Fig. 4. Analysis of failed estimations.

The scatter plots are generated from 100 independent experiments, with the x-axis representing the average relative error in parameter estimation for both plots. In the left panel, the y-axis represents the observable drug effect on the long-term growth rate of the CSCs and CNSCs mixture. In the right panel, the y-axis represents the observable drug effect on the stable proportion between the CSCs and CNSCs.

Fig. 5. Visualization of the simulated data with varying information quality for estimation accuracy.

The left panel demonstrates data with insufficient information for accurate estimation, while the right panel demonstrates data with sufficient information. Each line plot illustrates total cell count data across 13 time points under a specific concentration level described in the legend. The error bar is derived from the standard deviation of 20 independent replicates.

Comparison between estimate from CLT Model 9 and LLN Model 11

In Section “Statistical Model”, we introduced a simplified LLN Model (11), derived from a law of large numbers type approximation, which disregards the covariance structure in the CLT Model (9). To evaluate the estimation performance of these two models, we compared their accuracy (measured by RE) using the same 100 simulated datasets that generated the results in Fig. 3.

It is worth noting that previous work indicates that the LLN model, which is based on a deterministic model of the population dynamics, is not able to distinguish between the cell symmetrical division rates and cell death rates¹¹. Due to this, we decided to compare estimation of the net growth rates, κ, as well as the symmetrical division rate α, for both cell types. For simplicity, we additionally focused on comparing the estimation of the GR₅₀ values for the drug effects on the sensitive subpopulation. These GR₅₀ values are denoted as GR_s,β and GR_s,ν, representing the cytotoxic and plasticity effects, respectively.

Figure 6 illustrates the comparison results between the CLT Model (represented by the blue box) and the LLN Model (red box). Based on the Wilcoxon rank-sum test, the CLT Model consistently demonstrates significantly higher estimation accuracy across all compared parameters. Moreover, the LLN Model fails to estimate the symmetrical cell division rate, α, for both subpopulations. As noted in^21,27, the variance structure inherent in stochastic models provides valuable insight into underlying cell dynamics. By effectively harnessing the structured variability arising from the stochastic nature of cell division and apoptosis, the CLT Model achieves markedly improved inference accuracy.

Fig. 6. Relative error of the parameter estimation from CLT Model (9) and LLN Model (11).

The boxplot contains 100 independent experiments based on the same datasets in Fig. 3. The gray horizontal line is a threshold when RE is equal to 0.2. The significance bar indicates the p-values derived from the Wilcoxon rank-sum test, with significance levels denoted as ***≤0.001≤**≤0.01≤*≤0.05.

Testing robustness of the framework using in silico datasets

To assess the robustness of our newly proposed statistical framework using in silico data, we designed a series of experiments to evaluate the performance of the framework when relaxing some of the assumptions in Section “Simplified model of drug effect on CSCs and CNSCs mixture”. Detailed experimental settings are described in Supplementary Note 5.

Relaxed drug effect assumption

In Assumption 4 of Section “Simplified model of drug effect on CSCs and CNSCs mixture”, we posit that the drug will not affect the CSCs. However, this assertion may seem too stringent for practical scenarios, where the varying microenvironment caused by different drug concentration levels could potentially affect the CSCs²⁸. Therefore, we conduct another set of experiments to examine the performance of our framework when the drug does influence the CSCs. While the mechanisms underlying drug resistance in CSCs are not yet fully understood^29,30, for testing purposes, we make the simplifying assumption that the drug affects CSCs in a similar way to CNSCs, manifesting through cytotoxic effects and increased asymmetrical divisions, albeit to a lesser extent than for CNSCs. As noted in Section “Drug-effect Model”, our framework is also adaptable to scenarios where the drug might decrease the differentiation rate.

Figure 7 presents the RE from 100 independent experiments. We observe that the estimates of certain parameters, such as (ν_rs, b_s,ν, E_s,ν), show deterioration compared to Fig. 3, with more than 50% of estimates having RE above the 0.2 threshold value. However, our framework continues to provide reasonably accurate estimates for parameters like α_s, β_s, b_s,β.

Fig. 7. Relative error of the parameter estimation when relaxing Assumption 4.

The boxplot contains 100 independent experiments generated by randomly selected true parameter sets. The gray horizontal line is a threshold when RE is equal to 0.2.

As mentioned in the previous section, the estimation accuracy may be correlated to the significance of the drug effect. In the current experiment, we find more compelling evidence to support this observation. In particular, the estimation accuracy for the CSC drug effect parameters (b_r,β, E_r,β, b_r,ν, E_r,ν) is significantly lower than the accuracy in estimating the analogous parameters for CNSCs, due to our assumption of a more pronounced impact of the drug on CNSCs.

Relaxed initial proportion assumption

Our next experiment aims to investigate the possibility of estimating the initial subpopulation structure under arbitrary initial proportions, relaxing Assumption 5 of Section “Simplified model of drug effect on CSCs and CNSCs mixture”. To accommodate this situation, we reintroduce two parameters, p_r and p_s, with the constraint that p_r + p_s = 1, representing the initial proportions of CSCs and CNSCs, respectively.

The results in Fig. 8 indicate that relaxing the initial proportion assumption does not deteriorate the accuracy of the estimation of other parameters, and the initial proportion is estimated reasonably well. This highlights the potential of our framework to deconvolute the phenotypic subpopulation structure with an unknown initial proportion. It is also worth noting that we allow the p_r and p_s to be uneven, for example, p_r = 0.03 and p_s = 0.97.

Fig. 8. Relative error of the parameter estimation when relaxing Assumption 4.

The boxplot contains 100 independent experiments generated by randomly selected true parameter sets. The gray horizontal line is a threshold when RE is equal to 0.2.

Limited division CNSC dynamics assumption

We finally consider a more general CNSC division scheme, where CNSCs can only undergo symmetric division a limited number of times, reflecting a gradual loss of proliferation potential. We refer to CNSCs produced through asymmetric division of CSCs as first-generation CNSCs, which can only divide up to G times. For the first G generations of CNSCs, we assume that the g-th generation can symmetrically divide into two (g + 1)-th generation CNSCs, asymmetrically divide into one CSC and one g-th generation CNSC, or die. The (G + 1)-th generation CNSCs can only die or asymmetrically divide into one CSC and one (G + 1)-th generation CNSC. All CNSCs share the same symmetric division rate α_s, death rate β_s and asymmetric division rate ν_sr (Fig. 9). In addition, the drug effect parameters b_s,β, E_s,β, b_s,ν, E_s,ν are the same for CNSCs in the different generations. This scheme encompasses a broader spectrum of CNSC dynamics, ranging from the scenario where CNSCs cannot self-renew (G = 0) to the scenario where CNSCs have an infinite capacity for self-renewal with a certain rate α_s (G → ∞). Theoretically, our current framework can accurately capture the extreme scenarios where G = 0 or G → ∞. We are interested in the practical performance of our framework in the intermediate case, i.e., 0 < G < ∞.

Fig. 9. Four different types of cellular division under the assumption of limited CNSC division.

Square shapes denote CSC and round shapes represent CNSC. Different colors indicate variations in division potential across successive CNSC generations. a A CSC symmetrically divides into two CSCs. b A CSC asymmetrically divides into one CSC and one first generation CNSC. c A CNSC of generation g symmetrically divides into two (g+1)-th generation CNSCs. d A CNSC of generation g asymmetrically divides into one CSC and one g-th generation CNSC.

We first want to determine whether our framework can reliably estimate the symmetric division rate of CNSCs, α_s. To streamline the experiment, we selected the same true value ${\alpha }_s^{*}>0$ for G ∈ {0, 1, 2, 3}. Instead of using the RE, we adopt the ratio between the estimated value and the true value of α_s as the metric of accuracy. Specifically, for an estimator $\widehat{x}$ and the true value x^*, we compute the accuracy metric as:

$$Er\left(\widehat{x};x^{*}\right)=\frac{\widehat{x}}{x^{*}}.$$ 2

This new metric effectively captures two important scenarios: (i) the framework should produce an estimation ${\widehat{\alpha }}_s=0$ when G = 0, resulting in a ratio ${\widehat{\alpha }}_s/{\alpha }_s^{*}=0$, and (ii) the framework should accurately estimate ${\widehat{\alpha }}_s={\alpha }_s^{*}$ when G is sufficiently large.

The results of our experiment (Fig. 10) align well with these two scenarios. Specifically, our framework accurately infers that α_s = 0 when the maximum division number is G = 0, indicating that CNSCs cannot undergo self-renewal. However, when G = 1, 75% of estimates for α_s are below the true value. Surprisingly, our framework accurately recovers the true value of α_s when G = 2 and G = 3. For other parameters, we generally observed systematic biases in the estimation, as detailed in Supplementary Note 5. However, interestingly, we found that the drug effects’ inflection points, E_s,ν and E_s,β, were accurately estimated across different values of G. This observation demonstrates the robustness of our framework in estimating the drug effects’ inflection points under varying differentiation dynamics. In other words, through detecting the changes in cell population growth dynamics, our framework can estimate the concentration levels at which such changes occur, even though it may not fully explain these changes. For the other parameters, we note that it is possible to fully capture the dynamics of CNSCs with limited proliferation potential by expanding our model beyond two subpopulations, as we further discuss in the conclusion section (Section “Discussion”).

Fig. 10. Estimation error of the CNSCs division rate under the assumption of limited division.

The ratios between the estimated division rate and the true division rate are presented to varying maximum division numbers G. The boxplot is based on 30 independent experiments for each G ∈ {0, 1, 2, 3} with varying true parameter sets. The gray horizontal line illustrates the ratio when CNSCs division rate is accurately estimated.

Using model selection criteria to detect drug-induced resistance

One potential application of our newly proposed statistical framework in practical settings is to detect the presence of drug-induced plasticity using model selection criteria. Here, we employ the well-known Akaike Information Criterion (AIC) calculated by the formula:

$$\text{AIC}=2\mid \theta \mid -2\log \left(L^{*}\right),$$ 3

where ∣θ∣ is the number of free parameters in the model and L^* is the maximum value of the likelihood function for the model. AIC, rooted in information theory, quantifies the information lost by a model, with lower AIC values indicating higher model quality.

In this section, we present an in silico example illustrating the use of AIC to detect drug-induced resistance. Specifically, we analyze in silico datasets generated using the true parameter set ${\theta }_{+}^{*}$ and ${\theta }_{-}^{*}$ provided in Table 2. The experiment based on ${\theta }_{+}^{*}$ assumes the presence of drug-induced plasticity, whereas the experiment using ${\theta }_{-}^{*}$ does not, allowing us to verify a false positive inference. In both cases, we assume that the experiment begins with only the sensitive subpopulation, p_s = 1, and that the sensitive subpopulation cannot transition to the resistant subpopulation spontaneously, ν_sr = 0. Unlike the illustrative example in Section “Deconvolution of cellular dynamics and drug effects from in silicon datasets”, we also account for the drug’s cytotoxic effect on the resistant subpopulation. Detailed descriptions of the data generation and AIC computation processes are presented in Supplementary Note 5.

Table 2.

True parameter sets ${\mathbf{\theta }}_{+}^{*}$ and ${\mathbf{\theta }}_{-}^{*}$ in AIC illustrative example

	pr	αr	βr	νrs	br,β	Er,β	ps	αs	βs	νsr	bs,β	Es,β	bs,ν	Es,ν	c
${\mathbf{\theta }}_{+}^{*}$	0	0.54	0.52	0.004	0.87	0.96	1	0.16	0.14	0	0.84	1.04	1.06	0.34	8.7
${\mathbf{\theta }}_{-}^{*}$	0	0.55	0.47	0.006	0.85	1.86	1	0.43	0.38	0	0.83	0.54	1	0.5	3.08

For these in silico datasets, we evaluate four distinct model assumptions, two based on the CLT-type approximation (9) and two based on the LLN-type approximation (11). Within each approximation type, we consider one model assumption that includes drug-induced plasticity and one that excludes it, referred to as CLT_DIP(LLN_DIP) and CLT_nDIP(LLN_nDIP), respectively. The free parameters estimated from the dataset for each model are summarized in Table 3. Notably, the LLN-type approximation cannot disentangle the cell symmetrical division rate from the cell death rate (see Section “Deconvolution of cellular dynamics and drug effects from in silicon datasets”); thus, we estimate only the net growth rate κ for each subpopulation. Additionally, when assuming no drug-induced plasticity, only the sensitive subpopulation is present because the experiment begins with an entirely sensitive population.

Table 3.

Free parameters in the AIC illustrative example

Model	∣θ∣	Free parameters
CLTDIP	12	αr, βr, νrs, br,β, Er,β, αs, βs, bs,β, Es,β, bs,ν, Es,ν, c
CLTnDIP	5	αs, βs, bs,β, Es,β, c
LLNDIP	10	κr, νrs, br,β, Er,β, κs, bs,β, Es,β, bs,ν, Es,ν, c
LLNnDIP	4	κs, bs,β, Es,β, c

The AIC values are presented in Table 4 and 5. As expected, the models incorporating drug-induced plasticity are generally preferred when such effects are present. Conversely, in the absence of drug-induced plasticity, these models are less favored, as indicated by their higher AIC values. Furthermore, the CLT-type approximation models exhibit superior estimation quality compared to the LLN-type approximation models. This illustrative example demonstrates that our newly proposed model can identify drug-induced plasticity using AIC model comparison criteria.

Table 4.

AIC value for 4 varying model assumptions in AIC illustrative experiment using ${\mathbf{\theta }}_{+}^{*}$, where drug-induced plasticity is present

Experiment using ${\mathbf{\theta }}_{+}^{*}$	Drug-induced plasticity	no Drug-induced plasticity
CLT-type approximation	AIC(CLTDIP): 26575	AIC(CLTnDIP): 26735
LLN-type approximation	AIC(LLNDIP): 31850	AIC(LLNnDIP): 32201

Table 5.

AIC value for 4 varying model assumptions in AIC illustrative experiment using ${\mathbf{\theta }}_{-}^{*}$, where drug-induced plasticity is absent

Experiment using ${\mathbf{\theta }}_{-}^{*}$	Drug-induced plasticity	no Drug-induced plasticity
CLT-type approximation	AIC(CLTDIP): 26467	AIC(CLTnDIP): 26460
LLN-type approximation	AIC(LLNDIP): 33798	AIC(LLNnDIP): 33797

Detecting CPX-O induced plasticity in gastric carcinoma cell lines

Data overview

To validate our framework using in vitro data, we utilized data from a recent study⁷ that investigated four human gastric carcinoma cell lines (AGS SORE6 −, AGS SORE6+, Kato III SORE6 −, Kato III SORE6+). Our focus in this section is specifically on the AGS SORE6 − and AGS SORE6+ cell lines. SORE6 serves as a reporter system indicating the stemness of these cancerous gastric cells, where SORE6 − and SORE6+ denote gastric CNSCs and CSCs, respectively. In the study⁷, ciclopirox olamine (CPX-O) was found to reprogram gastric CNSCs into CSCs. To support these findings, the authors cultivated pure AGS SORE6 − cells and monitored both cell viability data and the percentage of AGS SORE6+ cells using the SORE6 reporter system.

In our study, we mainly utilized two experimental datasets provided in⁷:

1. AGS Conc (X_Conc): AGS SORE6 − samples were cultivated under nine concentration levels of CPX-O:

   $${\mathcal{D}}_1=\left\{0,0.125,0.25,0.5,1,2,4,8,16\right\}\left(\mu M\right).$$

   Total cell count data (TC) and stem cell proportion (SC) were observed after 48 h of treatment, i.e. ${\mathcal{T}}_1=\left\{48\right\}$ (hours).
2. AGS Time (X_Time): AGS SORE6 − samples were cultivated under three concentration levels of CPX-O:

   $${\mathcal{D}}_2=\left\{0,4,8\right\}\left(\mu M\right).$$

   Total cell count data (TC) and stem cell proportion (SC) were observed after 2, 6, 12, 24, and 48 h of treatment:

   $${\mathcal{T}}_2=\left\{2,6,12,24,48\right\}\left(\mathrm{hours}\right).$$

Unfortunately, neither of these experiments provides enough information to estimate all parameters under our framework. Specifically, three concentration levels in the AGS Time data are not sufficient to estimate the four parameter effects of the drug in our framework, and a single time point in the AGS Conc data is inadequate to capture the heterogeneous dynamics of cell growth. This limitation may arise from the need to employ the FACS technique to distinguish between the CSC and CNSC populations. To overcome this limitation, we propose to employ a data imputation on the AGS Conc data based on the information from the AGS Time data. Further details of imputation can be found in Supplementary Note 6. We present the imputed dataset in Table 6 and Fig. 11.

Table 6.

In vitro total cell count (TC) data: AGS Conc data with augmented estimation according to AGS Time

The colored data are estimated data according to the AGS Time data, while the black data are actual data from⁷.

Fig. 11. Imputed AGS cancerous gastric bulk cell count data treated under nine different concentration levels of CPX-O.

Color-coded lines distinguish data across different dosages, connecting mean data points of two replicates. Blue crosses and black circles represent data used in model inference, with blue indicating imputed values and black showing actual AGS Conc data from⁷.

Candidate model description and AIC model selection results

We consider a total of four different models. The first two models, based on the framework described in Eq. (9), differ in their assumptions regarding the induction of plasticity by CPX-O. We denote the model that assumes drug-induced plasticity as model I and the alternative as model Ia. In fitting the in vitro data, we reintroduce the Hill parameter m to our drug-effect model (Section “Drug-effect Model”). Additionally, it is assumed that the drug does not affect the dynamics of CSCs.

Model I has 12 free parameters, as detailed in Table 7, to depict the cell growth dynamics of CSCs and CNSCs and the drug response of CNSCs. In contrast, Model Ia requires only 6 free parameters, as it assumes no drug-induced plasticity for CPX-O, resulting in no parameters necessary for gastric CSCs. Indeed, since the in vitro experiments in question are started with isolated CNSC subpopulations (p_s = 1), and no natural plasticity of CNSCs is assumed (ν_sr = 0), no CSCs should emerge in the experiments according to the assumptions of Model Ia.

Table 7.

Free parameters in models for AGS–CPX-O experiment

Model	∣θ∣	Free parameters
I	12	αr, βr, νrs, αs, βs, bs,β, Es,β, ms,β, bs,ν, Es,ν, ms,ν, c
Ia	6	αs, βs, bs,β, Es,β, ms,β, c
II	14	αr, βr, νrs, αs, βs, bs,β, Es,β, ms,β, bs,ν, Es,ν, ms,ν, k, t0, c
IIa	8	αs, βs, bs,β, Es,β, ms,β, k, t0, c

In the other two models, we introduce a time-delayed drug-effect which is present in the AGS Time data, since similar cell growth patterns are observed within the first 12 h across different drug concentration levels, with pronounced variations emerging thereafter. Further details can be found in Supplementary Note 6. Therefore, we assume that the maximum drug effect parameter b varies over time following a two-parameter logistic function as described in Section “Drug-effect Model”. Like the previous two models, one of these time-delayed drug-effect models, denoted as II, assumes that CPX-O can induce plasticity, while the other, denoted as IIa, does not. Model II has 14 free parameters, whereas Model IIa involves 8 free parameters, as detailed in Table 7.

The AIC results, summarized in Table 8, indicate a preference for the models assuming drug-induced plasticity under the AIC model selection criteria. Furthermore, comparing the AIC values between models with and without the time-delay assumption reveals the significant role of the time-delay effect in the current data. While our newly proposed framework may not fully capture the experimental data in⁷, it nonetheless yields conclusions consistent with the actual observations reported in⁷. It should also be stressed that our framework is able to infer the presence of drug-induced resistance from total cell count data, whereas this conclusion is drawn in⁷ using additional data on the CSC proportion over time. Of course, the inference of drug-induced plasticity is in this case facilitated by the ability to start the experiments from isolated CNSCs, but our framework is applicable to a broader range of experimental setups without this ability.

Table 8.

AIC value for 4 varying model assumptions in AGS–CPX-O experiment

	Drug-induced plasticity	no Drug-induced plasticity
Without time-delay effect	AIC(I): 653.4866	AIC(Ia): 677.9985
With time-delay effect	AIC(II): 543.7607	AIC(IIa): 576.5120

Detecting vemurafenib induced plasticity in COLO858 cell line

In³¹, the authors observed drug-induced de-differentiation of melanoma cells, leading to adaptive resistance. Specifically, they monitored the response of a melanoma cell line, COLO858, following exposure to the BRAF inhibitor Vemurafenib. Single-cell analysis and molecular profiling revealed an up-regulation of a de-differentiated NGFR^High state in Vemurafenib-treated cells. The authors also collected live-cell imaging bulk cell count data for the COLO858 cell line under various Vemurafenib concentration levels. This live-cell imaging data was provided in the follow-up work²³. We now wish to show that our framework can be used to detect drug-induced plasticity in this dataset. For consistency with previous definitions, we refer to the NGFR^High and NGFR^Low states of the COLO858 cell line as CSCs and CNSCs, respectively, for the remainder of this subsection.

Data overview

In²³, the COLO858 cell line was treated with six doses of Vemurafenib:

$${\mathcal{D}}_v=\left\{0\left(\text{DMSO}\right),0.032,0.1,0.32,1,3.2\right\}\left(\mu M\right),$$

over a period of 120 h. The full dataset is shown in Fig. 12.

Fig. 12. COLO858 bulk cell count data treated under six different concentration levels of Vemurafenib.

Errorbars, color-coded for each drug concentration, represent the mean and scaled standard deviation from four replicates of bulk cell count data from²³.

Similar to the AGS–CPX-O dataset, there is a time-delayed drug effect at the beginning of these experiments. Fortunately, we have already proposed a time-delayed drug response model in Section “Drug-effect Model”. It is worth reiterating that incorporating the time-delayed drug response model will complicate the computation of the mean matrix and the covariance matrix. Therefore, we decided to employ a simpler statistical approach, the LLN Model, for analyzing this dataset, as detailed in Section “Statistical Model”.

As is apparent from Fig. 12, the current dataset exhibits a logistic growth dynamic under DMSO conditions, which is not consistent with the exponential growth assumption of our framework. Therefore, we only use data from the first 60 h of the experiments, which includes a total of 30 time points at 2-h intervals, where exponential growth is a reasonable assumption.

Candidate model description and AIC model selection results

Similar to the AGS–CPX-O dataset, we propose using the AIC model selection criterion to determine whether the drug induces de-differentiation of the COLO858 cell line treated with Vemurafenib solely using live-cell imaging bulk cell count data. Before defining various models of interest, we begin with two model assumptions derived from³¹:

• Initially, the population consists entirely of melanoma CNSCs. (p_s = 1, p_r = 0).

• Vemurafenib may have cytotoxic effects on both CSCs and CNSCs; however, it promotes transitions only from CNSCs to CSCs. (b_r,ν = 1).

These assumptions closely align with those made in Section “Simplified model of drug effect on CSCs and CNSCs mixture”. However, we would like to emphasize that we now allow the drug to have a cytotoxic effect on the CSC population, aligning with the observation in³¹ that the COLO858 CSCs are less sensitive but still inhibited by the Vemurafenib.

We are fundamentally interested in determining whether the drug induces de-differentiation from CNSCs to CSCs, but since it is unknown whether there is natural plasticity between the two subpopulations or not, we will allow for both alternatives. This leads to four candidate models. The models DIP_nAsy and nDIP_nAsy assume no natural plasticity (ν_rs = ν_sr = 0), with the former model assuming the presence of drug-induced plasticity and the latter assuming no such drug effect. The models DIP_Asy and nDIP_Asy assume the possibility of natural plasticity, where the former model again assumes the presence of drug-induced plasticity. The free parameters for each model are provided in Table 9.

Table 9.

Free parameters in models for COLO858–Vemurafenib experiment

Model	∣θ∣	Free parameters
DIPnAsy	13	αr, βr, br,β, Er,β, αs, βs, bs,β, Es,β, bs,ν, Es,ν, k, t0, c
nDIPnAsy	8	αs, βs, bs,β, Es,β, ms,β, k, t0, c
DIPAsy	15	αr, βr, νrs, br,β, Er,β, αs, βs, νsr, bs,β, Es,β, bs,ν, Es,ν, k, t0, c
nDIPAsy	13	αr, βr, νrs, br,β, Er,β, αs, βs, νsr, bs,β, Es,β, k, t0, c

The AIC results are demonstrated in Table 10. See Supplementary Note 6 for the details of the computation. We conclude that models assuming drug-induced plasticity are generally preferred, regardless of whether natural transitions between the two subpopulations are assumed. These results show how our framework, using only data on the bulk cell population, is able to detect the presence of drug-induced resistance, which has been confirmed using single-cell analysis in³¹.

Table 10.

AIC value for 4 varying model assumptions in COLO858–Vemurafenib experiment

	Drug-induced plasticity	No drug-induced plasticity
Without natural plasticity	AIC(DIPnAsy): 6183	AIC(nDIPnAsy): 6443
With natural plasticity	AIC(DIPAsy): 6145	AIC(nDIPAsy): 6370

In a recent paper³², Sontag et al. employed a deterministic model to analyze the same COLO858–Vemurafenib dataset. Their estimation scheme assessed drug effects at different concentration levels with distinct parameters, estimating non-zero drug-induced plasticity. In contrast, our statistical framework employed the Hill equation to quantify the dose-response relationship and concluded the presence of drug-induced plasticity across the entire dataset using the AIC model selection criteria.

Data fitting

To evaluate the performance of the DIP_Asy model in fitting the data, we present the fitting results in Fig. 13, based on the parameters estimated from the model. Across the first five concentration levels, the model effectively captures the mean behavior of the data. However, under 3.2 μM of Vemurafenib, the fitting plot fails to represent the final plateauing behavior. Since we only consider data from the first 60 h to avoid violating the exponential growth assumption, the available data depicting this plateauing effect under 3.2 μM of Vemurafenib is insufficient to inform the model of this behavior.

Fig. 13. COLO858–Vemurafenib datasets fitting results.

The fitting subfigures display results for 0, 0.032, 0.1, 0.32, 1, 3.2 μM of Vemurafenib. Errorbars indicate the mean and the standard deviation of the data. The red solid line represents the fitted total cell count.

Discussion

In this study, we introduced a novel statistical framework designed for analyzing HTS bulk data involving multiple subpopulations of cells. Using an asymmetrical birth multi-type branching process, we extended a model introduced in our previous work²¹ to accommodate transitions between the different subpopulations, thereby enhancing its range of applications. One such application involves inferring drug-induced plasticity in a mixture population of cancer stem-like cells (CSCs) and cancer non-stem-like cells (CNSCs). The asymmetrical birth multi-type branching process naturally accounts for the differentiation of CSCs and the de-differentiation of CNSCs. Additionally, we incorporated the Hill equation to characterize the cytotoxic effects of anti-cancer drugs and drug-induced de-differentiation rates. We tested our approach using both in silico and in vitro data.

In our in silico experiments, we used stochastic simulation to generate datasets involving drug-treated mixtures of CSCs and CNSCs. The datasets consisted of total cell counts collected at predetermined drug concentration levels and time points. Drawing inspiration from recent research on drug-induced plasticity⁷, we formulated five assumptions to simulate the dynamics of CSCs and CNSCs, along with their respective drug effects. Under these assumptions, our newly proposed framework not only identifies the presence of drug-induced plasticity, but also accurately predicts several key features of the mixture dynamics. Specifically, in an illustrative example, the framework accurately recovered the drug-affected stable proportions between CSCs and CNSCs across various drug concentration levels. Furthermore, it determined the GR₅₀ values for both the drug’s cytotoxic effect on CNSCs and its induced plasticity effect. Through further numerical experiments, we confirmed that the newly proposed framework consistently provides precise estimations for each model parameter, capturing the growth dynamics of individual subpopulations, transitions between the subpopulations, as well as drug-induced toxicity and plasticity.

We further investigated the robustness of our framework by relaxing assumptions regarding the dynamics of the CSCs and CNSCs and how the drug affects each supbopulation. By allowing the CSC and CNSC mixture to start from an arbitrary initial proportion instead of the stable drug-free proportion, our framework accurately estimated not only the mixture’s growth dynamics and drug effects but also the initial proportion. However, relaxing the assumption that the drug only affects CNSCs resulted in a more complex drug effect profile, which degraded the identifiability of each individual effect. In particular, the drug effects on CSCs were less identifiable, which may be due to our assumption that the drug has a more significant effect on CNSCs. It should also be kept in mind that we only assume data on total cell counts, as opposed to cell counts for individual subpopulations, which is an inherent limitation for teasing apart complex evolutionary dynamics and drug effect profiles involving multiple parameters. Using this kind of data, some simplifying assumptions must be made to ensure the identifiability of all model parameters.

Additionally, we explored the performance of our framework under the assumption of limited division of CNSCs, portraying the differentiation process as a gradual loss of cell proliferation potential. Our framework successfully estimated the CNSC division rate when the maximum division number (G) of CNSCs equaled 0, 2 or 3. Furthermore, it consistently provided reliable estimates of the half-maximum effects parameter (E) for each drug effect on the CNSCs. Even though parameter estimates generally degraded, these results demonstrate that our newly proposed framework can offer valuable insights even under more complex and realistic scenarios. To fully capture these dynamics, we propose expanding our model beyond two subpopulations, by modeling each generation of CNSCs as a distinct subpopulation. Specifically, we can represent the symmetric division of g-th generation cells into two (g + 1)-th generation cells as birth events where a cell of one type produces two cells of another type. Currently, our asymmetric division framework does not account for this behavior, as it models subpopulation transitions through asymmetric divisions. However, modeling subpopulation transitions through symmetric division is straightforward using the multi-type branching process framework²⁴. We plan to explore this in future work.

We validated our framework using data from the gastric carcinoma cell line (AGS) treated with ciclopirox olamine (CPX-O)⁷, as well as BRAF-mutated melanoma cell line (COLO858) treated with Vemurafenib²³. Based on single-cell tracking analysis, these two studies confirm the presence of drug-induced plasticity in both datasets. Coupled with the AIC model selection scheme, our novel statistical framework is able to indicate the presence of drug-induced plasticity in both datasets using live-cell imaging bulk data alone. Validation using data from⁷ posed challenges due to the limited in vitro data available. This discrepancy is partly due to the experimental procedure described in⁷, which involves using FASC to separate CSCs and CNSCs—a step that our framework does not require. Nevertheless, our framework indicates that CPX-O induces plasticity in the AGS gastric carcinoma cell line, aligning with the conclusions in⁷. In the COLO858–Vemurafenib validation experiment, an abundance of time-course data over a period of 120 h is provided in²³. However, we observed a logistic type of proliferation dynamics, potentially caused by carrying capacity, which is not assumed in our model. Therefore, we chose to employ only the initial 60 h data, which demonstrated a clear exponential type of proliferation. Incorporating logistic proliferation dynamics is a potential future direction.

In general, our framework requires many parameters to model various drug effects, necessitating multiple data points at distinct concentration levels to adequately capture mixture dynamics under drug influence. However, this requirement may be offset by the advantage of not needing a more sophisticated technique for subpopulation separation. Moreover, the quality of the data is equally as important as the quantity of it. As observed in our in silico experiments, some true parameter sets became unidentifiable under the specific concentration levels used in the experiments. To enhance the framework’s utility, one future direction involves developing techniques for experimental design, meaning optimal selection of drug concentrations and time points for accurate parameter inference. Such analysis could markedly improve the framework’s ability to extract insights from HTS data.

Another challenge identified in the in vitro experiment was the time-inhomogeneous drug effect. While our drug-effect base model assumes time-homogeneous drug effects on the target tumor sample, data from^7,23 show significant heterogeneity in drug effects over time. In Section “Drug-effect Model”, we addressed this by developing a two-parameter logistic function, resulting in a better fit to the in vitro data.

There are several avenues for extension of the framework beyond those already mentioned. First, while we have focused on inferring drug-induced plasticity in a CSC and CNSC mixture, the newly proposed framework is based on a versatile mathematical model (multi-type branching process) which can be adapted to a broader range of transition dynamics between two or more subpopulations. These subpopulations are defined by their heterogeneous responses to controllable interferences – specifically, drugs in this case. Second, we have in this work focused on cellular responses to single-drug interventions. One potential extension involves expanding our framework to encompass more complex multi-drug responses, potentially unveiling more intricate underlying systems. Moreover, external factors such as nutrient levels, hypoxia, stromal content, and other microenvironmental factors could be integrated into our framework in future work.

Methods

In this section, we start by introducing a general model to describe heterogeneous tumor growth dynamics, capable of accommodating an arbitrary number of distinct phenotypes. Then, we develop a drug-effect model using the classical Hill equation, which is widely employed to represent dose-response relationships. By integrating these two models, we propose a novel statistical framework for inferring underlying tumor dynamics under the treatment. Importantly, our framework is capable of inferring the proliferation dynamics of distinct subpopulations and transitions between subpopulations using HTS total cell count data alone, without requiring subpopulation counts. Finally, we tailor this framework to a specific case involving two subpopulations within the tumor, CSCs and CNSCs, which will be the focus of our analysis in the remainder of the paper. In what follows, we denote vectors and matrices in boldface letters and all the vectors are row vectors.

Asymmetrical birth multi-type branching process model

Assume there are K subpopulations of phenotypes within a tumor, each exhibiting its own growth dynamics and drug response. These subpopulations are indexed by $\mathcal{K}=\left\{1,\cdots ,K\right\}$. Increasing evidence indicates that asymmetric cell division is a key mechanism for both differentiation^33,34 and de-differentiation³⁵. Therefore, we assume that each phenotype can transition to one or more other phenotypes via asymmetric cell division. Our framework can also readily accommodate phenotypic transitions that occur independently of cell divisions, as discussed for example in¹¹.

We employ an asymmetric birth multi-type branching process model in continuous time²⁴. In the model, a type-i cell divides into two type-i cells at rate α_i, experiences natural death at rate β_i, and asymmetrically divides into one type-i cell and one type-j cell at rate ν_ij. Specifically, within a given infinitesimal time interval Δt > 0, a type-i cell has a probability α_iΔt of dividing into two type-i cells, β_iΔt of dying, and ν_ijΔt of asymmetrically dividing into one type-i and one type-j cell. The net growth rate of type-i cells, κ_i, is defined as κ_i: = α_i − β_i. The dynamics of the model are encoded in the infinitesimal generator matrixA, which is the K × K matrix

$$\mathbf{A}:=\left[\begin{array}{cccc}\hfill {\kappa }_1\hfill & \hfill {\nu }_{12}\hfill & \hfill \cdots \hfill & \hfill {\nu }_{1K}\hfill \\ \hfill {\nu }_{21}\hfill & \hfill {\kappa }_2\hfill & \hfill \cdots \hfill & \hfill {\nu }_{2K}\hfill \\ \hfill ⋮\hfill & \hfill ⋮\hfill & \hfill \ddots \hfill & \hfill ⋮\hfill \\ \hfill {\nu }_{K1}\hfill & \hfill {\nu }_{K2}\hfill & \hfill \cdots \hfill & \hfill {\kappa }_K\hfill \end{array}\right].$$ 4

In this matrix, the (i, j)-th element is the net rate at which a type-i cell produces a type-j cell. For all K phenotypes, we assume a strictly positive birth rate (${\alpha }_i>0,i\in \mathcal{K}$), a non-negative death rate (${\beta }_i\ge 0,i\in \mathcal{K}$), and a non-negative asymmetric birth rate (ν_i,j≥0, i ≠ j). We furthermore assume that κ_i > 0 for all $i\in \mathcal{K}$, i.e. the net birth rate is positive, though transition and death may not occur for all phenotypes.

For simplicity, we assume that the multi-type branching process is irreducible, meaning that each subpopulation can eventually produce descendants in any other subpopulation, possibly through intermediate types. Mathematically, this means that the infinitesimal generator matrixA cannot be transformed into a block upper triangular matrix via simultaneous row or column permutation.

We denote the cell counts of each phenotype as a random vector: B(t) = [B₁(t), B₂(t), ⋯ , B_K(t)], where B_i(t) is the number of type-i cells at the time t. For the special case where the process is started by a single type-i cell, the cell count vector at time t is denoted by ${\mathbf{B}}^{\left(i\right)}\left(t\right)=\left[B_1^{\left(i\right)}\left(t\right),B_2^{\left(i\right)}\left(t\right),\cdots ,B_K^{\left(i\right)}\left(t\right)\right]$, where B⁽ⁱ⁾(0) = e_i is the i-th identity vector. The mean vector and covariance matrix for B⁽ⁱ⁾(t) are denoted by

$$\begin{array}{ccc}\hfill {\mathbf{m}}^{\left(i\right)}\left(t\right)& \hfill :=\hfill & \mathrm{E}\left[{\mathbf{B}}^{\left(i\right)}\left(t\right)\right],\hfill \\ \hfill {\mathbf{\Xi }}^{\left(i\right)}\left(t\right)& \hfill :=\hfill & \mathrm{E}\left[{\left({\mathbf{B}}^{\left(i\right)}\left(t\right)-{\mathbf{m}}^{\left(i\right)}\left(t\right)\right)}^{⊺}\left({\mathbf{B}}^{\left(i\right)}\left(t\right)-{\mathbf{m}}^{\left(i\right)}\left(t\right)\right)\right],\hfill \end{array}$$

where t ≥ 0 and ${\left({\mathbf{B}}^{\left(i\right)}\left(t\right)-{\mathbf{m}}^{\left(i\right)}\left(t\right)\right)}^{⊺}$ represents the transpose of the vector $\left({\mathbf{B}}^{\left(i\right)}\left(t\right)-{\mathbf{m}}^{\left(i\right)}\left(t\right)\right)$. To explicitly compute the mean vector, we define the mean matrix using the matrix exponential:

$$\mathbf{M}\left(t\right)=\exp \left(t\mathbf{A}\right)=\sum _{j=0}^{\infty }\left(\frac{t^j}{j!}\right){\mathbf{A}}^j.$$

The mean vectorm⁽ⁱ⁾(t) can be computed as the i-th row of the mean matrix,

$${\mathbf{m}}^{\left(i\right)}\left(t\right)={\mathbf{e}}_i\mathbf{M}\left(t\right),$$

and the covariance matrix Ξ⁽ⁱ⁾(t) can be computed as shown in Proposition 1 below.

Drug-effect Model

Before modeling the drug effect, we introduce a fixed drug dose d and denote the cell count at time t and concentration level d as B(t, d). The mean matrixM(t, d) and the covariance matrix Ξ⁽ⁱ⁾(t, d) follow. Note that we use concentration level and dose interchangeably.

To model the drug effect, we define the well-known Hill equation with parameters (b, E, m) as

$$H\left(d;b,E,m\right)=b+\frac{1-b}{1+{\left(d/E\right)}^m}.$$

The Hill equation is a classic sigmoidal function used to describe a dose-response³⁶. In the equation, the parameter b represents the maximum drug effect, since H(0; b, E, m) = 1 and $\underset{d\to \infty }{lim}H\left(d;b,E,m\right)=b.$ For b ∈ (0, 1), the Hill equation is strictly decreasing, while it is strictly increasing for b > 1 (Fig. 14). The parameter E indicates the concentration at which 50 percent of the maximum effect is attained, known as the inflection point. The last parameter m is the Hill coefficient, which controls the steepness of the Hill equation around the inflection point. As a simplification, we fix the Hill parameter m = 1 for all drug effects when conducting our in silico experiments. We then reintroduce this parameter when modeling in vitro data.

Fig. 14. Hill equation illustrations.

Parameter sets used to generate Hill equations are (a) b = 0.85, E = 0.125, m = 2 and b b = 1.15, E = 0.125, m = 2.

We initially assume the effects of a single drug are time-homogeneous. We distinguish between two types of drug responses, drug toxicity response and drug-induced plasticity response, thereby capturing multiple effects from a single drug.

• Drug toxicity response (cytotoxic effect): The parameters related to this response are denoted as (b_i,β, E_i,β) for type i. We assume that b_i,β ∈ (0, 1) and that the death rate of type-i under dose d is given by (for the case m = 1):

  $${\beta }_i\left(d\right)={\beta }_i-\log \left(H\left(d;b_{i,\beta },E_{i,\beta }\right)\right)={\beta }_i-\log \left(b_{i,\beta }+\frac{1-b_{i,\beta }}{1+\left(d/E_{i,\beta }\right)}\right).$$ 5

  The net growth rate of a type i cell, κ_i, is therefore negatively affected by increasing drug concentration, i.e. ${\kappa }_i\left(d\right)=\left({\alpha }_i-{\beta }_i\right)+\log \left(H\left(d;b_{i,\beta },E_{i,\beta }\right)\right)$. In this scenario, the drug is assumed to act through a cytotoxic mechanism, meaning that higher doses result in increased rates of cell death. We note that our framework can also easily incorporate cytostatic effects, where higher doses lead to a lower cell division rate, using a similar Hill function.
• Drug-induced plasticity response: The parameters related to this response are denoted as (b_i,ν, E_i,ν) for type-i. For simplicity, we assume here that the drug effect on type-i cell transitions is independent of the target phenotype j. Drug-induced phenotypic transition is a widely observed effect of cancer treatment^37–39. Particularly, drug-sensitive phenotypes may be induced to develop resistance when exposed to the drug. When modeling the drug effect on the asymmetric birth rate, we assume b_i,ν≥1 to account for elevated transitions between subpopulations caused by the therapeutic environment. We furthermore assume that the asymmetric birth rate of type i to any other type j under dose d is given by (for the case m = 1):

  $${\nu }_{ij}\left(d\right)={\nu }_{ij}+\log \left(H\left(d;b_{i,\nu },E_{i,\nu }\right)\right)={\nu }_{ij}+\log \left(b_{i,\nu }+\frac{1-b_{i,\nu }}{1+\left(d/E_{i,\nu }\right)}\right).$$ 6

  The assumption b_i,ν≥1 incorporates the situation where the drug does not impact phenotypic transitions, when b_i,ν = 1, ν_ij(d) = ν_ij for all d≥0. It is worth mentioning that our framework can also model drug inhibition of the asymmetric birth rate by letting b_i,ν ∈ (0, 1).

Logistic time-delayed drug response

Our framework can be extended to accommodate a more complex drug-effect model, which may involve time-inhomogeneous drug effects. In particular, for the in vitro dataset from⁷ and²³, there appear to be delayed drug effects, which have been observed in other studies as well^22,40,41. To address this behavior, we allow for the possibility of a time-inhomogeneous drug effect, which is modeled using a two-parameter logistic function. In this case, we assume that the maximum drug effect parameter b depends on time t through

$$\mid 1-b\left(t\right)\mid =\frac{1}{1+\exp \left(k\left(t_0-t\right)\right)}\mid 1-b\mid ,$$

where k, t₀ are logistic function parameters. In this way, we can model the drug’s maximum potential b(t) gradually reaching its ultimate value as time increases. Absolute values are taken to allow both the cytotoxic effect b(t) < 1 and the drug-induced plasticity effect b(t) > 1 to be time-inhomogeneous.

Notably, incorporating this time-dependent drug effect model introduces a time-dependent infinitesimal generator matrixA(t, d), complicating the computation of the mean matrix and the covariance matrix. Detailed implementation can be found in Section “Statistical Model”.

Long-run behavior

As the HTS experiments being considered are run for a longer time duration, it becomes important to understand the long-run behavior of the process B(t, d). According to the Perron-Frobenius Theorem, the irreducible infinitesimal generator matrixA(d) has a strictly positive largest eigenvalue λ₁(d), which corresponds to the largest eigenvalue $e^{{\lambda }_1\left(d\right)t}$ of the mean matrix M(t, d). Additionally, the corresponding left eigenvector π(d) is strictly positive. A well-known limiting result reviewed in²⁴ states that there exists, almost surely, a non-negative numerical random variable W such that

$$\underset{t\to \infty }{lim}\mathbf{B}\left(t,d\right)e^{-{\lambda }_1\left(d\right)t}=W\mathbf{\pi }\left(d\right).$$

This result characterizes two aspects of the long-run behavior of the process B(t, d). First, the stochastic process B(t, d) will grow with a deterministic exponential long-run growth rateλ₁(d). Second, when the eigenvector π(d) is normalized so that ${\sum }_{i=1}^K{\pi }_i\left(d\right)=1$, then π_i(d) describes the long-run proportion of type-i cells in the population. Therefore, we refer to the normalized version of π(d) as the stable proportion between phenotypes.

Statistical Model

Now, we construct a statistical framework to infer heterogeneous tumor growth dynamics and drug responses using HTS live-cell imaging bulk data. Here, ‘bulk data’ refers to aggregated total cell counts across all subpopulations within the heterogeneous tumor. Our framework is designed to accommodate data collected via live-cell imaging techniques, allowing efficient capture of bulk cell counts from a single sample across multiple time points⁴².

Assume the bulk dataset X is collected across a set of drug concentration levels $\mathcal{D}=\left\{d_1,\cdots ,d_{N_D}\right\}$ and a set of time points $\mathcal{T}=\left\{t_1,\cdots ,t_{N_T}\right\}$, where $0<t_1<\cdots <t_{N_T}$. For each drug dose $d\in \mathcal{D}$, N_R samples are cultivated, and live-cell imaging technique is used to collect bulk cell count data at time points $t_1,\dots ,t_{N_T}$. Let ${\mathbf{x}}_{\mathcal{T},d,r}=\left(x_{1,d,r},\dots ,x_{N_t,d,r}\right)$ denote the data collected for the r-th replicate under drug concentration d, where x_k,d,r is the bulk cell count at time point t_k. Taking into account experimental measurement error, we propose the statistical model

$${\mathbf{x}}_{\mathcal{T},d,r}={\mathbf{Y}}_{d,r}\left(\mathcal{T}\right)+{\mathbf{Z}}_{d,r},$$ 7

where ${\mathbf{Y}}_{d,r}\left(\mathcal{T}\right)$ denotes bulk cell counts at the time points $\mathcal{T}$ under the drug-affected evolution dynamics model outlined in Sections “Asymmetrical birth multi-type branching process model” and “Drug-effect Model”, and Z_d,r ~ N(0, c²I) are independent multivariate normally distributed noise terms. In other word, ${\mathbf{Y}}_{d,r}\left(\mathcal{T}\right)$ is a random vector with N_T elements, each obtained by summing the random vector B(t_i, d) for i = 1, ⋯ , N_T. Assuming that n_i is the starting number of cells of type i in each experiment, we can write

$${\mathbf{Y}}_{d,r}\left(\mathcal{T}\right):=\sum _{i=1}^K\sum _{j=1}^{n_i}{\mathbf{y}}_j^{\left(i\right)}\left(\mathcal{T},d,r\right),$$ 8

where ${\mathbf{y}}_j^{\left(i\right)}\left(\mathcal{T},d,r\right)$ denotes the size of the clone started by the j-th initial type i cell at the time points $\mathcal{T}$ under the stochastic model. Let n denote the initial total cell count and p = [p₁, p₂, ⋯ , p_K] denote the initial subpopulation distribution, with n_i = np_i for all i. In this study, we assume that p_i is independent of n for all i, implying that n_i → ∞ as n → ∞.

Central limit theorem (CLT)

For simplicity, we temporarily omit the explicit notation for drug concentration levels and experimental replicates and write $\mathbf{Y}\left(\mathcal{T}\right)$ and ${\mathbf{y}}_j^{\left(i\right)}\left(\mathcal{T}\right)$. When bulk cell counts are observed as opposed to individual subpopulation counts, the stochastic process Y(t) is no longer a Markov process. Therefore, the exact probability distribution for $\mathbf{Y}\left(\mathcal{T}\right)$ becomes overly complex²¹. Given that drug screening experiments are usually started by a relatively large number of cells (in⁴³ for example, 2500 cells were deployed in each well), it is natural to focus on the asymptotic behavior of $\mathbf{Y}\left(\mathcal{T}\right)$ as the initial cell number increases (n → ∞). For that, we need to understand the asymptotic behavior of ${\sum }_{j=1}^{n_i}{\mathbf{y}}_j^{\left(i\right)}\left(\mathcal{T}\right)$. We denote the expectation

$$\mathrm{E}\left[{\mathbf{y}}_j^{\left(i\right)}\left(\mathcal{T}\right)\right]:={\mathbf{\mu }}^{\left(i\right)}\left(\mathcal{T}\right)=\left[⟨{\mathbf{m}}^{\left(i\right)}\left(t_1\right),\mathbf{1}⟩,\cdots ,⟨{\mathbf{m}}^{\left(i\right)}\left(t_{N_T}\right),\mathbf{1}⟩\right],$$

where 1 is a size K all ones vector, 〈 ⋅ , ⋅ 〉 is the inner product, and m⁽ⁱ⁾(t) denotes the mean vector described in Section “Asymmetrical birth multi-type branching process model”. We furthermore define a centered and normalized process for each $i\in \mathcal{K}$ by

$${\mathbf{W}}_{n_i}^{\left(i\right)}\left(\mathcal{T}\right):=\frac{1}{\sqrt{n_i}}\sum _{j=1}^{n_i}\left({\mathbf{y}}_j^{\left(i\right)}\left(\mathcal{T}\right)-{\mu }^{\left(i\right)}\left(\mathcal{T}\right)\right).$$

A direct application of the multivariate central limit theorem gives the following result, where ‘ ⇒ ’ denotes converge in distribution.

Proposition 1

As n_i → ∞:

$${\mathbf{W}}_{n_i}^{\left(i\right)}\left(\mathcal{T}\right)\Rightarrow N\left(\mathbf{0},{\mathbf{V}}^{\left(i\right)}\left(\mathcal{T}\right)\right)$$

where

$${\mathbf{V}}_{a,b}^{\left(i\right)}\left(\mathcal{T}\right)=\mathbf{1}{\mathbf{\Xi }}^{\left(i\right)}\left(t_a\right)\mathbf{M}\left(t_b-t_a\right){\mathbf{1}}^{⊺}\mathrm{for}1\le a\le b\le N_T.$$

Here, M(t) is the mean matrix, and Ξ⁽ⁱ⁾(t_a) is a covariance matrix (Section “Asymmetrical birth multi-type branching process model”) given by

$$\begin{array}{cc}{\mathbf{\Xi }}^{\left(i\right)}\left(t\right)\hfill & =\mathrm{diag}\left({\mathbf{m}}^{\left(i\right)}\left(t\right)\right)-{\mathbf{m}}^{\left(i\right)}{\left(t\right)}^{⊺}{\mathbf{m}}^{\left(i\right)}\left(t\right)+{\int }_0^t{\left(\mathbf{M}\left(t-\tau \right)\right)}^{⊺}{\mathbf{C}}^{\left(i\right)}\left(\tau \right)\left(\mathbf{M}\left(t-\tau \right)\right)d\tau ,\hfill \end{array}$$

where

$${\mathbf{C}}_{jk}^{\left(i\right)}\left(\tau \right)=\left\{\begin{array}{cc}{\nu }_{kj}{\mathbf{M}}_{ik}\left(\tau \right)+{\nu }_{jk}{\mathbf{M}}_{ij}\left(\tau \right)\hfill & j\ne k\hfill \\ 2{\alpha }_j{\mathbf{M}}_{ij}\left(\tau \right)\hfill & j=k.\hfill \end{array}\right)$$

This result can be derived from a more general result, which is stated and proved in Supplementary Note 4. According to Proposition 1, we can approximate ${\sum }_{j=1}^{n_i}{\mathbf{y}}_j^{\left(i\right)}\left(\mathcal{T}\right)$ as

$$\sum _{j=1}^{n_i}{\mathbf{y}}_j^{\left(i\right)}\left(\mathcal{T}\right)\approx n_i{\mathbf{\mu }}^{\left(i\right)}\left(\mathcal{T}\right)+N\left(0,n_i{\mathbf{V}}^{\left(i\right)}\left(\mathcal{T}\right)\right)$$

for sufficiently large n_i. We can approximate $\mathbf{Y}\left(\mathcal{T}\right)$ accordingly. It is important to note that this approximation holds true for each fixed drug concentration level d.

Maximum likelihood estimation (MLE) framework

Now we reintroduce the drug concentration level $d\in \mathcal{D}$ and use the approximation

$${\mathbf{Y}}_{d,r}\left(\mathcal{T}\right)\approx \sum _{i=1}^K{n}_i{\mu }^{\left(i\right)}\left(\mathcal{T},d\right)+N^{\left(r\right)}\left(0,n_i{\mathbf{V}}^{\left(i\right)}\left(\mathcal{T},d\right)\right),$$

where $N^{\left(r\right)}\left(0,n_i{\mathbf{V}}^{\left(i\right)}\left(\mathcal{T},d\right)\right)$ represents independent and identical copies of a random vector with distribution $N\left(0,n_i{\mathbf{V}}^{\left(i\right)}\left(\mathcal{T},d\right)\right)$. We then formulate our newly proposed statistical model, CLT Model, as

$${\mathbf{x}}_{\mathcal{T},d,r}\approx \sum _{i=1}^K{n}_i{\mu }^{\left(i\right)}\left(\mathcal{T},d\right)+N^{\left(r\right)}\left(0,n_i{\mathbf{V}}^{\left(i\right)}\left(\mathcal{T},d\right)\right)+N^{\left(r\right)}\left(0,c^2\mathbf{I}\right),$$ 9

where the final term captures experimental measurement error as before. Based on this statistical model, we compute the likelihood function $L\left(\theta \left(\mathcal{K}\right)\mid \mathbf{X}\right)$, which represents the probability of obtaining the observed dataset X given the set $\theta \left(\mathcal{K}\right)$ of all parameters in the model:

$$\theta \left(\mathcal{K}\right)=\left\{{\left({\alpha }_i,{\beta }_i,{\left({\nu }_{ij}\right)}_{j\ne i},b_{i,\beta },E_{i,\beta },b_{i,\nu },E_{i,\nu }\right)}_{i\in \mathcal{K}},c\right\}.$$

We then apply the maximum likelihood estimation framework to obtain the parameter set that is most likely to explain the observed dataset. The estimated parameter set $\widehat{\theta }\left(\mathcal{K}\right)$ is computed by minimizing the negative log-likelihood, which is equivalent to maximizing the likelihood function:

$$\widehat{\theta }\left(\mathcal{K}\right)=\underset{\mathbf{\theta }\left(\mathcal{K}\right)\in \widehat{\mathbf{\Theta }}}{\arg \min }-\log \left(L\left(\theta \left(\mathcal{K}\right)\mid \mathbf{X}\right)\right),$$ 10

where $\widehat{\mathbf{\Theta }}$ is the set of feasible parameters. In Table 11, we provide an overview of notation.

Table 11.

Table of definitions in Section “Asymmetrical birth multi-type branching process model”, Section “Drug-effect Model”, Section “Long-run behavior”, and Section “Statistical Model”

Notation	Dimension	Description	Definition/Range
αi	1	Subtype i symmetric division rate	α > 0
βi(d)	1	Subtype i death rate	β(0)≥0
νij(d)	1	Subtype i asymmetric division rate to subtype j	ν(0)≥0
λ1(d)	1	Long-run growth rate	λ1(0) > 0
$\mathcal{K}$	1 × K	Subpopulation index set	$\mid \mathcal{K}\mid =K$
B(t, d)	1 × K	Vector of subpopulation counts	B(t, d)≥0
B(i)(t, d)	1 × K	B(t) generated from single type i cell	B(i)(t, d)≥0
m(i)(t, d)	1 × K	Expected value of B(i)(t, d)	m(i)(t, d)≥0
Ξ(i)(t, d)	K × K	Covariance matrix of B(i)(t, d)	det(Ξ(i)(t, d)) > 0
A(d)	K × K	infinitesimal generator matrix	Aii(d) = αi − βi(d), Aij = νij(d)
M(t, d)	K × K	Mean matrix	$\underset{t\to 0}{lim}\mathbf{M}\left(t,d\right)=\mathbf{I}$
I	K × K	Identity matrix	I
C(i)(t, d)	K × K	Factor matrix for covariance	see Proposition 1
b	1	Maximum drug effect parameter	b > 0
E	1	Half maximum drug effect parameter	E > 0
m	1	Drug effect steepness parameter	n > 0
n	1	Initial total cell count	n ∈ [1000, 10000]
c	1	Standard deviation of the i.i.d. observation noise	c ∈ (0, 0.1n)
p	1 × K	Vector of initial proportion pi for subtype i	${\sum }_{i=1}^K{p}_i=1,n_i=n{p}_i$
π(d)	1 × K	Stable proportion	π(d)A(d) = λ1(d)π(d)
$\mathcal{D}$	1 × ND	Set of concentration levels applied	$\mid \mathcal{D}\mid =N_D$
$\mathcal{T}$	1 × NT	Set of time points	$\mid \mathcal{T}\mid =N_T$
${\mathbf{W}}_{n_i}^{\left(i\right)}\left(\mathcal{T},d\right)$	1 × NT	Centered and normalized process of total cell counts	${\mathbf{W}}_{n_i}^{\left(i\right)}\left(\mathcal{T},d\right)=\frac{1}{\sqrt{n_i}}{\sum }_{j=1}^{n_i}\left({\mathbf{y}}_j^{\left(i\right)}\left(\mathcal{T},d\right)-{\mathbf{\mu }}^{\left(i\right)}\left(\mathcal{T},d\right)\right)$
${\mathbf{\mu }}^{\left(i\right)}\left(\mathcal{T},d\right)$	1 × NT	Expected value of the total cell counts	${\mu }^{\left(i\right)}\left(\mathcal{T},d\right)\in {\mathbb{R}}^{1\times N_T}$
${\mathbf{V}}^{\left(i\right)}\left(\mathcal{T},d\right)$	NT × NT	Covariance matrix of ${\mathbf{W}}_{n_i}^{\left(i\right)}\left(\mathcal{T},d\right)$	${\mathbf{V}}^{\left(i\right)}\left(\mathcal{T},d\right)\in {\mathbb{R}}^{N_T\times N_T}$

Deterministic approximation and time-delayed drug-effect model

The statistical model (9) utilizes the central limit theorem approximation derived in Proposition 1. Alternatively, one can employ a law of large numbers type approximation to derive a simpler statistical model. Since the random vectors $\left\{{\mathbf{y}}_j^{\left(i\right)}\left(\mathcal{T},d\right);j\ge 1\right\}$ are i.i.d., the law of large numbers (LLN) states that:

$$\underset{n_i\to \infty }{\lim }\frac{1}{n_i}\sum _{j=1}^{n_i}{\mathbf{y}}_j^{\left(i\right)}\left(\mathcal{T},d\right)={\mu }^{\left(i\right)}\left(\mathcal{T},d\right)\left(a.s.\right),$$

i.e. the mean of the i.i.d. random vectors ${\mathbf{y}}_j^{\left(i\right)}\left(\mathcal{T},d\right)$ converges to their expected value. This result allows us to formulate a simpler statistical model, LLN Model, as follows:

$${\mathbf{x}}_{\mathcal{T},d,r}\approx \sum _{i=1}^K{n}_i{\mu }^{\left(i\right)}\left(\mathcal{T},d\right)+N^{\left(r\right)}\left(0,c^2\mathbf{I}\right),$$ 11

where the variability within the data is solely attributed to the observation noise with a standard deviation of c. We note that this model assumes deterministic cell growth dynamics, as ${\mathbf{\mu }}^{\left(i\right)}\left(\mathcal{T},d\right)$ is a deterministic function of time and dosage. Since this model relies solely on the mean behavior of our stochastic model of the cell dynamics, it will have less predictive power than the CLT-type approximation model (9) when the true cell dynamics have a stochastic nature. In¹⁴, Greene et al. have studied a similar model; however, they did not specify the dose-response relationship using the general Hill equation and instead suggested a more restrictive linear relationship.

Although we are mainly interested in investigating the performance of the CLT Model (9) through in silico experiments, the LLN Model (11) allows for easier computation of the likelihood function; we therefore use this model when analyzing in vitro data with a time-delayed drug response. Specifically, we define the time-inhomogeneous mean behavior of the asymmetrical division multi-type branching process as:

$${\tilde{m}}^{\left(i\right)}\left(t,d\right)=\mathrm{E}\left[{\mathbf{B}}^{\left(i\right)}\left(t,d\right)\right]={\mathbf{e}}_i\exp \left({\int }_0^t\mathbf{A}\left(\tau ,d\right)d\tau \right),$$

where the infinitesimal generator matrixA(t, d) depends on both time and concentration levels, as described in the logistic time-delayed drug effect model proposed in Section “Drug-effect Model”. Although the corresponding covariance for B⁽ⁱ⁾(t, d) can be computed in a similar manner, calculating the variance within a realistic time frame is challenging. Therefore, we employ the simplified statistical model (11) with ${\mathbf{\mu }}^{\left(i\right)}\left(\mathcal{T},d\right):=\left[⟨{\tilde{m}}^{\left(i\right)}\left(t_1,d\right),\mathbf{1}⟩,\cdots ,⟨{\tilde{m}}^{\left(i\right)}\left(t_{N_T},d\right),\mathbf{1}⟩\right]$ when incorporating the logistic time-delayed drug effect in analyzing the in vitro datasets.

Simplified model of drug effect on CSCs and CNSCs mixture

In the “Results” section, we investigate the performance of our newly proposed framework on a tumor population consisting of two phenotypes: 1. CSCs, 2. CNSCs. Our primary focus is on identifying drug-induced plasticity, that is drug-induced transitions from the CNSC phenotype to the CSC phenotype. Given that we only have two subpopulations, we identify the parameters for the CSC resistance subpopulation with an r subscript and those for the CNSC sensitive subpopulation with a s subscript, as shown in Table 12:

Table 12.

CSCs and CNSCs subpopulation parameters

Cell type	Initial proportion	α rate	β rate	ν rate	b for β	E for β	b for ν	E for ν
CSCs	pr	αr	βr	νrs	br,β	Er,β	br,ν	Er,ν
CNSCs	ps	αs	βs	νsr	bs,β	Es,β	bs,ν	Es,ν

The observation noise standard deviation c is the final parameter of the statistical model (9). In practice, some of these parameters can be fixed by making assumptions according to prior biological knowledge, resulting in a more compact statistical model. We next state the assumptions allowing us to fix some of the parameters. The remaining parameters, which are estimated from data using the maximum likelihood approach, are referred to as ‘free parameters’.

To investigate heterogeneous drug effects on CSCs and CNSCs, we adopt assumptions inspired by the experimental work^7,31. First, we specify assumptions related to the cell growth dynamics:

Assumption 1

CNSCs exhibit a non-negative proliferation rate, while CSCs possess a positive natural proliferation rate and a positive differentiation rate, expressed as κ_r > 0, κ_s ≥ 0, ν_rs > 0.

Assumption 2

CNSCs do not exhibit natural plasticity, implying no transition from CNSCs to CSCs in the absence of drug.

We note that under Assumption 2, the irreducibility condition of Section “Asymmetrical birth multi-type branching process model” is violated in the absence of drug. However, our framework can still be applied under the following non-extinction assumption, as discussed further in Supplementary Note 1.

Assumption 3

CSCs do not go extinct, ensuring the persistence of the CSC subpopulation.

Additionally, we constrain the pharmacologic dynamics of cells as follows:

Assumption 4

The drug does not affect CSCs, reflecting a resistant behavior in CSCs.

We note that if the starting population of CSCs is sufficiently large and CSCs have a positive net growth rate, κ_r > 0, then Assumption 3 will follow from Assumption 4, since a large starting population of proliferating cells unaffected by the drug is unlikely to go extinct.

The final assumption pertains to the initial distribution between CSCs and CNSCs. In⁷, the researchers experimentally separated CSCs and CNSCs, and they performed assays on isolated subpopulations of CNSCs. However, assuming the stable proportion between CSCs and CNSCs at the start (Section “Long-run behavior”)—without employing a separation technique—is more natural. Hence, we begin with Assumption 5 and later relax it in Section “Testing robustness of the framework using in silico datasets”.

Assumption 5

The initial proportion of CSCs and CNSCs in the absence of the drug is the stable proportion.

By adopting this assumption, there is no need to estimate the initial proportions p_r and p_s; these parameters are inferred directly from the cell dynamic parameters (α, β, ν) for each subpopulation.

The corresponding mathematical formulations for the above assumptions are summarized in Table 13.

Table 13.

Assumptions and corresponding parameter constraints

Assumptions	Reflection in parameter
Assumption 1	κr > 0, κs≥0, νrs > 0
Assumption 2	νsr = 0
Assumption 4	br,β = br,ν = 1
Assumption 5	p = π(0)

Author contributions

C.W., E.B.G., J.F., and K.L. conceived of the study; C.W. and K.L. designed model; C.W.performed computational experiments; C.W. conducted data analysis; K.L. supervised the research;C.W., E.B.G., J.F., and K.L. wrote the paper.

Data availability

The in vitro data analyzed in this study were accessed from previously published experimental literature. Citations to all sources of data are included in the manuscript. The in silico data produced and analyzed in this study are available on GitHub at this link: https://github.com/chenyuwu233/Cancer-Stem-Cells-Drug-induced-Plasticity.

Code availability

All code is available at GitHub and can be accessed at this link: https://github.com/chenyuwu233/Cancer-Stem-Cells-Drug-induced-Plasticity.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41540-025-00560-8.