scStateDynamics: deciphering the drug-responsive tumor cell state dynamics by modeling single-cell level expression changes

Abstract

Understanding tumor cell heterogeneity and plasticity is crucial for overcoming drug resistance. Single-cell technologies enable analyzing cell states at a given condition, but catenating static cell snapshots to characterize dynamic drug responses remains challenging. Here, we propose scStateDynamics, an algorithm to infer tumor cell state dynamics and identify common drug effects by modeling single-cell level gene expression changes. Its reliability is validated on both simulated and lineage tracing data. Application to real tumor drug treatment datasets identifies more subtle cell subclusters with different drug responses beyond static transcriptome similarity and disentangles drug action mechanisms from the cell-level expression changes.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13059-024-03436-y.

Background

Drug resistance is one of the major challenges for tumor therapy. High molecular heterogeneity of tumor cells frequently leads to intrinsic drug resistance, while the dynamic plasticity of tumor cells further causes adaptive or acquired resistance over time. Therefore, it is crucial for improving the tumor therapeutic efficacy by investigating the biological mechanisms underlying the tumor cell state dynamics during drug treatment [1–3].

Recent advances in single-cell RNA sequencing (scRNA-seq) technologies provide a great opportunity to characterize and analyze the heterogeneity of cell states at a given condition [4]. However, it is still challenging to understand the tumor cell state dynamics during drug treatment by integrating the cell state “snapshots” (as scRNA-seq data) at different time points [5]. Due to the limitations of current sequencing technologies, cells are destroyed during transcriptome profiling, making it very hard to track the dynamic gene expression of any individual cell at multiple time points. Hence, it is promising if any computational method can virtually align the unpaired single-cell level snapshots between two or more time points as a temporal-like “video”.

Aligning at cluster-level, through direct integrated clustering [6, 7] or nonlinear distribution comparison [8–11], has emerged as a widely utilized strategy and leads to some valuable biological discoveries. But it does not fully consider the variation between cells and is hard to model complex multi-fate characteristics of tumor cell populations in response to drugs. Alternatively, a few cell-level algorithms based on mathematical modeling (such as differential equations) have been proposed to align cells during development and differentiation processes [12–16]. More recently, data-driven strategies have also been developed to cope with the uncertainty of cell fate trajectories when analyzing perturbation responses under diverse stimulation conditions (such as cytokines, drugs, etc.) [17, 18]. However, especially for the tumor cell state dynamics under drug treatment, the high heterogeneity and plasticity of tumor cells lead to more complex drug responses in both cell population abundance and state, which requires special consideration in computational modeling.

Here, we present scStateDynamics, a computational method to thoroughly decipher the tumor cell state dynamics in response to drug treatment by modeling single-cell level expression changes. This method first infers the dynamic characteristics of tumor cell states by minimizing the overall changes in gene expression while also considering distinct proliferation or inhibition rates across cell populations. Then, we identify cluster-shared and cluster-specific components (gene factors) from the cell-level expression changes to dissect the drug action mechanisms. By testing on both the simulated data and the data with lineage tracing information, scStateDynamics shows reliable performance in aligning the cells at different time points. By applying to more real datasets from different tumor types under distinct therapeutic strategies, we highlight the significance of modeling cell-level expression changes in uncovering the intrinsic and acquired intra-cluster heterogeneity of tumor cells in response to drugs and also in dissecting the dynamic mechanisms of drug action.

Results

Overview of scStateDynamics

scStateDynamics is designed to infer tumor cell state dynamics under drug treatment and dissect tumor drug response mechanisms by modeling expression changes. First, we define each tumor cell state with the average expression profiles of several highly similar cells (Additional file 1: Fig. S1a). Then, we align the cells between pre- and post-time points based on the principle of minimizing the overall changes in gene expression, which can be formally modeled using optimal transport (OT) theory. OT was originally developed to determine the most efficient way to move a pile of sands from one location to another, and has been widely applied to compare two probability distributions [19]. Here, we normalize the number of cells in all tumor cell states at each time point into a discrete probability distribution. This allows us to align cells by seeking an optimal transport plan between these two cell state probability distributions. To measure the degree of cell state change (transport cost) and determine which alignment relationship (transport plan) is optimal, we adopt the idea of manifold learning [15, 20, 21], which propagates the local neighbor relationships among cell states to obtain their global distances along the low-dimensional manifold in high-dimensional space. Based on this distance matrix (also called the cost matrix) and the corresponding two cell state probability distributions, we derive an optimal transport plan matrix, which provides an alignment plan between cells. Further, by partitioning the cells into several distinct clusters, we can also derive the dynamic cell flows at the subcluster level (Additional file 1: Fig. S1b). These flows can be evaluated based on their average weighted transport costs, which offer a quantitative measure of the extent of cell state changes (“Methods”).

Notably, ignoring the diverse proliferation or inhibition rates between tumor cell populations may lead to the unreasonable identification of several cell flows. In detail, certain source cells had relatively higher proliferation rates, but their corresponding target cells lacked sufficient probability masses to accommodate them. Consequently, the OT algorithm aligned these increased source cells to other target cells, leading to unreasonable flows with abnormally high transport costs (outliers). Therefore, we categorize the cell flows as either “state-keeping”, “state-changed”, or “unreasonable flows” based on their transport costs and calculate the relative proliferation rates of clusters by correcting the unreasonable flows. In this way, by iteratively performing OT and correcting flows several times until convergence (Additional file 1: Fig. S2 provides an illustrative example of the iterative process for the simulation dataset 1 mentioned later), we obtain the final cell alignment relationships and quantify the changes in cell states and cell abundances (proliferation, inhibition, or death). These results enable the identification of subclusters with distinct drug response fates and facilitate the exploration of intra-cluster intrinsic or acquired heterogeneities (“Methods”).

Furthermore, to dissect the biological mechanisms underlying drug action, we calculated the cell-level changes ($\mathrm{\Delta }$) in gene expression profiles according to the previously inferred cell alignment results and designed a Bayesian factor analysis (FA) model to decompose the cell-level expression changes into the static components shared within each pre- or post-cluster and the dynamic components (gene factors) reflecting drug action mechanisms across clusters (“Methods”). This provides a new insight of integrating dynamic information to characterize drug action mechanisms and cell cluster heterogeneities (Fig. 1).

Fig. 1.

Overview of the scStateDynamics algorithm. scStateDynamics is designed to decipher the tumor cell state dynamics under drug treatment by modeling cell-level gene expression changes. Given the pre- and post-treatment single-cell gene expression data, scStateDynamics first measures the distances between cell states in low-dimensional manifold space and infers initial alignment relationships between cell states by minimizing the overall changes based on optimal transport theory. Integrating the clustering results allows for the further derivation of the subcluster flows. Then, according to the estimated transport cost of each subcluster flow, we categorize the flow type as either state-keeping, state-changed, or unreasonable flow. By iteratively correcting the unreasonable flows, we finally estimate distinct proliferation or inhibition rates of clusters and determine the types of abundance changes they exhibit. Based on the inferred cell-level dynamics, scStateDynamics implements a Bayesian factor analysis model to decompose the expression changes ($\mathrm{\Delta }$) into static cluster-specific variations and dynamic cluster-shared gene factors. This provides a novel perspective of integrating dynamic information to dissect drug effects, characterize cell pairs, and compare cluster heterogeneities

Performance validation of scStateDynamics by using simulated data

To assess the reliability of scStateDynamics, we simulated the transcriptome profiles of tumor cells in the pre- and post-treatment stages of three distinct drug response scenarios using Splatter [22] (Additional file 2: Table S1). In scenario 1, the Pre_0 cluster was sensitive to the drug and transitioned to Post_2 with a decreased proportion. The Pre_1 cluster adapted to the drug treatment by adjusting its state and transitioned to Post_0. Conversely, the Pre_2 cluster maintained its intrinsic resistance and transitioned to Post_1 (Fig. 2a). In scenario 2, we added a design in which the Pre_0 cluster initially exhibited sensitivity and experienced cell death, but gradually transitioned into a resistant state, resulting in the emergence of the Post_1 cluster (Fig. 2b). In scenario 3, we designed the clusters to either undergo cell death (Pre_1) or be driven towards an intrinsic resistant state (Pre_0, Pre_2, and Pre_3 transitioning to Post_0) (Fig. 2c). By applying scStateDynamics on these data, we successfully inferred the alignment relationships between clusters (Fig. 2d-f), except for three tiny erroneous flows (Pre_1- > Post_1 and Pre_1- > Post_2 in scenario 1, and Pre_1- > Post_1 in scenario 2) due to the inherent biases of unsupervised clustering at cluster boundaries. Furthermore, the inferred proliferation or inhibition rates (Fig. 2g-i), indicative of relative sensitivity or resistance types, were consistent with the simulation settings. The estimated transport distance effectively quantified the extent of cell state changes, thereby facilitating accurate determination of the intrinsic or acquired resistance types (Fig. 2d–f). Further, we benchmarked scStateDynamics against the recently proposed method CINEMA-OT (causal independent effect attribution + optimal transport) and its variant CINEMA-OT-W (adding a reweighting step to overcome differential abundance) [18]. The results showed that scStateDynamics achieved higher accuracy in identifying the simulated alignment relationships between clusters (Additional file 1: Fig. S3).

Fig. 2.

scStateDynamics accurately identifies the drug response characteristics in simulated data. a–c Left side is the low-dimensional PCA representation of cells in three simulation scenarios, with point color indicating the pre- or post-cluster labels. Right side is the corresponding predesigned cluster alignment relationships and drug response types. d–f Left side is a Sankey plot showing the cell subcluster alignments inferred by scStateDynamics. The color of the rectangles corresponds to the pre-clusters and post-clusters, while the height of each rectangle indicates the cluster’s proportion in all cells. Right side is histogram plots showing the distribution of the transport distances (costs) in all flows, with the color representing the average transport cost of each flow. g–i Barplots showing each pre-cluster’s relative proliferation or inhibition rate estimated by scStateDynamics. A positive height indicates an increased proportion (proliferation), while a negative height indicates a decrease (inhibition). Red stars mark the ground truths of the proliferation or inhibition rates predesigned in the simulation

The inferred results of scStateDynamics are supported by lineage tracing information

Lineage tracing technologies have emerged as a recent advancement, utilizing inherited DNA sequences (barcodes) to track cell clones [23, 24]. These barcode labels can be seen as strong evidences for linking cells across time [24]. Here, we collected the scRNA-seq data of PC9 lung cancer cell lines at four time points (days 0, 3, 7, and 14) during osimertinib treatment, in which the cell lineages were simultaneously tracked using the Watermelon system [25] (Additional file 2: Table S2). We performed pre-processing and clustering on the data (Fig. 3a) and applied scStateDynamics to infer the dynamic alignment relationships (subcluster flows) between clusters from adjacent time points (Fig. 3b). To evaluate the reliability of the identified subcluster flows, we counted the number of lineage barcodes connecting cells in each pair of clusters. To avoid the influence of cluster sizes, we normalized the lineage barcode counts using the square root of the product of the cell numbers in each cluster pair (“Methods”). Notably, we observed that almost all combinations of pre-clusters and post-clusters were supported by lineage barcodes at varying levels. scStateDynamics successfully identified the majority of these cluster flows. The remaining unidentified flows had only exceedingly few supporting barcodes, which may be due to potential random fate biases with minimal probabilities (Fig. 3c). We compared the cell lineage barcodes with the cell alignments inferred by scStateDynamics, CINEMA-OT, and CINEMA-OT-W and found that scStateDynamics exhibited higher correctness and completeness (Additional file 1: Fig. S4, “Methods”). Besides, we demonstrated that scStateDynamics outperformed the conventional joint-clustering method in aligning cells at cluster level (Additional file 1: Fig. S5, “Methods”).

Fig. 3.

The inferred results of scStateDynamics are supported by lineage tracing information. a UMAP (Uniform Manifold Approximation and Projection) plots of the cells in PC9 lung cancer cell line at days 0, 3, 7, and 14 after osimertinib treatment in Watermelon lineage tracing scRNA-seq data. The color indicates the cluster label of the cells at each time point. b The inferred cell subcluster alignment relationships and the quantified flow costs at each pair of adjacent time points. c Barplots showing the normalized lineage barcode count of each subcluster flow. The height of the bar indicates the strength of confidence supported by lineaging barcodes. The color denotes whether the subcluster flow is identified by scStateDynamics (green for yes, red for no)

Furthermore, we evaluated the performance of scStateDynamics in inferring the proliferation or inhibition rates of cell clusters. We collected a lineage tracing dataset, termed ReSisTrace, which utilizes lentiviral labeling to track the lineage of ovarian cancer cells after treatment of drug olaparib, a PARP enzyme inhibitor [26]. The dataset comprises two replicates (named Olaparib_1 and Olaparib_2), and the cells were sampled before and after olaparib treatment (Additional file 2: Table S2). We firstly clustered the cells (Additional file 1: Fig. S6a and S6e) and analyzed the fraction of lineage barcodes in each pre-cluster that were either inherited or disappeared after drug treatment (Additional file 1: Fig. S6b and S6f). Notably, we found that cluster S1 in Olaparib_1 exhibited a lower inherited fraction compared to other clusters (Additional file 1: Fig. S6b). The analysis of scStateDynamics also inferred that cluster S1 had higher inhibition rates (Additional file 1: Fig. S6c). Furthermore, we compared the expression levels of PARP1 (the target of olaparib) across different clusters and found that cluster S1 exhibited higher expression values (Additional file 1: Fig. S6d), suggesting it may be more sensitive to olaparib. Similar results were also observed for clusters S1 and S2 in Olaparib_2 dataset (Additional file 1: Fig. S6f, S6g and S6h). The consistency of these three aspects of information proves the reliability of the proliferation and inhibition rates estimated by scStateDynamics.

scStateDynamics improves the characterization of tumor subcluster heterogeneity in drug response

To showcase the utility of scStateDynamics in unraveling the underlying heterogeneity and plasticity of tumor drug responses, we collected some real scRNA-seq datasets of tumor cells under drug treatment (Additional file 2: Table S3).

Taking the hepatocellular carcinoma (HCC) dataset as an example, the scRNA-seq data were detected from clinical patient biopsies before and after the immunotherapy with tremelimumab/durvalumab [27]. Through clustering (Fig. 4a) and inter-cluster differential expression analysis, we observed that cluster S0 exhibited higher activity in normal hepatocyte functions, while cluster S1 displayed enhanced malignant proliferation. Cluster S2 expressed immune-related pathways and genes, suggesting potential sensitivity to immunotherapy (Fig. 4b and Additional file 1: Fig. S7a). Cox regression models further confirmed these malignancy patterns (Additional file 1: Fig. S7b, “Methods”).

Fig. 4.

scStateDynamics facilitates comprehensive analysis of both intrinsic and acquired heterogeneities in tumor drug response. a UMAP plots of HCC cells before and after immunotherapy with tremelimumab/durvalumab. The color indicates the clustering results, with labels prefixed by “S” (for pre-treatment) and “T” (for post-treatment). b Bubble heatmap showing the hallmark pathway enrichment analysis results of the differentially expressed genes (DE genes) at inter-cluster level. c The inferred cell subcluster alignment relationships and the quantified flow costs by scStateDynamics. The inferred proliferation or inhibition rates of pre-clusters are presented at the bottom. d Sketch map showing the analysis of intra-cluster heterogeneity. Blue arrowed lines indicate intrinsic heterogeneity analysis, which compares the subclusters with different fates within each pre-cluster. Red arrowed lines indicate acquired heterogeneity analysis, which compares cells before and after treatment in each subcluster flow. e The enrichment analysis results of the DE genes at intra-cluster level. Each panel represents one pre-cluster, and the columns in it indicate its subclusters with different fates. f Forest plot showing the hazard ratio (HR) of the DE gene signature of each subcluster in the TCGA-LIHC cohort. The solid line indicates HR = 1. CI, confidence interval. g Bubble heatmap showing the change values of pathway scores ($\mathrm{\Delta }$ score) in each subcluster flow. h Selected DE genes between the subclusters in S0 flowing to T0 and T1. i The expression changes ($\mathrm{\Delta }$ expr) of metabolism-related genes in flows S1- > T1 and S1- > T2. j The distribution of the drug tolerant persister (DTP) scores in the source and target cells of four flows. One-sided Wilcoxon rank-sum test. ****p < 0.0001. k The mean of the signature scores in the source (circle) and target (diamond) cells of the identified DTP flows. The directions of the arrows reflect increasing or decreasing trends. One-sided Wilcoxon tests were performed on their distribution. **p < 0.001, ***p < 0.0001, ****p < 0.00001

Beyond routine inter-cluster comparisons, the cell-level alignment via scStateDynamics allowed us to identify subclusters with distinct sources or targets within each cluster (Fig. 4c). This provides novel perspectives for analyzing the intra-cluster intrinsic and acquired heterogeneities of tumor cells in response to drug treatment (Fig. 4d). We performed enrichment analyses on the differentially expressed genes between the subclusters exhibiting different fates within each per-treatment cluster. Within cluster S0, the subcluster transitioning to cluster T2 (S0- > T2) displayed a notable loss of metabolic function compared to the other subclusters (S0- > T0 and S0—> T1). Within cluster S1, the subclusters flowing to clusters T1 and T2 (S1- > T1 and S1- > T2) exhibited higher activity in proliferation-related pathways, in contrast with the subcluster flowing to T0 (S1- > T0) (Fig. 4e). Cox proportional hazards regression analysis confirmed these findings, indicating that the subcluster in S0 flowing to T2 was more malignant potentially due to impaired metabolic function, and the subclusters in S1 flowing to T1 and T2 displayed high malignancy possibly due to enhanced proliferation ability (Fig. 4f). Further, to assess the robustness of our findings, we randomly shuffled the inferred cell fate labels within each cluster 100 times and then performed the same intra-cluster intrinsic heterogeneity analysis. The results showed that none of the genes reached a statistically significantly differential expression (adjusted p-value < 0.05) after any of the 100 permutations, which implied that scStateDynamics effectively assisted in the identification of intra-cluster heterogeneity compared to random shuffling of cell fates (Additional file 1: Fig. S8).

Then, to investigate the acquired heterogeneity arising within each subcluster flow, we calculated the change values of pathway scores ($\mathrm{\Delta }$ scores) based on the cell alignment relationships inferred by scStateDynamics (“Methods”). Comparing the average $\mathrm{\Delta }$ scores of the cells in each subcluster flow revealed distinct drug response characteristics. In cluster S0, the subclusters generally maintained their states, but flow S0- > T1 exhibited slightly increased activity in proliferation-related pathways (Fig. 4g). To investigate the potential molecular causes of this unfavorable drug response in flow S0- > T1, we further checked the intrinsic differences between the subclusters flowing to T0 and T1. The analysis revealed that, although both subclusters resembled normal hepatocytes (as shown in Fig. 4e above), the subcluster flowing to T1 exhibited higher expression of several specific genes (such as TPT1, TMEM97, CELF1, KLC4, MAN2A1, VEGFA, and CTNND1) (Fig. 4h), which have been previously reported to be involved in tumor progression or immunotherapy resistance [28–34]. In cluster S1, the subclusters displayed more pronounced acquired heterogeneity. The flows S1- > T0 and S1- > T2 exhibited significantly reduced activity in proliferation-related pathways, suggesting effective inhibition of tumor cell proliferation. Conversely, the flow S1- > T1 maintained high proliferation ability (intrinsic characteristics of the subcluster in S1 flowing to T1), indicating insensitivity to treatment (Fig. 4g). Although both S1- > T1 and S1- > T2 displayed enhanced activities in metabolism-related pathways, further analysis revealed that S1- > T1 exhibited malignancy-related metabolic reprogramming (increased expression levels of the genes CD36, IDH1, PGD, and PFKM) [35–38], while S1- > T2 exhibited the recovery of normal metabolic function (increased expression levels of the genes ABCA6, LEAP2, APOA1, and APOE) (Fig. 4i). Notably, the original study of this HCC dataset used joint hierarchical clustering analysis and found that the cells overall changed their states after drug treatment [27], while our method of cell-level alignment effectively distinguished the cell subclusters flows with varying levels of acquired heterogeneity, highlighting the values of inferring cell-level dynamics.

We also tested scStateDynamics on two other cancer types under distinct treatment strategies: glioblastoma (GBM) samples undergoing temozolomide chemotherapy [39] and non-small-cell lung carcinoma (NSCLC) samples receiving erlotinib-targeted therapy [40] (Additional file 1: Fig. S9, “Methods”). Notably, acquired heterogeneity analysis identified some intriguing flows, which appeared to enter a state known as drug-tolerant persisters (DTPs) [41–43], including the flows C1- > D1 and C2- > D1 in GBM2 dataset and the flows S0- > T0 and S1- > T0 in NSCLC dataset. DTPs, characterized by tumor cells becoming quiescent or slow cycling to evade drug-induced death while retaining the ability to resume growth upon drug removal, are increasingly recognized as being associated with tumor relapse [43]. To validate this hypothesis, we utilized a DTP signature [42] to score these subclusters and observed indeed higher DTP scores after drug treatment (Fig. 4j). Furthermore, we examined the functional changes in these flows and found that the cells displayed increased autophagy capability [44, 45] but decreased proliferation ability and mTORC1 pathway activity, consistent with previous reports [42] (Fig. 4k and Additional file 1: Fig. S10). This was also in accordance with the findings reported in the original studies of GBM2 and NSCLC datasets [39, 40].

In summary, scStateDynamics improves the resolution of investigating tumor cell heterogeneity from inter-cluster to intra-cluster level. This enables the identification of subclusters with different fates and facilitates the dissection of both intrinsic and acquired heterogeneity of tumor drug response from a dynamic perspective. We observed that despite overall similar expression profiles, some subtle molecular differences may potentially lead to distinct drug responses, which is consistent with the conclusion that diverse resistant clones evolve from homogeneous tumor cells in a recent experimental study [46]. These downstream analyses deepen our understanding about the molecular mechanisms underlying anti-tumor drug responses and provide valuable insights for developing more effective therapeutic strategies by targeting intrinsic drug resistance and drug-induced cell plasticity.

scStateDynamics disentangles the cluster-shared and cluster-specific drug effects

We devised a Bayesian factor analysis (FA) model in scStateDynamics to extract the biological factors (gene signatures) that contribute to the observed cell-level changes in gene expression ($\mathrm{\Delta }\mathit{X}$) before and after treatment. The dynamic changes in expression ($\mathrm{\Delta }\mathit{X}$) are determined by two types of variation: (i) the initial and final static molecular heterogeneity between cells, which can be characterized by cell’s cluster identity (${\mathit{U}}_s$ and ${\mathit{V}}_t$), indicating that the cells belonging to the same cluster shared some common effects and (ii) the dynamic molecular mechanisms of drug action, which are shared by all clusters and can be seen as a combination of multiple expression program factors ($\mathit{W}\mathit{Z}$). To quantify these static and dynamic variations, we defined probability distributions of the variables and estimated their values by variational inference (Fig. 5a, “Methods”).

Fig. 5.

scStateDynamics disentangles the expression changes under drug action. a The probabilistic graphical model representation of the Bayesian factor analysis (FA) model in scStateDynamics, illustrating hidden random variables as circles and observed variables as shaded circles. Edges denote the statistical dependences between the variables. Boxes (plates) signify independent replications. The table provides a description of the variables and their corresponding probabilistic distributions. b Heatmaps showing the pathway scores of the identified factors in three datasets. c Scatter plots showing the gene comparisons for each pair of pre-clusters in the HCC dataset. The x-axis represents the differences between the $\mathit{U}$ vectors of two pre-clusters, while the y-axis represents the differences between the average expression value vectors of two pre-clusters. Square points denote the differential expression genes (DEGs) identified based on pre-treatment expression, with colors corresponding to the cluster colors used in Fig. 4a. Triangular points in red or blue denote the marker genes newly identified by subtracting $\mathit{U}$ vectors. Some top genes associated with tumor progression are labeled with their gene symbols. Gray circle points denote other genes

We applied this FA model to previously used real datasets. First, we observed that $\mathit{Z}$, as a new embedding of cell pairs, was indeed not relevant to either the pre- or post-cluster labels, proving the successful decomposition of these two types of variations (Additional file 1: Fig. S11). Then, we utilized the gene weights ($\mathit{W}$) to calculate the factor scores for some known pathways (“Methods”). Notably, the factors identified in all the datasets were biologically meaningful and associated with the mechanisms of the corresponding drugs (Fig. 5b, Additional file 1: Fig. S12 and S13). For example, in HCC sample undergoing immunotherapy, factor 1 was strongly associated with pathways involved in immune response modulation. In GBM1 sample undergoing chemotherapy, factors 1, 2, and 3 were all related to DNA replication pathway, consistent with the drug’s mechanism of action on DNA. In NSCLC sample undergoing targeted therapy, factor 1 displayed a remarkably strong association with ERBB signaling pathway, which corresponds precisely to the target gene of Erlotinib. In addition, the decomposed static cluster-specific variations ($\mathit{U}$ and $\mathit{V}$) provide a novel representation for each cluster that considers dynamic information. We observed that some genes with highly positive or negative effects in $\mathit{U}$ vectors were associated with tumor progression and prognosis, which may be conventionally ignored due to their relatively low average expression levels (Additional file 1: Fig. S14). By comparing the $\mathit{U}$ vectors between each pair of pre-clusters (Fig. 5c), we identified some new cluster-specific and tumor-related genes (red and blue triangular points) that were not captured by conventional differential expression analysis based on static pre-treatment expression (square points). For example, MLKL, a necroptosis regulator, plays a complex but poorly understood role in cancer development and metastasis [47], and it was recently reported that it can promote immune evasion in HCC [48]. High MAOA expression was reported to be associated with an immunosuppressive tumor microenvironment and poor prognosis [49]. And CCDC50 was shown to promote HCC growth via Ras/Foxo4 signaling [50, 51]. These results highlight the profound value of decomposing gene expression changes in characterizing drug mechanisms and cell cluster heterogeneity from a novel dynamic perspective.

Discussion

scStateDynamics is an algorithm to investigate the tumor cell state dynamics in response to drug treatment by modeling single-cell level gene expression changes. To cope with the challenge of unpaired cells, we employed the principle of minimizing the overall weighted changes in gene expression along a low-dimensional manifold to align the cells before and after treatment. Considering the varying drug sensitivities across tumor clusters due to their high heterogeneity and plasticity, we used a data-driven approach to discern the types of cell subcluster flows and estimated the proliferation or inhibition rates of the clusters. To dissect the biological characteristics underlying drug actions by integrating dynamic information, we designed a Bayesian FA model to decompose the gene expression changes into cluster-specific static variations and cluster-shared dynamic gene factors.

For the investigation on the dynamics of complex systems, we think modeling the change values ($\mathrm{\Delta }$) is very important. Thus, we adopt the strategy of minimizing the overall changes to align cells and decompose the change value matrix to uncover the mechanisms of drug action. The significance of analyzing dynamic changes has been validated on previous studies, such as analyzing differential protein–DNA interactions across different biological conditions by designing dPCA algorithm [52]. Moreover, learning common and specific effects is also an effective strategy to disentangle the characteristics of complex system, which has been applied to compare the patterns across distinct cancer types or multiple differentiation stages via matrix factorization, such as CSMF algorithm [53]. Here, we adopt this strategy and design a Bayesian model to dissect the cluster-common and cluster-specific factors. Bayesian models have numerous advantages. Specifically, they provide increased flexibility and scalability in capturing intricate relationships among variables, enable the quantification of uncertainty associated with variables from a probabilistic view, and enhance the interpretability of the model structure to understand the complex interplay among variables. Besides, the resolution at cell-level is crucial for analyzing the heterogeneity of tumors. Single-cell technologies offer a promising avenue, but its high noise is very annoying. Constructing metacells, as a preprocessing step, effectively reduces the dropout noise, so that a meta-cell can be regarded as a denoised representation of a cell. For the sake of brevity and understandability, we use the term “cell-level” instead of “metacell-level” or “denoised cell-level” in the study. Further, low-noise single-cell sequencing technologies are expected to bring more solid biological insights.

By applying scStateDynamics on real data of tumor cells during drug treatment, we found that some subclusters exhibited generally similar expression profiles, but subtle differences in a few genes may lead to their distinct drug responses. This inspires us to consider not only the conventional static characteristics of cells (what they are like) but also their dynamic information (where they come from and where they go) when clustering cells and analyzing tumor drug responses. Notably, we accurately identified some cell flows entering the drug-tolerant persister states, which has attracted increasing attention for their role in tumor drug resistance. In addition, by disentangling the drug action mechanisms, we uncovered some candidate genes that may affect tumor progression; these genes were easily overlooked when using only conventional differential analysis on pre-treatment expressions.

Currently, we operate under the assumption that short-term drug treatment cannot induce drastic changes in tumor cell states and result in completely different expression profiles. Thus, we infer the cell dynamics by minimizing the overall gene expression changes. However, the dynamics of tumor drug response are inherently complex. If the time interval is very long, tumors cell may undergo substantial alterations in their molecular states. If the drug is more aggressive, a significant proportion of tumor cell clones may be killed, leaving only a small clone to proliferate under drug treatment. In these scenarios, our initial hypothesis may not hold and scStateDynamics may not be very suitable. Therefore, lineage tracing information becomes invaluable for investigating cancer dynamics. By utilizing this information, we can analyze the pattern of change in tumor cell states and population sizes, thereby refining the model to align cells more accurately. Similarly, in cases of temporally unmatched tumor samples (such as from different patients), the high inter-tumor heterogeneity often obscures the gene expression changes induced by drug treatment. Hence, scStateDynamics may also not be suitable in such scenarios. In addition, batch effect noise is very annoying, as it often coincides with real biological signals. Therefore, a reasonable evaluation and correction for batch effect noise are essential to obtain more accurate measurements of cell–cell distances in scStateDynamics. Besides, scStateDynamics may identify an extremely small number of tiny cluster flows. We attribute this phenomenon partly to the inherent biases of unsupervised clustering and also to the potential randomness that exists in cellular fate. How to better model this randomness of cell fates is also a problem worth exploring. Additionally, our primary focus here is on tumor cells, without considering their interaction with the tumor microenvironment, which plays an important role in regulating tumor cell states. Future works can attempt to model the dynamics of tumor cells, immune cells, and stromal cells simultaneously from a systematic perspective, which may be able to offer more insights into tumor therapy.

Overall, deciphering tumor cell state dynamics under drug treatment is highly valuable but challenging. scStateDynamics fully considers the highly heterogeneous and varying molecular characteristics of tumor drug responses and offers a powerful algorithm framework to analyze tumor cell state dynamics by modeling single-cell level gene expression changes. The analysis results pave the way for understanding the dynamic mechanisms of tumor drug resistance, and further explorations hold great potential for developing more effective treatment strategies.

Conclusions

In this study, we present scStateDynamics, a novel algorithm to infer tumor cell state dynamics under drug treatment and dissect tumor drug response mechanisms by modeling gene expression changes. By testing on simulated dataset, we show that scStateDynamics has superior performance on inferring cell cluster dynamics and can accurately identify predesigned drug response types. By testing on the data with lineage tracing labels, we confirm scStateDynamics has higher correctness and completeness in aligning cells and can avoid the influence of random biases to identify more reliable cluster flows. Furthermore, we apply scStateDynamics to real data from different cancer types treated with immunotherapy, chemotherapy, or targeted therapy. The results demonstrate that scStateDynamics facilitates the identification of cell subclusters with distinct drug response fates, enabling the analysis of both intrinsic and acquired intra-cluster heterogeneity. Moreover, scStateDynamics effectively captures known and potential drug action mechanisms in no matter immunotherapy, chemotherapy, or targeted therapy and offers a novel perspective of integrating dynamic information to characterize cell pairs and compare cluster heterogeneities.

Methods

Inferring the cell state dynamic relationships

To characterize the cell states in a high-dimensional gene expression space and identify their alignment relationships, we designed the following computational steps in scStateDynamics.

Identifying cell states and estimating their transcriptomic profiles

Due to the stochastic nature of gene transcription and the limited detection capacity for rare transcripts, scRNA-seq data often exhibit high-frequency dropout events, resulting in a large proportion of zeros in the gene expression matrix (Additional file 1: Fig. S15) [54–56]. To mitigate the influence of dropout event noise, we add a preprocessing step of grouping several similar cells into a metacell to represent a type of cell state [57–59]. In practice, we utilize the “Scanpy” package [60] to perform clustering with a large ‘resolution’ parameter setting (such as 30 or 50). Each resulting cluster is considered as a metacell, generally consisting of approximately three to ten cells, to represent a specific cell state. This adaptive numbers of cells within metacells can better cope with the varying densities of cell distribution in gene expression space. Then, we characterize each metacell by calculating the average expression profiles of its constituent cells. Generally, the cells with very similar expression profiles can be seen as being resampled from a common cell state, and the probability of dropout events occurring simultaneously in these similar cells for a given gene is relatively low. Therefore, this preprocessing step of aggregating similar cells and utilizing their average expression values to represent the corresponding cell states can reduce the dropout noise. The analysis on our testing datasets also validated its necessity and effectiveness (Additional file 1: Fig. S15), thereby a meta-cell can also be regard as a denoised representation of a cell. Besides, it can reduce the computational complexity of subsequent analyses. Notably, to make this preprocessing step more flexible and robust, we provide an interface for users to either supply their own metacell labels or bypass this preprocessing step through setting unique metacell labels.

Measuring the distances between cell states along the low-dimensional manifold

In the high-dimensional gene expression space, cells are usually distributed on a low-dimensional manifold. This observation suggests that direct calculation of global Euclidean distances may not provide an accurate representation of the similarities between cells along the manifold. Hence, we refer to the methodologies of PHATE [21], Diffusion Map [20], and MuTrans [15] algorithms and utilize local distance diffusion to assess the global relationships among different cell states. In detail, we use the Gaussian kernel function ${\mathit{K}}_{ϵ}\left(x,,,y\right)=\text{exp}\left(-,\frac{{\Vert {\mathit{E}}_x-{\mathit{E}}_y\Vert }^2}{ϵ}\right)$ to transform the global Euclidean distances between the joint-embedded expression profiles ${\mathit{E}}_x$ and ${\mathit{E}}_y$ of metacells $x$ and $y$ into local affinities. This transformation ensures that the affinity degree between two cell states is greater than 0 only when they are close enough, so that we can obtain the local neighbor relationships based on the affinity degrees (Additional file 1: Fig. S16a). The parameter $ϵ$ governs the local bandwidth constrained by the kernel function (Additional file 1: Fig. S16b). Considering that the density of the cell distribution is uneven, the k-nearest-neighbor distance ${ϵ}_k\left(x\right)$ is used as an adaptive bandwidth. Additionally, to control the heavy tail of the Gaussian kernel when ${ϵ}_k\left(x\right)$ is large, the exponent $\alpha$ is introduced (Additional file 1: Fig. S16b). Consequently, the resulting kernel function is obtained by taking the average of two kernel values to ensure the symmetry of the affinity matrix: ${\mathit{K}}_{k,\alpha }\left(x,,,y\right)=\frac{1}{2}\text{exp}\left(-,{\left(\frac{{\Vert {\mathit{E}}_x-{\mathit{E}}_y\Vert }^2}{{ϵ}_k\left(x\right)}\right)}^{\alpha }\right)+\frac{1}{2}\text{exp}\left(-,{\left(\frac{{\Vert {\mathit{E}}_x-{\mathit{E}}_y\Vert }^2}{{ϵ}_k\left(y\right)}\right)}^{\alpha }\right)$. Then we normalize the affinity matrix by its row sums to generate the following random-walk transition probability matrix among cell states $\mathit{P}\left(x,,,y\right)=\frac{{\mathit{K}}_{k,\alpha }\left(x,,,y\right)}{{\sum }_z{\mathit{K}}_{k,\alpha }\left(x,,,z\right)}$. Furthermore, to propagate the local neighbor relationships and assess the long-range affinity values along the low-dimensional manifold, we perform a $t$-step random walk and calculate the $t$-step diffusion probability ${\mathit{P}}^{(t)}={\mathit{P}}^{(t-1)}\mathit{P}$. Here, the parameter $t$ is a positive integer that denotes how many steps of long-range diffusion are considered acceptable, and it can be determined based to the distribution density of cells. And ${\mathit{P}}^{(0)}$ is an identity matrix. Finally, we measure the distance between two cell states by calculating the $l^2$-norm distance of their logarithmic $t$-step diffusion probabilities to other cell states: ${\mathit{D}}_{\mathit{xy}}=\sqrt{{\Vert \text{log}\left({\mathit{P}}_{x·}^{\left(t\right)}\right)-\text{log}\left({\mathit{P}}_{y·}^{\left(t\right)}\right)\Vert }^2}$. Notably, at this distance measurement step, we assume that the influence of batch effects on measuring distances between cells along the manifold is slight. In the practice, we advise users to evaluate the extent of batch effect noise using approaches such as low-dimensional co-projection, or quantitative indicators and then determine whether and how to correct the batch effect before utilizing our algorithm. If the batch effect noise is pronounced, it is essential to correct it beforehand.

Aligning the cell states between two time points

Here, we assume that the overall changes in cell states is relatively small during drug treatment and only a portion of cells have significant state changes. Using the distances along manifold calculated above to measure the extent of cell state changes, we adopt the principle of minimizing the overall changes to align cells in a low-dimensional manifold between two time points, which can be achieved by the optimal transport algorithm. At each time point, all cells form a discrete cell state probability distribution in gene expression space. Therefore, aligning the cell states can be regarded as seeking a transport plan between two probability distributions.

In the discrete case, given the source and target probability distributions

$\mathit{a}\in {\mathbb{R}}_{+}^1\left({\sum }_{i=1}^1,{\mathit{a}}_i,=,1\right)$ and $\mathit{b}\in {\mathbb{R}}_{+}^2\left({\sum }_{j=1}^2,{\mathit{b}}_j,=,1\right)$

and a $n_1\times n_2$ transport cost matrix $\mathit{C}\in {\mathbb{R}}_{+}^{n_1\times n_2}$, where each element ${\mathit{C}}_{\mathit{ij}}$ indicates the cost when transporting from state $i$ to $j$. Then, the objective of the optimal transport problem is to find a transport matrix $\mathit{T}\in {\mathbb{R}}_{+}^{n_1\times n_2}$ that minimizes the total cost (overall changes)

$⟨\mathit{C},\mathit{T}⟩={\sum }_{i,j}{\mathit{C}}_{\mathit{ij}}{\mathit{T}}_{\mathit{ij}}$ subject to ${\sum }_j^2{\mathit{T}}_{\mathit{ij}}={\mathit{a}}_i$ and ${\sum }_i^1{\mathit{T}}_{\mathit{ij}}={\mathit{b}}_j$

where ${\mathit{T}}_{\mathit{ij}}$ indicates the identified probability mass transporting from $i$ to $j$. This means that the total cost is the inner product of the cost matrix $\mathit{C}$ and the transport matrix $\mathit{T}$, while ensuring that the row sum and column sum of $\mathit{T}$ are equal to $\mathit{a}$ and $\mathit{b}$, respectively.

In our cell state alignment problem, the number of cells in all metacells (cell states) at the pre- and post- time points can be normalized as probability distributions $\mathit{a}$ and $\mathit{b}$, respectively. Taking the distances calculated above (matrix $\mathit{D}$) as the cost matrix $\mathit{C}$, we can obtain the transport matrix $\mathit{T}$ based on the optimal transport algorithm, in which each element ${\mathit{T}}_{\mathit{ij}}$ indicates the probability mass transforming from the pre-timepoint to the post-timepoint. According to these transport probabilities, we can align the cells in all metacells between the two time points. This step is implemented with the “ot” Python package [61].

Grouping the cell alignment relationships into subcluster flows

To investigate the dynamics at cluster level, we conduct clustering on the cells at each time point individually using “Scanpy” package [60], with a small “resolution” parameter setting (Additional file 1: Fig. S1a). This step of separate clustering, rather than joint clustering, can avoid the influence of batch effect noise. In this way, we connect the clusters between two time points based on cell flows and identify distinct fates among cells. According to the cell fates (which post-cluster the cell transition to), we further divide the pre-clusters into distinct subclusters (Additional file 1: Fig. S1b). This operation of identifying cell subcluster flows can help infer the distinct proliferation or inhibition rates of clusters and support the subsequent differential expression analysis to dissect intra-cluster intrinsic and acquired heterogeneity between distinct cell fates.

Quantifying the dynamic characteristics of cell populations

After drug treatment, distinct clusters of tumor cells may exhibit varying degrees of sensitivity, leading to differences in proliferation or inhibition rates. Besides, some cells can also change their states to adapt to the external environment. Hence, the dynamic characteristics of cell populations include the changes in both cell states and abundances (proportions). The extent of changes in cell states can be quantified by the transport costs. However, for the cell abundances, the classical optimal transport theory we used could not model the increase or decrease in probability mass that reflects the relative proliferation or inhibition of cell populations. To address this limitation, we first identify the unreasonable cell subcluster flows caused by the neglect of cell proliferation and then correct them to obtain more reliable cell dynamics.

Identifying the types of cell subcluster flows

To quantify the extent of changes in cell states and distinguish the pattern of cell subcluster flows, we calculate the average transport cost for each flow. Given that the metacell sets at pre-timepoint $s$ and post-timepoint $t$ of a cell subcluster flow are $Q_s$ and $Q_t$, we define the average weighted transport cost of it as

$$\text{FlowCost}\left(Q_s,,,Q_t\right)=\frac{{\sum }_{i\in Q_s,j\in Q_t}{\mathit{C}}_{\mathit{ij}}\ast {\mathit{T}}_{\mathit{ij}}}{{\sum }_{i\in Q_s,j\in Q_t}{\mathit{T}}_{\mathit{ij}}}$$

According to the distribution of average transport costs of all subcluster flows, we think they can be categorized into either state-keeping, state-changed, or unreasonable flows. State-keeping corresponds to the cells that maintain high similarity, resulting in flows with low transport costs. State-changed indicates that cells adaptively adjust their states, leading to flows with relatively high transport costs. Unreasonable flows refer to some incorrect alignments identified by OT algorithm arising from the neglect of distinct proliferation or inhibition rates among cell clusters. As a result, when certain source cell populations are inhibited, their reduced probability masses have to be allocated to other target cell populations exhibiting high proliferation rates. These incorrect alignments lead to some unreasonable flows with abnormally large transport costs. To determine the type of flows, an analysis of the average transport cost distribution for all subcluster flows is conducted through plotting a histogram. If the distribution exhibits a distinct trimodal pattern, a Gaussian mixture model (GMM) can be applied to category the three peaks into state-keeping, state-changed, or unreasonable flows. Instead, if there are several outliers that are clearly distant from the majority of data points, outlier detection approaches (e.g., using 1.5 times the interquartile range above the third quartile as a threshold) can be employed to identify these outliers as unreasonable flow. Besides, manually setting the thresholds to identify unreasonable flows is also optional.

Correcting the unreasonable cell flows

Based on the identified type of cell flows, we aim to correct the unreasonable flows and estimate the proliferation or inhibition rates of clusters.

Given an unreasonable flow with a probability mass of $\delta$ from a subset of cluster Pre_A at pre-timepoint to a subset of cluster Post_B at post-timepoint, we can infer that the cluster Pre_A is inhibited, and the inhibition rate can be calculated by dividing $\delta$ by the total probability masses of cluster Pre_A. Meanwhile, the cluster Post_B should originate from the cells that are more similar to it. Hence, if there are other reasonable sources (without loss of generality, we refer to them as Pre_C and Pre_D) for Post_B, we assign the probability mass $\delta$ to Pre_C and Pre_D based on their relative fractions. If Post_B completely originates from Pre_A, we think the cluster with the highest similarity to Post_B should have a greater proliferation rate. In this way, the probability masses of unreasonable flows can be assigned to more appropriate source clusters. By re-normalizing the probability masses at the pre-timepoint, we obtain the updated source probability distribution, denoted as $\mathit{a}_{}^{\mathbf{\prime }}$. Then, we replace $\mathit{a}$ with $\mathit{a}\prime$ to re-perform optimal transport and re-correct the identified unreasonable flows iteratively, until no outlier flows exist or the results stabilize.

In the end, by comparing the final updated source probability distribution with the initial source distribution $\mathit{a}$, we can estimate the final proliferation or inhibition rate of each cluster at pre-timepoint.

Decomposing the expression changes into static variations and dynamic biological effects

The changes in gene expression profiles between the cells at two time points can provide insights into the dynamic biological mechanisms of drug action. According to the obtained optimal transport matrix $\mathit{T}$, an element ${\mathit{T}}_{\mathit{ij}}$ greater than 0 indicates a potential dynamic alignment between the $i$ th cell state (metacell) at the pre-timepoint and the $j$ th cell state at the post-timepoint. Consequently, assuming that there are $M$ elements greater than 0 in $\mathit{T}$, we identify $M$ alignment relationships among cell states, resulting in the formation of $M$ distinct cell pairs. Subsequently, according to the coordinates of these $M$ elements in $\mathit{T}$, we can extract the corresponding $M$ gene expression vectors at the pre- and post-timepoints. These vectors collectively compose the matrices ${\mathit{X}}^{(\text{pre})}\in {\mathbb{R}}_{+}^{G\times M}$ and ${\mathit{X}}^{(\text{post})}\in {\mathbb{R}}_{+}^{G\times M}$, where $G$ denotes the number of genes. Then the matrix of gene expression changes $\mathrm{\Delta }\mathit{X}\in {\mathbb{R}}^{G\times M}$ can be calculated by ${\mathit{X}}^{(\text{post})}-{\mathit{X}}^{\left(\text{pre}\right)}$. Here, we think these dynamic changes are determined by two types of effects: (i) the initial and final cell expression profiles and (ii) the molecular mechanisms of drug action. To disentangle these two types of variations, we design a Bayesian factor analysis model by characterizing the first type of static variations with the cell cluster identities and decomposing the second type of dynamic effects into a combination of gene factors (signatures).

Model representation

Assuming that there are $S$ and $T$ clusters at the two time points, then for a cell pair $m\in \left\{\text{1,2},,,\cdots ,,,M\right\}$, consisting of a metacell within cluster $s\in \left\{\text{1,2},,,\cdots ,,,S\right\}$ and a metacell within cluster $t\in \left\{\text{1,2},,,\cdots ,,,T\right\}$, we decompose its change vector $\mathrm{\Delta }{\mathit{X}}_m$ into a sum of three components and an additive Gaussian noise:

$$\mathrm{\Delta }{\mathit{X}}_m={\mathit{U}}_s+{\mathit{V}}_t+\mathit{W}{\mathit{Z}}_m+\mathbf{\Psi }$$

Here, vector ${\mathit{U}}_s\in {\mathbb{R}}^{G\times 1}$ denotes the common effects shared by the cells in cluster $s$ at pre-timepoint, while vector ${\mathit{V}}_t\in {\mathbb{R}}^{G\times 1}$ denotes the effects shared by the cells in cluster $t$ at post-timepoint. We model them with normal distributions

$$P\left({\mathit{U}}_s\right)=\mathcal{N}\left(\mathbf{0},,,{\sigma }_u^2,\mathbf{I}\right)$$

$$P\left({\mathit{V}}_t\right)=\mathcal{N}\left(\mathbf{0},,,{\sigma }_v^2,\mathbf{I}\right)$$

The matrix $\mathit{W}\in {\mathbb{R}}_{+}^{G\times K}$ denotes the regulatory weights of $K$ factors on genes, and each factor can be seen as a gene signature related to the dynamic biological mechanism of drug action. We constrain the elements in it to be positive and use independent log-normal distribution to model it

$$P\left(\mathit{W}\right)={\prod }_{k=1}^K\text{LogNormal}\left(\mathbf{0},,,{\sigma }_w^2,\mathbf{I}\right)$$

The vector ${\mathit{Z}}_m\in {\mathbb{R}}^{K\times 1}$ denotes the composition coefficients (activities) of the factors for cell pair $m$, and can also be seen as an embedding of this cell pair in the space spanning by these gene factors. We model it with normal distribution

$$P\left({\mathit{Z}}_m\right)=\mathcal{N}\left(\mathbf{0},,,{\sigma }_z^2,\mathbf{I}\right)$$

The vector $\mathbf{\Psi }\in {\mathbb{R}}^{G\times 1}$ is Gaussian residual noise

$$P\left(\mathbf{\Psi }\right)=\mathcal{N}\left(\mathbf{0},,,d,i,a,g,\left({\mathit{\tau }}^{-1}\right)\right)$$

We use the precision parameter ${\mathit{\tau }}_g$ to capture the gene-specific variation and define an independent conjugate prior for it

$$P\left(\mathit{\tau }\right)={\prod }_{g=1}^G\text{Gamma}\left({\alpha }_{\tau },,,{\beta }_{\tau }\right)$$

In these distributions, ${\sigma }_u$, ${\sigma }_v$, ${\sigma }_w$, ${\sigma }_z$, ${\alpha }_{\tau }$, and ${\beta }_t$ are hyperparameters $.$

Hence, we can imply the likelihood as

$$P\left(\mathrm{\Delta },\mathit{X},,|,,\mathit{W},,,\mathit{Z},,,\mathit{U},,,\mathit{V},,,\mathit{\tau }\right)={\prod }_{m=1}^M\mathcal{N}\left(\mathrm{\Delta },{\mathit{X}}_m,|\mathit{W},{\mathit{Z}}_m,+,{\mathit{U}}_{s(m)},+,{\mathit{V}}_{t(m)},,,d,i,a,g,\left({\mathit{\tau }}^{-1}\right)\right)$$

Inference

Given the observation variable $\mathrm{\Delta }\mathit{X}$, the latent variables $\mathit{W}$, $\mathit{Z}$, $\mathit{U}$, $\mathit{V}$, and $\mathit{\tau }$ are not mutually independent. Exact inference for the posterior distribution $P\left(\mathit{W},,,\mathit{Z},,,\mathit{U},,,\mathit{V},,,\mathit{\tau },,|,\mathrm{\Delta },\mathit{X}\right)$ requires integrals, which are computationally intractable. Hence, we use the strategy of variational inference to approximate the true posterior distributions. We introduce a parameterized variational distribution $Q_{\varphi }\left(\mathit{W},,,\mathit{Z},,,\mathit{U},,,\mathit{V},,,\mathit{\tau }\right)$, which can be factorized as

$$Q_{\varphi }\left(\mathit{W},,,\mathit{Z},,,\mathit{U},,,\mathit{V},,,\mathit{\tau },\right)=Q_{\varphi }\left(\mathit{W}\right)Q_{\varphi }\left(\mathit{Z}\right)Q_{\varphi }\left(\mathit{U}\right)Q_{\varphi }\left(\mathit{V}\right)Q_{\varphi }\left(\mathit{\tau }\right)$$

where $\varphi$ are all the variational parameters in each of the following distributions

$$Q_{\varphi }\left(\mathit{W}\right)=\prod _{g=1}^G\prod _{k=1}^K\text{LogNormal}\left({\mu }_{w_{\mathit{gk}}},,,{\sigma }_{w_{\mathit{gk}}}^2\right)$$

$$Q_{\varphi }\left(\mathit{Z}\right)=\prod _{k=1}^K\prod _{m=1}^M\mathcal{N}\left({\mu }_{z_{\mathit{km}}},,,{\sigma }_{z_{\mathit{km}}}^2\right)$$

$$Q_{\varphi }\left(\mathit{U}\right)=\prod _{g=1}^G\prod _{s=1}^S\mathcal{N}\left({\mu }_{u_{\mathit{gs}}},,,{\sigma }_{u_{\mathit{gs}}}^2\right)$$

$$Q_{\varphi }\left(\mathit{V}\right)=\prod _{g=1}^G\prod _{t=1}^T\mathcal{N}\left({\mu }_{v_{\mathit{gt}}},,,{\sigma }_{v_{\mathit{gt}}}^2\right)$$

$$Q_{\varphi }\left(\mathit{\tau }\right)=\prod _{g=1}^G\text{Gamma}\left({\alpha }_g,,,{\beta }_g\right)$$

Hence, our goal is to optimize the parameters $\varphi$ to find the best possible variational distribution $Q_{\varphi }$ that effectively approximates the posterior distribution. When we use Kullback–Leibler divergence $\text{KL}\left(Q_{\varphi },\left(\mathit{W},,,\mathit{Z},,,\mathit{U},,,\mathit{V},,,\mathit{\tau },\right),|P,\left(\mathit{W},,,\mathit{Z},,,\mathit{U},,,\mathit{V},,,\mathit{\tau },,|,,\mathrm{\Delta },\mathit{X}\right)\right)$ to measure the distance between these two probability distributions, the optimization problem can be converted to maximize the evidence lower bound (ELBO)

$$\text{ELBO}={\mathbb{E}}_{Q_{\varphi }}\left[\text{log},P,\left(\mathrm{\Delta },\mathit{X},,,,\mathit{W},,,\mathit{Z},,,\mathit{U},,,\mathit{V},,,\mathit{\tau }\right),-,\text{log},Q_{\varphi },\left(\mathit{W},,,\mathit{Z},,,\mathit{U},,,\mathit{V},,,\mathit{\tau }\right)\right]$$

The optimization and update of variational parameters $\varphi$ are performed by using stochastic gradient descent algorithm (Adam optimizer).

Datasets and pre-processing

Simulated data generation

To evaluate the performance of scStateDynamics, we used the “paths” method of the “splatSimulate” function within the Splatter package [22] to generate three scRNA-seq datasets that simulate distinct scenarios of tumor drug responses. To design the characteristics of these dynamic processes, we mainly manipulated the following parameters. We used the “group.prob” to control the sizes of cell clusters and used the “path.from” parameter to determine the dynamic relationships between clusters. Then, to model the extent of changes in cell states, we adjusted the “de.prob” parameter. Besides, we utilized the “path.skew” parameter to fine-tune the distribution of cells towards either the source or target population.

Data pre-processing

The pre-processing for the simulated and real data was performed based on the Scanpy [60], Seurat [62], and scCancer [63, 64] packages. First, data quality control is performed by filtering the potential lysed cells, low-quality cells, and doublets, based on the number of detected transcripts and genes, as well as the percentage of transcripts from mitochondrial genes. Besides, mitochondrial genes, ribosomal genes, and the genes expressed in fewer than three cells are also filtered. For the remaining cells and genes, we calculate the relative expression values by performing data normalization and log-transformation. Next, the highly variable genes of the data at two time points are identified, and their union set is used as the final selected genes. Then, we regress out the unwanted variance sources and perform data centering and scaling. Further, to project the cells of two time points into a shared low-dimensional space, we perform similar pre-processing steps on the combined expression matrix and conduct principal component analysis (PCA). These low-dimensional representations are subsequently used to calculate the distances between the cell states.

Downstream comparison and analyses

In this section, we provide the method details of the downstream analyses based on the results of scStateDynamics.

Comparing the inferred cell alignment relationships with lineage tracing information

We first screen the lineage barcodes that appear at both of the two time points, so that we could leverage the dynamic relationships represented by them to evaluate the confidence level of each cell subcluster flow. For example, if a lineage barcode is observed in $n_1$ cells within cluster Pre_1 at the pre-timepoint and $n_2$ cells within cluster Post_2 at post-timepoint, we interpret this as $n_1\ast n_2$ barcode evidences supporting the Pre_1- > Post_2 flow. To eliminate the influence of clone sizes, we normalize this count by dividing it by the square root of the product of the total number of cells labeled by this barcode at the two time points. Then, we integrate the evidences derived from all lineage barcodes and calculate the sum of their normalized counts for each flow. Further, to avoid the influence of cluster sizes, we also divide these summation results by the square root of the product of the cell numbers in the source cluster and target cluster for each flow. In this way, we obtain the final normalized lineage barcode counts, as shown in Fig. 3c.

Further, we also define two metrics to evaluate the correctness and completeness of the inferred cell alignments by comparing with the cell lineage barcodes, as shown in Fig. S4b of Additional file 1. We first determine the possible target clusters of each cell at the pre-timepoint based on its lineage barcode label. Then, by comparing them with the results inferred by the algorithms (scStateDynamics, CINEMA-OT, or CINEMA-OT-W), we define the correctness metric as the proportion of cells with correct fate inference. Besides, by measuring whether all the fates supported by lineage information in each clone are identified by the algorithms, we define the completeness metric as the average fate recognition rate across all cells at the pre-timepoint.

Besides, we conduct a comparative analysis between the performance of scStateDynamics and joint-clustering, a conventional cluster-level alignment method, based on the Watermelon lineage tracing dataset. In detail, we employ the BBKNN algorithm [65] to integrate the data from adjacent timepoints, and then apply Leiden graph-clustering method [66] to jointly cluster cells based on the Scanpy package [60]. Within each cluster, cells from pre- and post-timepoints are considered temporally aligned. To assess performance, we calculate the mean squared errors (MSEs) between the transition probability matrix (TPM) based on the lineage tracing labels (considered as ground truth) and the TPMs obtained through joint-clustering or scStateDynamics (Fig. S5).

Inter-cluster and intra-cluster heterogeneities analyses

To investigate the intrinsic heterogeneities at inter-cluster and intra-cluster levels, we conduct differential expression analysis between the clusters or the subclusters with distinct fates based on the Wilcoxon rank-sum test method in Scanpy package [60]. Then the significantly differentially expressed genes (DE genes) of each cluster or subcluster are subjected to enrichment analysis on the cancer hallmark pathways in MSigDB [67]. Furthermore, to compare the malignancy degrees among clusters or subclusters, we regard their DE genes as a signature to represent the subcluster and apply them to TCGA bulk samples with the same cancer type. We define the signature scores of the bulk samples by calculating the average expression values of the genes in the signature and utilize Cox proportional hazards regression to analyze the effect of the signatures on survival. In this way, the hazard ratios obtained can be used to quantify the malignancy degrees of the clusters or subclusters (Fig. 4f and Additional file 1: Fig. S7b).

To analyze the acquired heterogeneities, we calculate the change values of gene expression and pathway scores ($\mathrm{\Delta }\text{score}$ in Fig. 4g) by performing subtraction between the post-treatment and pre-treatment cells according to their inferred alignment relationships. Here, the log-normalized gene expression values are adopted. The pathway (signature) scores are defined as the average expression values of the genes within the pathways. In the case of the DTP signature [42], where weights are assigned to genes, we multiply these weights by the gene expression values before calculating the average. In addition, we also perform a Wilcoxon rank sum test on the DTP-related pathway scores to measure the increase or decrease in pathway activity induced by drug treatment (Fig. 4j, 4k and Additional file 1: Fig. S10).

Annotating the factors with signal pathways

We collect the cancer hallmark pathway and drug-related pathway information to provide biological annotations for the identified factors. For each factor $k$ in the decomposed gene-factor weight matrix $\mathit{W}$, we define its initial pathway score $s_{\text{init}}$ as the average weight of the genes in the respective pathway. Then to make the scores comparable, we generate a background distribution by randomly shuffling the gene weights 1000 times and calculate the scores as previously described. By utilizing the mean $\mu$ and standard deviation $\sigma$ of these 1000 scores, we transform the initial pathway score into its final z-score formation by $\frac{s_{\text{init}}-\mu }{\sigma }$, as shown in Fig. 5b.

Authors' contributions

J.G. conceived this study. W.G. developed the methods, performed the analyses, and drew the figures. J.G, W.G., and X.L. discussed the methods and results. D.W., F.Y., X.Z., and J.Y. provided suggestions for the experiments. W.G., X.L., N.Y., and Q.H. tested the package. W.G. and J.G. wrote the manuscript with the assistance of the other authors. J.G. and J.Y. supervised the project.

Funding

This work was supported by funding from the National Key Research and Development Program of China (nos. 2020YFA0712403 and 2021YFF1200901), the National Natural Science Foundation of China (NSFC) (nos. 62133006 and 92268104), the Tsinghua University Initiative Scientific Research Program (no. 20221080076), and the China Postdoctoral Science Foundation (no. 2022M721839).

Data availability

The datasets supporting the conclusions of this article are available in the Gene Expression Omnibus (GEO) repository. The Watermelon system lineage tracing scRNA-seq data were downloaded from Gene Expression Omnibus (GEO) with accession number GSE150949 [68]. The ReSisTrace data were download from GSE223003 [69]. The HCC data were downloaded from GSE151530 [70]. The GBM data were downloaded from GSE195682 [71]. The NSCLC data were downloaded from GSE134836 [72]. The Python package implementing the scStateDynamics and its tutorial material are freely available at GitHub (http://lifeome.net/software/scStateDynamics/) [73] and Zenodo [74]. The source code is released under the MIT license.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors Fan Yang and Jianhua Yao are from a commercial company (AI Lab, Tencent, Shenzhen, China). All the other authors declare no competing interests.