Markovian approaches to modeling intracellular reaction processes with molecular memory

Significance

Modeling intracellular processes has long relied on the markovian assumption. However, as soon as a reactant interacts with its environment, molecular memory definitely exists and its effects cannot be neglected. Since the Markov theory cannot translate directly to modeling and analysis of nonmarkovian processes, this leads to many significant challenges. We develop a formulation, namely the stationary generalized chemical-master equation, to model intracellular processes with molecular memory. This formulation converts a nonmarkovian question to a markovian one while keeping the stationary probabilistic behavior unchanged. Both a stationary generalized Fokker–Planck equation and a generalized linear noise approximation are further developed for the fast evaluation of fluctuations. These formulations can have broad applications and may help us discover new biological knowledge.

Abstract

Many cellular processes are governed by stochastic reaction events. These events do not necessarily occur in single steps of individual molecules, and, conversely, each birth or death of a macromolecule (e.g., protein) could involve several small reaction steps, creating a memory between individual events and thus leading to nonmarkovian reaction kinetics. Characterizing this kinetics is challenging. Here, we develop a systematic approach for a general reaction network with arbitrary intrinsic waiting-time distributions, which includes the stationary generalized chemical-master equation (sgCME), the stationary generalized Fokker–Planck equation, and the generalized linear-noise approximation. The first formulation converts a nonmarkovian issue into a markovian one by introducing effective transition rates (that explicitly decode the effect of molecular memory) for the reactions in an equivalent reaction network with the same substrates but without molecular memory. Nonmarkovian features of the reaction kinetics can be revealed by solving the sgCME. The latter 2 formulations can be used in the fast evaluation of fluctuations. These formulations can have broad applications, and, in particular, they may help us discover new biological knowledge underlying memory effects. When they are applied to generalized stochastic models of gene-expression regulation, we find that molecular memory is in effect equivalent to a feedback and can induce bimodality, fine-tune the expression noise, and induce switch.

Quantitative understanding of the dynamics of single living cells is one of the main goals of modern molecular-systems biology. Traditionally, modeling of intracellular biochemical processes is based on the markovian hypothesis, i.e., the stochastic motion of the reactants is uninfluenced by previous states, only by the current state. This memoryless property implies that markovian reaction kinetics can be described by poissonian processes with constant rates, which are characterized by exponential waiting-time distributions (1, 2). The mathematical tractability of markovian reaction processes enables great simplifications in problem formulation, leading to important successes in the description of many intracellular processes (1–4). These studies revealed that the characteristic parameters in the waiting-time distributions, as well as the complex properties of the intracellular process, can have a strong impact on the reaction kinetics.

However, intracellular reaction processes are not necessarily markovian but may be nonmarkovian. First, as a general rule, the dynamics of a given reactant resulting from its interactions with the environment cannot be described as a markovian process since this interaction can create “molecular memory” characterized by nonexponential waiting-time distributions. Second, the reduction of multistep reactions into a single step one may lead to a nonmarkovian process (5). Third, synthesis of a macromolecule would involve several small single- or multimolecular reaction steps (referring to Fig. 1A), creating a memory between individual events (6–9). This is evidenced by the fact that inactive phases of the promoter involving the prolactin gene in a mammalian cell are differently distributed, showing strong memory (10). More generally, the complex control process of gene expression, which would involve several repressors, transcription factors (TFs), and mediators as well as chromatin remodeling or changes in supercoiling, can generate nonexponential time intervals between transcription windows. Indeed, molecular memory, which can result in nonmarkovian (or nonpoissonian) reaction kinetics, has been confirmed by the increasing availability of time-resolved data on different kinds of interactions (10–16).

Fig. 1.

A framework for analysis of stochastic reaction processes with molecular memory. (A) A possibly involved multistep multimolecular reaction process with exponential waiting times is mapped onto a single-step reaction process with nonexponential waiting times, where the initial and final states are observable or measurable, but the intermediate states, which are consequences of single-molecular or multimolecular reactions, would be unknown or have not been unspecified. Note that $S_i\to S_{i+1}$ does not represent a reaction but represents the transformational relation between states $S_i$ and $S_{i+1}$, which possibly involves single- or multimolecular reactions. (B) Our theory shows that a nonmarkovian problem can be converted into a markovian one via an sgCME.

The extensive existence of molecular memory raises important yet unsolved questions: e.g., how does this memory affect nonmarkovian reaction kinetics? In what way and how accurately do life forms achieve the order required to develop and sustain their lives from the disordered reaction events? These questions were partially addressed in a seminal paper (6) wherein Pedraza and Paulsson analyzed a queuing model of stochastic gene expression with molecular memory, which was further generalized and analyzed (7, 17, 18). Recently, Park et al. (19) analyzed another class of queuing models of stochastic gene expression with molecular memory and presented the chemical fluctuation theorem, which provides an accurate relationship between the environment-coupled chemical dynamics of gene expression and gene-expression variability. These analyzed models are based on queuing theory, which is useful in treating simple single-molecular reaction networks such as birth–death processes, but it seems difficult to extend it to complex bi- or multimolecular reactions, which are a characteristic of many intracellular reaction networks. On the other hand, the continuous time random walk (CTRW) framework provides a different modeling way for stochastic reaction processes and has been extensively used in modeling and analysis of stochastic processes (20–22). Despite this, we still lack a general theory for modeling and analysis of general reaction networks with intrinsic waiting-time distributions. Since the sources of nonmarkovianity and network topology can jointly influence the behavior of the whole biochemical system, the reliable tools and mathematical machinery of the Markov theory do not translate directly to modeling and analysis of nonmarkovian reaction processes, leading to many significant challenges. It is needed to develop new methods to characterize nonmarkovian reaction kinetics.

In recent years, the interest in nonmarkovian processes within scientific communities has been blossoming, from the viewpoints of experimental observations (10–16), mathematical modeling (5–7, 17–30), and numerical simulation (30–38). However, because of difficulties in treating nonmarkovian reaction kinetics, only a few exact master equations are known in the literature, most of which are rather formal, or concerning specific systems or stochastic processes (6, 7, 17–20, 39, 40). The aim of this paper is to fill this gap by introducing useful formulations for a general biochemical-reaction system with molecular memory. The basic idea is that we first construct a topology-equivalent, memoryless reaction network by introducing an effective transition rate (ETR) for each reaction in the original nonmarkovian system and then establish an effective master equation for the equivalent reaction network, whose stationary equation is termed as a stationary generalized chemical-master equation (sgCME). The sgCME exactly captures the stationary probabilistic behavior of the original nonmarkovian system. In other words, the original nonmarkovian question is converted into a markovian one (Fig. 1B), greatly simplifying analysis of nonmarkovian processes. Based on the sgCME, we further develop another 2 useful techniques for studying stochastic reaction processes on networks: stationary generalized Fokker–Planck equation (sgFPE) and generalized linear noise approximation (gLNA), each enabling fast yet effective evaluation of fluctuations, including the effect of molecular memory.

We demonstrate the power of the above 3 techniques by analyzing 4 examples: a generalized model of constitutive gene expression, a generalized model of gene self-regulation, a generalized ON-OFF model, and a generalized model of genetic toggle switch, each considering molecular memory. These examples are chosen so that analytical or numerical approximations of solutions to the sgCMEs can easily be obtained, and these approximations are then used to clarify the origins of nonmarkovianity and fluctuations in each case and to trace the effects of different parameters on the stochastic properties of the systems. The results reveal the importance of the effect of molecular memory on reaction kinetics.

Methods

A Theoretical Framework for Reaction Processes on a Network.

Before presenting our sgCME, let us give the representation of a general reaction network in terms of intrinsic waiting-time distributions. In order to cast the dynamics of $N$ different species that participate in $L$ different reactions (numbered by $i$ with $1\le i\le L$) into a CTRW framework, we first define a state space. Denote by $X_j$ the chemical species and by $n_j$ the corresponding particle number, where $1\le j\le N$. Let $\mathit{n}={\left(n_1,\dots n_N\right)}^{\text{T}}$ represent the state vector of particle numbers, where $\text{T}$ denotes transpose. Denote by $r_{ij}$ and $s_{ij}$ the stoichiometric coefficients for the $i\text{th}$ reaction, i.e., $r_{ij}$ and $s_{ij}$ represent the loss and gain in particle number $n_j$ during the $i\text{th}$ reaction, respectively. On the other hand, a single event of reaction $i$ is characterized by the intrinsic waiting time ${\tau }_i$, whose probability density function (PDF), denoted by ${\psi }_i\left(t;\mathit{n}\right)$, depends, in general, on the system state $\mathit{n}$. To that end, the above reaction network can be represented by

$${\displaystyle \sum _{j=1}^N{r}_{ij}{\mathit{X}}_j}\stackrel{{\psi }_i\left(t;\mathit{n}\right)}{\to }{\displaystyle \sum _{j=1}^N{s}_{ij}{\mathit{X}}_j},i=\mathrm{1,2},\cdots ,L.$$ [1]

Note that ${\nu }_{ji}=s_{ij}-r_{ij}$ represents the net change of $X_i$ in the $i\text{th}$ reaction. Function ${\psi }_i\left(t;\mathit{n}\right)$ is called the intrinsic waiting-time distribution for reaction $i$ (20, 39, 40). We emphasize that such a waiting-time distribution is an extension of exponential waiting-time distribution in the markovian models but is different from waiting-time distributions in queuing theory (21). In addition, intrinsic waiting-time distributions are suitable to the description of any reactions, including bimolecular or multimolecular reactions, so they are also extensions of delay distributions introduced in refs. 5 and 28, wherein the authors proposed a methodology to represent chains of single-molecular reactions by simpler, reduced models.

If all of the reaction events happen in a markovian (or memoryless) manner, then ${\psi }_i\left(t;\mathit{n}\right)$ reduces to ${\psi }_i\left(t;\mathit{n}\right)=a_i\left(\mathit{n}\right)e^{-a_i\left(\mathit{n}\right)t}$ for all $t\ge 0$ (hereafter, we always assume $t\ge 0$), where $a_i\left(\mathit{n}\right)$ is the reaction-propensity function for reaction $i$. This expression is natural since exponential waiting-time distributions, which have a clear biophysical foundation (41), are a main characteristic of markovian reaction processes. In contrast to the times until the next molecular events happen in single-step processes (elementary reactions), the waiting times between reaction events are not exponentially distributed in general. If the process goes through a series of (identical) exponential steps, the waiting times will be gamma-distributed (42). If we consider an intrinsic waiting-time distribution of the gamma type: ${\psi }_i\left(t;\mathit{n}\right)=\left({\left({\lambda }_i\left(\mathit{n}\right)\right)}^{L_i}/\mathrm{\Gamma }\left(k_i\right)\right)t^{L_i-1}{e}^{-{\lambda }_i\left(\mathit{n}\right)t}$, where $L_i$ may be understood as the number of small yet possibly unspecified reaction steps, then $L_i=1$ corresponds to an exponential waiting-time distribution, whereas $L_i\ne 1$ (called memory index) to a nonexponential waiting-time distribution with the noise intensity (defined as the ratio of the variance over the squared mean) being $1/L_i$. Note that for a reaction network, as long as there is a reaction such that the corresponding memory index is not equal to 1, the whole reaction kinetics is nonmarkovian. In this paper, we will only consider intrinsic waiting-time distributions of $L_i=1$ (exponential) and $L_i\ne 1$ (nonexponential).

sgCME.

Effective transition rates.

For convenience, let ${\mathrm{\Psi }}_i\left(t;\mathit{n}\right)$ be the cumulative distribution function of ${\psi }_i\left(t;\mathit{n}\right)$, i.e., ${\mathrm{\Psi }}_i\left(t;\mathit{n}\right)={\displaystyle {\int }_0^t{\psi }_i\left(t\prime ;\mathit{n}\right)dt\prime }$. If we define ${\phi }_i\left(t;\mathit{n}\right)={\psi }_i\left(t;\mathit{n}\right){\displaystyle {\prod }_{j\ne i}\left[1-{\mathrm{\Psi }}_j\left(t;\mathit{n}\right)\right]}$, then ${\phi }_i\left(n;t\right)dt$ represents the probability that the $i\text{th}$ reaction happens and the reaction waiting time is in interval $\left[t,t+dt\right]$. Note that the conservative condition ${\displaystyle {\sum }_{i=1}^L{\displaystyle {\int }_0^{\infty }{\phi }_i\left(t\prime ;\mathit{n}\right)dt\prime }}=1$ always holds.

Now, we introduce a memory function for reaction $i$, denoted by $M_i\left(t;\mathit{n}\right)$, which is defined through the Lapalace transform: ${\stackrel{\sim }{M}}_i\left(s;\mathit{n}\right)=s{\stackrel{\sim }{\phi }}_i\left(s;\mathit{n}\right)/\left[1-{\displaystyle {\sum }_{j=1}^L{\stackrel{\sim }{\phi }}_j\left(s;\mathit{n}\right)}\right]$, where $1\le i\le L$, and the above bar represents the Laplace transform of a function, e.g., ${\stackrel{\sim }{\phi }}_i\left(s;\mathit{n}\right)={\displaystyle {\int }_0^{\infty }{e}^{-st}{\phi }_i\left(t;\mathit{n}\right)dt}$. To help the reader understand function $M_i\left(t;\mathit{n}\right)$, let us consider waiting-time distributions of special forms: ${\psi }_i\left(t;\mathit{n}\right)=a_i\left(\mathit{n}\right)e^{-a_i\left(\mathit{n}\right)t}$, with $a_i\left(\mathit{n}\right)$ being the reaction-propensity function for reaction $i$ $\left(1\le i\le L\right)$, implying that all of the reaction events happen in markovian manners. Then, by the expression of ${\stackrel{\sim }{M}}_i\left(s;\mathit{n}\right)$ in combination with the inverse of the Laplace transform, we can show $M_i\left(t;\mathit{n}\right)=a_i\left(\mathit{n}\right)$ for all $t\ge 0$. In this case, the memory functions reduce to the common reaction-propensity functions.

Interestingly, we find that the limit $\underset{s\to 0}{\lim }{\stackrel{\sim }{M}}_i\left(s;\mathit{n}\right)$ exists, given waiting-time distributions for a reaction network. Moreover, if this limit is denoted by $K_i\left(\mathit{n}\right)$, then it can be analytically expressed as (see SI Appendix for details)

$$K_i\left(\mathit{n}\right)=\frac{{\displaystyle {\int }_0^{+\infty }{\psi }_i\left(t;\mathit{n}\right){\displaystyle \prod _{j\ne i}\left[1-{\text{Ψ}}_j\left(t;\mathit{n}\right)\right]}dt}}{{\displaystyle {\int }_0^{+\infty }{\displaystyle {\prod }_{j=1}^L\left[1-{\text{Ψ}}_j\left(t;\mathit{n}\right)\right]}dt}}\hspace{0.17em},$$ [2]

which is termed as the ETR for reaction $i$, where $i=1,\cdots ,L$. If there exists some subscript $i_0$ such that ${\psi }_{i_0}\left(t;\mathit{n}\right)=a_{i_0}\left(\mathit{n}\right)e^{-a_{i_0}\left(\mathit{n}\right)t}$ for all $t\ge 0$, i.e., if the waiting time of the $i_0\text{th}$ reaction follows an exponential distribution, function $K_{i_0}\left(\mathit{n}\right)$ reduces to the common reaction-propensity function, i.e., $K_{i_0}\left(\mathit{n}\right)=a_{i_0}\left(\mathit{n}\right)$ (see SI Appendix for details). If waiting-time distributions for all of the reactions are exponential, the nonmarkovian reaction network reduces to a markovian reaction one. Therefore, ETRs are extensions of reaction-propensity functions. However, different from function $a_i\left(\mathit{n}\right)$ that is a polynomial of $\mathit{n}$, function $K_i\left(\mathit{n}\right)$ is in general a rational function of $\mathit{n}$. For example, consider a generalized birth–death process: $Ø\stackrel{{\psi }_1\left(t;n\right)}{\to }X$ and $X\stackrel{{\psi }_2\left(t;n\right)}{\to }Ø$, where $n$ represents the number of $X$ molecules. If ${\psi }_2\left(t;n\right)=n{\lambda }_2{e}^{-n{\lambda }_{2}t}$ and ${\psi }_1\left(t;n\right)=\left({\left({\lambda }_1\right)}^{L_1}/\mathrm{\Gamma }\left(k_1\right)\right)t^{L_1-1}{e}^{-{\lambda }_{1}t}$ with $L_1$ being a positive integer, we can show $K_2\left(n\right)=n{\lambda }_2$ and $K_1\left(n\right)={\left({\lambda }_1\right)}^{L_1}\left(n{\lambda }_2\right)/\left({\left({\lambda }_1+n{\lambda }_2\right)}^{L_1}-{\left({\lambda }_1\right)}^{L_1}\right)$, which is apparently a rational function of $n$ if $L_1>1$. SI Appendix also gives the explicit expressions of ETRs in the case of environmental perturbations or external noise that can lead to stochastic reaction delays.

Effective CME.

For the above (nonmarkovian) reaction network, if we let $P\left(\mathit{n};t\right)$ be the probability that the system is in state $\mathit{n}$ at time $t$, then according to the CTRW theory (20, 39, 40), we can derive the following CME expressed in Laplace transforms

$$s\stackrel{\sim }{P}\left(\mathit{n};s\right)=P\left(\mathit{n};0\right)+{\displaystyle \sum _{i=1}^L{\stackrel{\sim }{M}}_i\left(s;\mathit{n}-{\mathit{v}}_i\right)\stackrel{\sim }{P}\left(\mathit{n}-{\mathit{v}}_i;s\right)}-{\displaystyle \sum _{i=1}^L{\stackrel{\sim }{M}}_i\left(s;\mathit{n}\right)\stackrel{\sim }{P}\left(\mathit{n};s\right)}.$$ [3]

By the final value theorem (42), we know that if the limit $\underset{t\to \infty }{\lim }f\left(t\right)$ exists, 2 limits $\underset{t\to \infty }{\lim }f\left(t\right)$ and $\underset{s\to 0}{\lim }s\stackrel{\sim }{f}\left(s\right)$ are equal, i.e., $\underset{t\to \infty }{\lim }f\left(t\right)=\underset{s\to 0}{\lim }s\stackrel{\sim }{f}\left(s\right)$. Multiplying $s$ on both sides of Eq. 3, letting $s\to 0$ and using $K_i\left(\mathit{n}\right)=\underset{s\to 0}{\lim }{\stackrel{\sim }{M}}_i\left(s;\mathit{n}\right)$, we can obtain the following stationary equation (SI Appendix):

$${\displaystyle \sum _{i=1}^L\left[K_i\left(\mathit{n}-{\mathit{\nu }}_i\right)P\left(\mathit{n}-{\mathit{\nu }}_i\right)-K_i\left(\mathit{n}\right)P\left(\mathit{n}\right)\right]}=0,$$ [4]

which is called the sgCME, where $P\left(\mathit{n}\right)$ is the stationary distribution corresponding to $P\left(\mathit{n};t\right)$ and is assumed to exist (numerical simulations verifies this, referring to Fig. 2C). Eq. 4 with Eq. 2 is one of the main results of this paper.

Fig. 2.

Analysis of a generalized model of constitutive gene expression without self-regulation, i.e.,${\psi }_1\left(t;n\right)$ is independent of $n$. (A) Schematic representation of the model. (B) Transitions between states. (C) Protein distributions at several different time points, where curves with empty circles correspond to the original nonmarkovian model, whereas colored curves correspond to the constructed markovian model. (D) Influences of memory index $L_1$ $\left(L_2\right)$ on mean protein levels, where empty circles represent the results obtained by the modified “first-reaction” Gillespie algorithm (41), whereas the lines represent the results obtained by theoretical predictions. (E) Stationary protein distributions for different values of $L_1$ and $L_2$. Parameter values are set as: $L_1=4,{\lambda }_1=120$, and $L_2=1,{\lambda }_2=1$ (C) and ${\lambda }_1=20L_1,{\lambda }_2=L_2$ (D and E).

On the other hand, we can use $K_i\left(\mathit{n}\right)$ to construct a reaction network. This is done by taking $K_i\left(\mathit{n}\right)$ as the reaction-propensity function for the $i\text{th}$ reaction in an topologically equivalent reaction network with the same substrates but without molecular memory. For this reaction network, we can establish its CME. In fact, if we let $Q\left(\mathit{n};t\right)$ be the probability that the system is in state $\mathit{n}$ at time $t$, the corresponding CME takes the form

$$\frac{\partial Q\left(\mathit{n};t\right)}{\partial t}={\displaystyle \sum _{i=1}^L\left[K_i\left(\mathit{n}-{\mathit{\nu }}_i\right)Q\left(\mathit{n}-{\mathit{\nu }}_i;t\right)-K_i\left(\mathit{n}\right)Q\left(\mathit{n};t\right)\right]},$$ [5]

which is called an effective CME for the original nonmarkovian reaction system, where “effective” will be interpreted below.

Notably, the stationary equation corresponding to Eq. 5 is exactly the same as Eq. 4 except for notation, implying that the stationary probabilistic behavior of the original nonmarkovian reaction system is exactly the same as that of the constructed markovian reaction network. In this sense, the original nonmarkovian problem is converted to a markovian one. However, there would exist differences in dynamic probability behavior between the 2 reaction networks, referring to Fig. 2C, which also verifies that the stationary distribution indeed exists even in the presence of molecular memory. Despite this difference, Eq. 5 provides a way for studying complex nonmarkovian reaction kinetics.

Generalized Linear Noise Approximation.

In analysis of reaction networks, an extensively used technique is the linear noise approximation (LNA) (1, 2). Here, we derive a generalized LNA (gLNA) for the above general reaction system with arbitrary waiting-time distributions.

First, the rate equations corresponding to the constructed-above markovian reaction network read

$$\frac{d{x}_k}{dt}={\displaystyle \sum _{i=1}^L{\nu }_{ik}{K}_i\left(\mathit{x}\right)},k=\mathrm{1,2},\cdots ,N,$$ [6]

where $\mathit{x}={\left(x_1,\cdots ,x_N\right)}^{\text{T}}$, $x_k$ represents the concentration of reactive species $X_k$, i.e., $x_k=\underset{\mathrm{\Omega }\to \infty }{\lim }{n}_k/\mathrm{\Omega }$ with $\mathrm{\Omega }$ being the volume of the system, and $K_i\left(\mathit{x}\right)$ is given by Eq. 2 if $\mathit{n}$ is replaced with $\mathit{x}$. Let $\mathbf{S}=\left({\nu }_{ij}\right)$ be the stoichiometric matrix and $\mathit{K}\left(\mathit{x}\right)={\left(K_1\left(\mathit{x}\right),\cdots ,K_L\left(\mathit{x}\right)\right)}^{\text{T}}$ be a column vector of ETRs. Then, the algebraic equation

$$\mathbf{S}\mathit{K}\left(\mathit{x}\right)=\mathbf{0}$$ [7]

determines the steady state of the deterministic system described by Eq. 6, which is denoted by ${\mathit{x}}_s$.

Then, we adopt the Ω-expansion method (1, 2) to derive an algebraic equation for covariance matrix. Write $\mathit{n}=\mathrm{\Omega }\mathit{x}+{\mathrm{\Omega }}^{1/2}\mathit{z}$, where $\mathit{z}$ is a vector of random variables. Under this transform, the original $P\left(\mathit{n};t\right)$ becomes another $\mathrm{\Pi }\left(\mathit{z};t\right)$. According to SI Appendix, we can derive the following Lyapunov matrix equation:

$${\mathbf{{\rm A}}}_{\text{S}}{\mathbf{\Sigma }}_{\text{S}}+{\mathbf{\Sigma }}_{\text{S}}{\mathbf{{\rm A}}}_{\text{S}}^{\text{T}}+{\mathbf{D}}_{\text{S}}=\mathbf{0},$$ [8]

where ${\mathbf{{\rm A}}}_{\text{S}}=\left(A_{ij}\right)$ is a matrix with elements $A_{ij}={\displaystyle \sum _{k=1}^L{\nu }_{ki}\frac{\partial K_k\left({\mathit{x}}_{\text{s}}\right)}{\partial x_j}}=\frac{\partial \left[{\mathbf{S}}_i\mathit{K}\left({\mathit{x}}_s\right)\right]}{\partial x_j}$, ${\mathbf{D}}_s=\mathbf{B}{\mathbf{B}}^{\text{T}}$ is a noise matrix with elements ${\left[\mathbf{B}{\mathbf{B}}^{\text{T}}\right]}_{ij}={\displaystyle \sum _{k=1}^L{\nu }_{ki}{\nu }_{kj}{K}_k\left({\mathit{x}}_{\text{s}}\right)}={\left[{\mathbf{S}}^{\text{T}}\text{diag}\left(K_1\left({\mathit{x}}_s\right),\cdots ,K_L\left({\mathit{x}}_s\right)\right)\mathbf{S}\right]}_{ij}$, and ${\displaystyle {\sum }_s=\left(\langle {\left(\mathit{x}-{\mathit{x}}_s\right)}^{\text{T}}\left(\mathit{x}-{\mathit{x}}_s\right)\rangle \right)}$ is a covariance matrix to be unknown. Note that the diagonal elements of ${\mathbf{\Sigma }}_{\text{S}}$ represent the variances of the random variables, and the vector of the mean concentrations of the reactive species is approximately given by $\langle \mathit{x}\rangle \approx {\mathit{x}}_s$.

We point out that the above analysis framework is convenient for both clarifying the origins of nonmarkovianity (including fluctuations) and tracing the effects of different parameters on the stochastic properties of the underlying systems. SI Appendix provides details for examples of analysis by the gLNA. We point out that a slow-scale LNA (ssLNA) has been developed, which describes a class of nonmarkovian systems stemming from timescale separation (43). Specifically, starting with a markovian system composed of fast and slow species, the authors of that paper derived the LNA for the nonmarkovian system, which describes only the slow species (the observables).

sgFPE.

As an effective approximation of the CME in some situations, the Fokker–Planck equation (FPE) has extensively been used (1, 2, 44, 45), mainly because the latter is more easily analyzed and can often provide more intuitive understanding of a biochemical system than the former. However, the FPE has not been established in the presence of molecular memory. Here, we derive a sgFPE for a general nonmarkovian reaction network with arbitrary waiting-time distributions.

First, although Eq. 3 holds for the discrete variables, it also holds for the corresponding continuous variables. Second, Taylor-expanding the CME in the case of continuous variables to the second-order term yields

$$\begin{array}{l}s\stackrel{\sim }{P}\left(\mathit{x},s\right)-\stackrel{\sim }{P}\left(\mathit{x},0\right)\approx -{\displaystyle \sum _{i=1}^L{\displaystyle \sum _{k=1}^N{\nu }_{ik}\frac{\partial }{\partial x_k}\left[{\stackrel{\sim }{M}}_i\left(s;\mathit{x}\right)\stackrel{\sim }{P}\left(\mathit{x},s\right)\right]}}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+\frac{1}{2}{\displaystyle \sum _{i=1}^L{\displaystyle \sum _{k,l=1}^N{\nu }_{ik}{\nu }_{il}\frac{{\partial }^2}{\partial x_k\partial x_l}\left[{\stackrel{\sim }{M}}_i\left(s;\mathit{x}\right)\stackrel{\sim }{P}\left(\mathit{x},s\right)\right]}}\end{array}.$$ [9]

Multiplying $s$ on both sides of Eq. 9, letting $s\to 0$ and making use of the facts: $K_i\left(\mathit{n}\right)=\underset{s\to 0}{\lim }{\stackrel{\sim }{M}}_i\left(s;\mathit{n}\right)$ and $P\left(\mathit{x}\right)=\underset{s\to 0}{\lim }s\stackrel{\sim }{P}\left(\mathit{x},s\right)$, we can arrive at the following sgFPE:

$$-{\displaystyle \sum _{k=1}^N\frac{\partial }{\partial x_k}\left[{\displaystyle \sum _{i=1}^L{\nu }_{ik}{K}_i\left(\mathit{x}\right)}P\left(\mathit{x}\right)\right]}+\frac{1}{2}{\displaystyle \sum _{k,l=1}^N\frac{{\partial }^2}{\partial x_k\partial x_l}\left[{\displaystyle \sum _{i=1}^L{\nu }_{ik}{\nu }_{il}{K}_i\left(\mathit{x}\right)}P\left(\mathit{x}\right)\right]}=0.$$ [10]

In the next section, we will use Eq. 10 to analyze generalized stochastic models of gene expression and obtain some interesting results on the effect of molecular memory.

In the following section, we will apply the above general theory to 4 gene models: a generalized model of constitutive gene expression, a generalized model of gene self-regulation, a generalized ON-OFF model, and a generalized model of genetic toggle switch, where by “generalized,” we mean that each model considers molecular memory or nonmarkovianity. Then, we discover biological knowledge, e.g., molecular memory is in effect equivalent to a feedback and can induce bimodality, fine-tune the expression noise, and induce switch. (The related data will be available from the corresponding author upon request.)

Results

The Effect of Molecular Memory Is Equivalent to the Introduction of a Feedback.

Understanding how a gene is turned on at a mechanistic level has been one of the big challenges in molecular biology and has received extensive attention over decades. Identifying the actual sequence of events during gene expression and establishing the method of recruitment have turned out to be a surprisingly difficult task (46). Here, we introduce a generalized model to mimic complex biochemical processes underlying gene expression, referring to Fig. 2A, where the proteins are assumed to be produced instantaneously after messenger RNAs (mRNAs) are produced. Fig. 2B is a schematic representation of transitions between protein states with time. Let ${\psi }_1\left(t;n\right)$ and ${\psi }_2\left(t;n\right)$ be waiting-time distributions for protein synthesis and degradation, respectively, where $n$ represents the number of protein molecules.

First, consider the case without regulation but with molecular memory. Two waiting-time distributions are set as ${\psi }_1\left(t;n\right)={{\displaystyle {\lambda }_1}}^{L_1}/\mathrm{\Gamma }\left(L_1\right)t^{L_1-1}{e}^{-{\lambda }_{1}t}$ and ${\psi }_2\left(t;n\right)={\left(n{\lambda }_2\right)}^{L_2}{t}^{L_2-1}{e}^{-n{\lambda }_{2}t}$, where ${\lambda }_1$ and ${\lambda }_2$ are positive constants (which may be understood as the mean synthesis and degradation rates, respectively). Before presenting analytical results, we perform numerical calculation with results shown in Fig. 2, where Fig. 2C demonstrates that the stationary protein distribution indeed exists even in the presence of molecular memory. Hereafter, we will vary memory index ($L_1$ or $L_2$) while keeping the constant average time between successive reactions by scaling parameter ${\lambda }_1$ $\left({\lambda }_2\right)$ appropriately with $L_1$ $\left(L_2\right)$, i.e., keeping the ratio $L_i/{\lambda }_i$ fixed.

As pointed out above, the stationary probabilistic behavior of the original nonmarkovian reaction system is exactly the same as that of the constructed markovian reaction network, but there would exist differences in dynamic probability behavior between the 2 networks. Fig. 2C shows that 2 dynamic distributions are different at the initial stage, but this difference gradually reduces and finally disappears with time. Fig. 2E demonstrates how molecular memory (i.e., $L_1>1$ or $L_2>1$) affects the stationary protein distributions, whereas Fig. 2D shows that $L_1$ always decreases the mean protein number but $L_2$ always increases this number. These numerical results imply that the effect of molecular memory is equivalent to the introduction of a feedback. For this, we give an intuitive interpretation. First, note that in simulation, we keep the average waiting times between successive reactions constant by scaling ${\lambda }_1\left({\lambda }_2\right)$ appropriately with $L_1\left(L_2\right)$, i.e., we keep ratios $L_1/{\lambda }_1\equiv 1/{\lambda }_1^{\ast }$ and $L_2/{\lambda }_2=1/{\lambda }_1^{\ast }$ constant. This implies that the average of waiting times remains unchanged, but their variances decrease with increasing $L_1\left(L_2\right)$. Second, the waiting-time distribution will collapse onto a Dirac delta function due to zero variance if $L_1\left(L_2\right)$ tends to infinity. Third, note that the reaction event that actually occurs is the one whose waiting time is minimum. Therefore, for a fixed $L_2$, if $L_1$ increases, the variability in birth waiting times decreases. As a result, the probability of birth events decreases and hence the effective reaction rate decreases. Similarly, if the protein-decay rate decreases with $L_2$, the mean protein number will increase. Finally, we emphasize that such a memory-induced feedback stems from fluctuations in waiting times rather than changes in their means.

In order to obtain analytical results, we consider the case of $L=L_1>1$ and $L_2=1$. In this case, 2 ERTs are given by $K_1\left(n\right)=n{\lambda }_2{{\displaystyle {\lambda }_1}}^L/\left[{\left({\lambda }_1+{\lambda }_{2}n\right)}^L-{{\displaystyle {\lambda }_1}}^L\right]$ with $K_1\left(0\right)={\lambda }_1/L$ and $K_2\left(n\right)=n{\lambda }_2$. Note that function $f\left(x\right)=x{\lambda }_2{{\displaystyle {\stackrel{\sim }{\lambda }}_1}}^L/\left[{\left({\stackrel{\sim }{\lambda }}_1+x\right)}^L-{{\displaystyle {\stackrel{\sim }{\lambda }}_1}}^L\right]$ with $\stackrel{\sim }{\lambda }={\lambda }_1/{\lambda }_2$ has the following properties: $f\left(0\right)=0$ and the derivative $f\prime \left(x\right)$ is less than zero, i.e., $f\prime \left(x\right)<0$ for all $x\ge 0$. Therefore, the effect of molecular memory is equivalent to the introduction of a negative feedback. In addition, we can show that the stationary protein distribution is given by $P\left(n\right)={\left({{\displaystyle {\stackrel{\sim }{\lambda }}_1}}^L\right)}^n/\left[n!{\left(a_1\right)}_n\cdots {\left(a_{L-1}\right)}_n\rho \right]$ with $\rho ={\left[{}_1{{\displaystyle F}}_L\left(\mathrm{1,1},a_1,\cdots ,a_{L-1};{{\displaystyle {\stackrel{\sim }{\lambda }}_1}}^L\right)\right]}^{-1}$, where symbol ${}_1{{\displaystyle F}}_L\left(b,c_1,\cdots ,c_L;z\right)$ is a confluent hypergeometric function (47), and symbol ${\left(c\right)}_n$ is defined as ${\left(c\right)}_n=c\left(c+1\right)\cdots \left(c+n-1\right)$. Constants $a_1,\cdots ,a_{L-1}$ are determined by comparing the coefficients for the same power of $x$ in the equality of ${\stackrel{\sim }{\lambda }}^{L-1}+x{\stackrel{\sim }{\lambda }}^{L-1}+\cdots +x^{L-1}=\left(x+a_1\right)\cdots \left(x+a_{L-1}\right)$, where $a_1,\cdots ,a_{L_1-1}$ are assumed to be real (the case of complex roots can be similarly analyzed). This form of the distribution is similar to that of the stationary mRNA distribution in a stochastic gene model with a DNA loop (48).

Molecular Memory Can Induce Bimodality.

In the markovian case, gene self-regulating systems have been extensively studied, and some analytical results have been obtained (49–52). However, gene self-regulating processes are in general nonmarkovian as pointed out in the introduction, raising the question of how nonmarkovianity impacts gene-product distributions. To address this question, let us consider the following gene model: $\text{ON}\stackrel{{\psi }_i\left(t;\mathit{n}\right)}{\to }\text{ON}+\text{protein}$, $\text{protein}\stackrel{{\psi }_i\left(t;\mathit{n}\right)}{\to }{\psi }_2(t;\mathit{n})Ø$, where $n$ represents the number of protein molecules, and ${\psi }_1\left(t;n\right)$ and ${\psi }_2\left(t;n\right)$ are the intrinsic waiting-time distributions for protein synthesis and degradation, which take the forms ${\psi }_1\left(t;n\right)={\left({\lambda }_1\left(n\right)\right)}^L/\mathrm{\Gamma }\left(L\right)t^{L-1}{e}^{-{\lambda }_1\left(n\right)t}$ and ${\psi }_2\left(t;n\right)=n{\lambda }_2{e}^{-n{\lambda }_{2}t}$, where ${\lambda }_1\left(n\right)=\left[{\mu }_0+{\mu }_1{\left(n/K\right)}^H\right]/\left[1+{\left(n/K\right)}^H\right]$ is a regulation function of Hill type, $H$ is the Hill coefficient, and ${\mu }_0$, ${\mu }_1$ (representing the feedback strength), $K$, and ${\lambda }_2$ are positive parameters. This setting enables us to have a description in terms of one promoter state rather than 2 switching states but also hides the fact that proteins produced by the gene in the ON state bind to the gene and take it back to the OFF state. If promoter switching between ON and OFF states is very quick and if the protein–DNA binding rate is not much larger than the unbinding rate, the use of an effective Hill-type function is reasonable (53).

Similar to the case of no regulation, 2 effective transition rates, $K_1\left(n\right)$ and $K_2\left(n\right)$, can also be given analytically. Although the exact stationary protein distribution cannot be analytically given, the solution to Eq. 10, i.e., the stationary distribution of continuous variables, can be approximately expressed as (see SI Appendix for details)

$$P\left(x\right)=\mathcal{N}\frac{{\left[{\lambda }_1\left(x\right)+{\lambda }_{2}x\right]}^L-{\left[{\lambda }_1\left(x\right)\right]}^L}{{\left[{\lambda }_1\left(x\right)+{\lambda }_{2}x\right]}^L{\lambda }_{2}x}\exp \left(-2x+{\displaystyle {\int }_0^{x}4{\left[\frac{{\lambda }_1(x\prime )}{{\lambda }_1(x\prime )+{\lambda }_{2}x\prime }\right]}^{L}dx\prime }\right),$$ [11]

where $\mathcal{N}$ is a normalization factor.

Numerical results are demonstrated in Fig. 3, where Fig. 3 A and C shows how the number of the most probable protein molecules obtained by a statistical method depends on memory index $L$, whereas Fig. 3 B and D demonstrates that the stationary distribution predicted by Eq. 11 can well approximate that obtained by solving the sgCME. Note that $L=1$ corresponds to the markovian case (red dashed lines), whereas $L<1$ or $L>1$ to the nonmarkovian case. In Fig. 3 A and C, the shadowed areas represent that bimodality exists. Also note that the number of the most probable protein molecules shown in Fig. 3 A and C is in agreement with that predicted by a deterministic system (SI Appendix). From Fig. 3 B and D, we observe that bimodal protein distribution exists only for moderately large $L$ with $L>1$ or only for moderately small $L$ with $L<1$. Since memory index $L$ determines the strength of molecular memory, Fig. 3 implies that the molecular memory only with moderate strengths can induce bimodality. Here, we also give an intuitive explanation for the demonstrated numerical results. First, the occurrence of bimodality needs appropriate nonlinearity. Second, if $L$ is far away from 1, then the system’s nonlinearity enhances. However, values of $L$ must be appropriately chosen to generate bimodal protein distributions since only the appropriate nonlinearity can lead to bimodality.

Fig. 3.

Molecular memory can induce bimodal protein expression in the presence of feedback: the gene model depicted in Fig. 2A, where ${\psi }_1\left(t;n\right)$ depends on $n$. Empty circles represent the results obtained by the sgCME, whereas the solid lines represent the results predicted by Eq. 11. A and B correspond to the case of $L\ge 1$, where A demonstrates the dependence of most probable protein numbers on memory index $L$, whereas B shows stationary protein distributions. C and D correspond to the case of $L\le 1$, where C demonstrates the dependence of most probable protein numbers on $L$, whereas D shows stationary protein distributions. Parameter values are set as: ${\mu }_0=4L,{\mu }_1=14L,K=6,n=4,{\lambda }_2=1$(A and B); ${\mu }_0=2.5L,{\mu }_1=10L,K=6,n=4,{\lambda }_2=1$ (C and D).

Molecular Memory Can Fine-Tune the Gene Expression Noise.

Transcription is a key step in gene expression. Biochemical processes associated with transcription often involve a variety of TFs, which regulate the promoter kinetics. For bacterial cells, promoters can exist in a surprisingly large number of regulatory states, e.g., the PRM promoter of phage lambda in Escherichia coli is regulated by 2 different TFs binding to 2 sets of 3 operators that can be brought together by looping out the intervening DNA, and, as a result, the number of regulatory states of the PRM promoter is up to 128 (54). In contrast, eukaryotic promoters are more complex, involving nucleosomes competing with or being removed by TFs (55). In addition to the conventional regulation by TFs, the eukaryotic promoters can also be epigenetically regulated via histone modifications (56–58), and such regulation may lead to very complex promoter structures (59). Given this complexity, we introduce intrinsic waiting-time distributions to model promoter kinetics. Specifically, assume that the switch times from OFF to ON and vice versa follow gamma distributions given, respectively, by ${\psi }_{\text{on}}\left(t;m\right)=\frac{{{\displaystyle {\lambda }_{\text{on}}}}^{L_{\text{on}}}}{\mathrm{\Gamma }\left(L_{\text{on}}\right)}{t}^{L_{\text{on}}-1}{e}^{-{\lambda }_{\text{on}}t}$ and ${\psi }_{\text{off}}\left(t;m\right)=\frac{{{\displaystyle {\lambda }_{\text{off}}}}^{L_{\text{off}}}}{\mathrm{\Gamma }\left(L_{\text{off}}\right)}{t}^{L_{\text{off}}-1}{e}^{-{\lambda }_{\text{off}}t}$, where $m$ represents the number of mRNA molecules. In addition, assume that the waiting-time distributions for transcription and degradation are given by ${\psi }_{\text{g}}\left(t;m\right)=\frac{{\mu }^{L_{\text{g}}}}{\mathrm{\Gamma }\left(L_{\text{g}}\right)}{t}^{L_{\text{g}}-1}{e}^{-\mu t}$ and ${\psi }_{\text{deg}}\left(t;m\right)=\frac{{\left(m{\lambda }_{\text{deg}}\right)}^{L_{\text{deg}}}}{\mathrm{\Gamma }\left(L_{\text{deg}}\right)}{t}^{L_{\text{deg}}-1}{e}^{-m{\lambda }_{\text{deg}}t}$. The corresponding gene model is schematically depicted in Fig. 4A.

Fig. 4.

Effect of molecular memory on gene expression. (A) Schematic representation of a model of stochastic transcription. (B) Dependence of the mean mRNA and the noise intensity on $L_{\text{on}}$, where $L_{\text{off}}=L_{\text{g}}=L_{\text{deg}}=1$. (C) Dependence of the mean mRNA and the noise intensity on $L_{\text{off}}$, where $L_{\text{on}}=L_{\text{g}}=L_{\text{deg}}=1$. (D) Dependence of the mean mRNA and the noise intensity on $L_{\text{g}}$, where $L_{\text{on}}=L_{\text{off}}=L_{\text{deg}}=1$. (E) Dependence of the mean mRNA and the noise intensity on $L_{\text{deg}}$, where $L_{\text{on}}=L_{\text{off}}=L_{\text{g}}=1$. The parameter values are set as ${\lambda }_{\text{on}}=L_{\text{on}},{\lambda }_{\text{off}}=4L_{\text{off}},{\lambda }_{\text{g}}=20L_{\text{g}}$ and ${\lambda }_{\text{deg}}=L_{\text{deg}}$. This setting implies the average time between successive reactions is kept fixed by scaling ${\lambda }_i$ appropriately with $L_i$.

Let $K_i\left(m\right)$ $\left(i=\mathrm{1,2,3,4}\right)$ be ETRs for transition from OFF to ON, for transition from ON to OFF, for transcription, and for degradation, respectively. According to Eq. 2, we can obtain the analytical expressions of the ETRs, which are given in SI Appendix. Moreover, the analytical expression of $K_1\left(m\right)$ with $L_{\text{on}}>1$ or $K_2\left(m\right)$ with $L_{\text{off}}>1$ can imply that the effect of molecular memory is equivalent to the introduction of a feedback, as interpreted above. Let $P_0\left(m\right)$ and $P_1\left(m\right)$ be the probabilities that mRNA has $m$ molecules at states OFF and ON, respectively. Then, the corresponding sgCME takes the following form:

$$\begin{array}{l}-K_1\left(m\right)P_0\left(m\right)+K_2\left(m\right)P_1\left(m\right)+K_4\left(m+1\right)P_0\left(m+1\right)-K_4\left(m\right)P_0\left(m\right)=0\\ K_1\left(m\right)P_0\left(m\right)-K_2\left(m\right)P_1\left(m\right)+K_4\left(m+1\right)P_1\left(m+1\right)-K_4\left(m\right)P_1\left(m\right)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+K_3\left(m-1\right)P_1\left(m-1\right)-K_3\left(m\right)P_1\left(m\right)=0\end{array}.$$ [12]

In general, Eq. 12 has no analytical solution but can be solved numerically (see SI Appendix for details). Fig. 4 shows numerical results, where we vary memory index $\left(L\right)$ but always keep the constant average time between successive reactions by scaling the characteristic parameter (e.g., $\alpha$) appropriately with $L$.

From this figure, we observe that the mRNA mean is monotonically decreasing in memory index $L_{\text{on}}$ (Fig. 4B) or in memory index $L_{\text{g}}$ (Fig. 4D) if the other memory indices are set as 1, but monotonically increasing in memory index $L_{\text{off}}$ (Fig. 4C) or in memory index $L_{\text{deg}}$ (Fig. 4E) if the other memory indices are set as 1. Fig. 4B shows that memory index $L_{\text{on}}$ amplifies the mRNA noise, whereas Fig. 4 C–E demonstrates that the other 3 memory indices can reduce the mRNA noise. In a word, Fig. 4 indicates that molecular memory plays an unneglectable role in gene expression.

Molecular Memory Can Induce Switch.

Recall that a toggle-switch network (Fig. 5A) can model the cross-repression between the determinants of different cellular states, which can result in a definite choice between 2 outcomes (60–62). Conventional models of genetic toggle switch consider exponential waiting-time distributions. However, the expression of a gene in general involves a multistep process. Indeed, transcriptional repressor monomer (A or B) binds first to dimers and then to specific DNA sequences near the promoter, repressing the production of transcriptional repressor monomer (B or A). This multistep process can lead to nonexponential waiting times, creating a memory between individual reaction events. Here, we consider a generalized model of genetic toggle switch, which is schematically shown in Fig. 5A with 4 reactions listed in Fig. 5B, where ${\psi }_{\text{g}}^{\text{A}}\left(t;\mathit{n}\right),{\psi }_{\text{deg}}^{\text{A}}\left(t;\mathit{n}\right)$ and ${\psi }_{\text{g}}^{\text{B}}\left(t;\mathit{n}\right),{\psi }_{\text{deg}}^{\text{B}}\left(t;\mathit{n}\right)$ are intrinsic waiting-time distributions for the synthesis and degradation of protein A and protein B, respectively, and $\mathit{n}={\left(n_{\text{A}},n_{\text{B}}\right)}^{\text{T}}$ with $n_{\text{A}}$ and $n_{\text{B}}$ representing the numbers of protein A and protein B molecules, respectively. Assume that these waiting-time distributions are given by ${\psi }_{\text{g}}^{\text{A}}\left(t;\mathit{n}\right)={\left[\mathrm{\Gamma }\left(L_{\text{g}}^{\text{A}}\right)\right]}^{-1}{\left[{\lambda }_{\text{g}}^{\text{A}}\left(\mathit{n}\right)\right]}^{L_{\text{g}}^{\text{A}}}{t}^{L_{\text{g}}^{\text{A}}-1}{e}^{-{\lambda }_{\text{g}}^{\text{A}}\left(\mathit{n}\right)t}$ with ${\lambda }_{\text{g}}^{\text{A}}\left(\mathit{n}\right)={\alpha }_{\text{A}}/\left(1+{\beta }_{\text{A}}{n}_{\text{B}}^{H_{\text{A}}}\right)$, ${\psi }_{\text{deg}}^{\text{A}}\left(t;\mathit{n}\right)={\lambda }_{\text{deg}}^{\text{A}}{n}_{\text{A}}{e}^{-{\lambda }_2{n}_{\text{A}}t}$; ${\psi }_{\text{g}}^{\text{B}}\left(t;\mathit{n}\right)={\left[\mathrm{\Gamma }\left(L_{\text{g}}^{\text{B}}\right)\right]}^{-1}{\left[{\lambda }_{\text{g}}^{\text{B}}\left(\mathit{n}\right)\right]}^{L_{\text{g}}^{\text{B}}}{t}^{L_{\text{g}}^{\text{B}}-1}{e}^{-{\lambda }_{\text{g}}^{\text{B}}\left(\mathit{n}\right)t}$ with ${\lambda }_{\text{g}}^{\text{B}}\left(\mathit{n}\right)={\alpha }_{\text{B}}/\left(1+{\beta }_{\text{B}}{n}_{\text{A}}^{H_{\text{B}}}\right)$, ${\psi }_{\text{deg}}^{\text{B}}\left(t;\mathit{n}\right)={\lambda }_{\text{deg}}^{\text{B}}{n}_{\text{B}}{e}^{-{\lambda }_2{n}_{\text{B}}t}$. Note that $L_{\text{g}}^{\text{A}}=1$ and $L_{\text{g}}^{\text{B}}=1$ correspond to the markovian case, whereas $L_{\text{g}}^{\text{A}}>1$ or $L_{\text{g}}^{\text{B}}>1$ corresponds to the nonmarkovian case.

Fig. 5.

(A) Schematic representation of a genetic toggle-switch model with molecular memory, where 2 genes are repressed by each other. (B) Four reactions corresponding to A, where waiting times for synthesis and degradation of each protein follow distributions. Default parameter values are taken as: ${\beta }_{\text{A}}={\beta }_{\text{B}}=1,H_{\text{A}}=H_{\text{B}}=1,{\lambda }_{\text{deg}}^{\text{A}}={\lambda }_{\text{deg}}^{\text{B}}=1$. (C and F) Joint distributions of proteins A and B, obtained by a numerical algorithm (SI Appendix). (D and G) Heat maps in the plane of protein A and B. (E and H) Time series of the difference between the levels of protein A and B, obtained by sgGA. C–E correspond to exponential waiting times, where parameter values are set as $L_{\text{g}}^{\text{A}}=L_{\text{g}}^{\text{B}}=1,{\alpha }_{\text{A}}={\alpha }_{\text{B}}=20$. F–H correspond to nonexponential waiting times, where parameter values are set as $L_{\text{g}}^{\text{A}}=L_{\text{g}}^{\text{B}}=2,{\alpha }_{\text{A}}={\alpha }_{\text{B}}=40$. Prob., probability.

Numerical results are demonstrated in Fig. 5 C–H, where Fig. 5 C–E corresponds to the case of exponential waiting times, whereas Fig. 5 F–H to the case of nonexponential waiting times. We observe that if the waiting times for synthesis of protein A and B follow exponential distributions (i.e., if we set $L_{\text{g}}^{\text{A}}=L_{\text{g}}^{\text{B}}=1$), the steady-state joint distribution of proteins A and B is unimodal, referring to Fig. 5 C and D. However, if the waiting times for synthesis of protein A and B follow nonexponential distributions (e.g., if we set $L_{\text{g}}^{\text{A}}=L_{\text{g}}^{\text{B}}=2$), the steady-state joint distribution of proteins A and B is bimodal, referring to Fig. 5 F and G. To examine the time dependence of the populations of 2 proteins in a single cell, we first perform stochastic simulations with a numerical algorithm (see SI Appendix for details) and then calculate the difference between the levels of proteins A and B. Numerical results are shown in Fig. 5 E and H. Comparing Fig. 5E with Fig. 5H, we find that 2 switching states occur only in the case of nonexponential waiting times or molecular memory. Thus, we conclude from Fig. 5 that molecular memory can induce bimodal distributions in the toggle-switch model depicted in Fig. 5A or Fig. 5B.

Discussion

Previous studies of biochemical-reaction processes on networks are mainly based on markovian (i.e., memoryless) hypothesis. However, as soon as a reactant interacts with its environment, the effect of molecular memory cannot be neglected. We have derived an exact sgCME, an sgLNA, and an sgFPE for a general biochemical-reaction network with molecular memory characterized by nonexponential waiting-time distributions. These derived equations allow one to retain analytical and/or numerical tractability, being general in scope, and thus are of a potential applicability in a wide variety of problems that transcend pure physics applications. The derived sgCME is particularly useful in finding stationary distributions in a number of nonmarkovian biochemical systems, as demonstrated in this article. Analysis of stochastic gene expression examples has indicated that the sgCME can help us find new biological knowledge, e.g., the effect of molecular memory is equivalent to the introduction of a feedback, and molecular memory can induce bimodality, although the distribution is not bimodal in the corresponding markovian case. The power of the sgCME can be enhanced by analyzing other examples, such as nonmarkovian random walks and diffusion on networks (21, 63–65) and nonmarkovian open quantum systems (66).

Our general theory can reproduce some known results for queuing models of biological processes. First, recall that Pedraza and Paulsson (6) analyzed a $G{I}^X/M/\infty$ (including $G/M/\infty$) model of gene expression with a general queuing waiting-time distribution for the arrival of bursts and an exponential waiting-time distribution for the decay of mRNAs and derived an approximate formula for the mRNA noise. In SI Appendix, we have used the above theory to reproduce this formula. Second, environmental perturbations or external noise, which is often inevitable in cellular processes, can be modeled with time delay (20). We have derived the analytical expressions of effective transition rates and further established the corresponding gLNA (see SI Appendix for details). Functionally, this gLNA may be analogous to the chemical fluctuation theorem for $G_t/G/\infty$ models of gene expression (19), where subscript “$t$” represents that the corresponding waiting-time distributions are time-varying.

Our theoretical framework can also be used in the inference of the structure and parameters involved in system modes for a broad class of nonmarkovian biochemical-reaction processes on networks. For example, the structure of gene promoters and their kinetics, which would be complex due to, e.g., TF regulation, can be inferred based on experimental data. In fact, we can first infer the key parameter $k$ in the Erlang waiting-time distribution from experimental data, since it can represent the number of small, difficultly specified reaction steps involved in transitions from ON to OFF states or vice versa, implying that the promoter structure can be determined. Then, we can use the standard method (e.g., the maximum likelihood estimation) to infer the values of other parameters from the experimental data, such as the mean switching rates between ON and OFF states, the mean transcription or translational rate. These inferred kinetic parameters in turn determine promoter kinetics and gene-expression dynamics. Furthermore, the sgCME can be used in the analysis of the corresponding stationary probabilistic behavior. In a word, we expect that our analytical framework will be of use for studying a variety of phenomena in biological and physical sciences and, indeed, in other areas where individual-based models with general waiting-time distributions and/or delayed interactions are relevant.

In the realistic world, “non-Markov is the rule, Markov is the exception,” as remarked by N. G. van Kampen (67). A stochastic process (i.e., the biological phenomenon evolving in time) may be or may not be markovian, depending on the variables used to describe it. If all of the variables are observable or measurable, the process is markovian. In general, however, this is impractical and even impossible. Therefore, most of real stochastic processes we observe are nonmarkovian. To model real stochastic processes with some unobservable variables, many different methods of modeling have been proposed, e.g., queuing models (6, 7, 17–19), delay models (5, 28), Langevin equations with color noise (68), and CTRW models (20–22, 26). Correspondingly, some simulation algorithms have also been developed, e.g., those based on general renewal processes (30–33) and the one by introducing some exogenously reaction channels (38). These approaches, despite their own advantages, have finite applications, e.g., queuing models are inconvenient to treating bi- or multimolecular reaction networks.

Finally, we point out that CTRWs used in our theory incorporate the timing of move, where a random walker waits between 2 moves for a duration that independently follows an intrinsic waiting-time distribution. In other words, the move events are generated by a renewal process. On the other hand, CTRWs can be categorized into the 2 classes of active CTRWs and passive CTRWs, depending on whether a random walker actively initializes them as it travels or passively follows states when available (21). In active CTRWs, the interevent time of a state is reinitialized when a random walker lands on it. In passive CTRWs to which queuing models correspond, however, the interevent time of a state is not reset, and the waiting time depends on the last activation time. Usually, active CTRWs generates interevent times from a given PDF, more suitable to the analytical study of random processes, whereas passive CTRWs use interevent times observed in real data, less favorable to model and analyze. Based on the active CTRW framework, we have established a set of theories for a general biochemical network with arbitrary intrinsic waiting-time distributions. Our analysis of generalized birth and death processes based on the passive CTRW framework (SI Appendix) has provided a general thinking by establishing the relationship between the active and passive CTRWs.