A derivation of the master equation from path entropy maximization

Abstract

The master equation and, more generally, Markov processes are routinely used as models for stochastic processes. They are often justified on the basis of randomization and coarse-graining assumptions. Here instead, we derive nth-order Markov processes and the master equation as unique solutions to an inverse problem. We find that when constraints are not enough to uniquely determine the stochastic model, an nth-order Markov process emerges as the unique maximum entropy solution to this otherwise underdetermined problem. This gives a rigorous alternative for justifying such models while providing a systematic recipe for generalizing widely accepted stochastic models usually assumed to follow from the first principles.

I. INTRODUCTION

Markov chains^1,2 are often the starting point for modeling condensed phase stochastic dynamics in biophysics^3–8 and beyond.⁹ Markov chains are approximations of continuous system dynamics. They are primarily justified on the basis of coarse-graining approximations.¹⁰ Coarse-graining reduces classical phase space—with phase points dynamics governed by Liouville's equations—to a discrete set of states—with stochastic hopping between states determined by stationary transition probabilities. Such coarse-graining methods have recently been used to show how Markov models can describe the continuous dynamics of biomolecules evolving in complex potential landscapes.^11–13

A very different approach to stochastic dynamics is due to Filyukov and Karpov¹⁴ and later Jaynes.¹⁵ Using this approach, stochastic dynamical models can be inferred as unique solutions to an inverse problem. A model is defined by the probability for each stochastic path. Normally, the number of stochastic paths greatly outnumbers the constraints imposed from data. To find a unique solution to this underdetermined problem we ask: which model not only satisfies the limited experimental constraints but also maximizes the entropy for the path probabilities? This is exactly equivalent to finding a model for the path probabilities which satisfies the experimental constraints while satisfying these logical consistency axioms due to Shore and Johnson:¹⁶ (1) when A and B are independent data then the model for P(A and B) must reduce to P(A)P(B) and the model for P(A or B) must reduce to P(A) + P(B); and (2) furthermore, any prediction made from the model must be independent of the coordinate system used in the calculation.

This method of finding a stochastic model is mathematically similar to the maximum entropy principle for determining equilibrium probability distributions.^17–20 In earlier work, Ge et al.—which extended the work of Stock et al.²¹ and Ghosh et al.²²—showed that the first order Markov chain emerges as a natural consequence of path entropy maximization. Here we generalize this work in many important ways. (1) We do not limit ourselves to first order Markov processes; (2) we consider the conditions for which the master equation emerges as a solution to the procedure of path entropy maximization; (3) we consider how different types of constraints affect the emergent model; and (4) we consider very general (nonlinear) constraints.

To the best of our knowledge, this is the first time the master equation and, more generally, nth-order Markov processes are rigorously shown to follow from maximum entropy principles. This provides an alternative justification for the master equation—the basic tool of stochastic physics and biology—which is distinct from standard chemical or mechanistic justifications provided by van Kampen,¹ Zwanzig,²³ Gillespie,²⁴ and others. The master equation assumes from the onset a dynamics described by stationary transition probabilities and time-varying state occupation probabilities. Here we only assume data of a specific type are available and the basic logical consistency axioms required to justify maximum entropy as an inference tool.¹⁶ Posing the master equation as the solution of an inverse problem is significant because possible generalizations to the master equation are now derivable within this formalism. These generalizations can then be justified on the firm axiomatic basis of provided by Shore and Johnson.

II. MARKOV MODEL OF nth ORDER: DEFINITIONS AND NOTATIONS

In this section, we briefly introduce the mathematical notation necessary for the remainder of the paper. Consider a stochastic process in discrete time. Let the index i_t denote the state of the system at time t along the path C from time 0 to T where C = {i₀, i₁, i₂, …, i_T}. The probability distribution of path C is

$$\begin{array}{c}\hfill P\left(C\right)=p(i_0,i_1,...,i_T).\end{array}$$ (1)

An n-point joint probability is defined as follows

$$\begin{array}{ccc}& & p(a_1,...,a_m;t)\hfill \\ & & \equiv \sum _{i_0,i_1,...,i_{t-m},j_1,j_2,...,j_{T-t}}p(i_0,i_1,...,i_{t-m},\hfill \\ & & a_1,...,a_m,j_1,j_2,...,j_{T-t}).\hfill \end{array}$$ (2)

The explicit time index is required, as the result depends on which indices are summed over. Conditional—also called transition—probabilities are obtained by invoking Bayes' theorem:

$$\begin{array}{c}\hfill p(i_0,...,i_{t-1}\to i_t)\equiv \frac{p(i_0,...,i_t)}{p(i_0,...,i_{t-1})}.\end{array}$$ (3)

We call p(i₀, …, i_{t − 1} → i_t) a transition probability. When the transition probability depends only on the previous n-time steps

$$\begin{array}{ccc}\hfill p(i_0,...,i_{t-1}\to i_t)& =& p(i_{t-n},i_{t-n+1}...,i_{t-1}\to i_t;t)\hfill \\ & \equiv & \frac{p(i_{t-n},...,i_t;t)}{p(i_{t-n},...,i_{t-1};t-1)},\hfill \end{array}$$ (4)

the process is called an nth-order Markov process. When the transition probability is time-independent, it is called a time-homogeneous Markov process. When no specification is given, a Markov process is assumed first order, time-homogeneous.

III. DERIVATION OF FIRST ORDER MARKOV PROCESS WITH LINEAR CONSTRAINTS

Here we show how the first order Markov process is derived from path entropy maximization. We begin with the definition of path entropy

$$\begin{array}{c}\hfill H=-\sum _{\{i_0,i_1,...,i_T\}}p(i_0,i_1,...,i_T)\log p(i_0,i_1,...,i_T).\end{array}$$ (5)

We consider N₁ and N₂ linear constraints on one and two-point probabilities, respectively:

$$\begin{array}{ccc}\hfill F_0^{\left(\alpha \right)}& \equiv & \sum _{t=0}^T\sum _{i_t}{\varepsilon }_{i_t}^{\left(\alpha \right)}p(i_t;t)-(T+1)E_0^{\left(\alpha \right)}\hfill \\ & =& 0(\alpha =1,...,N_1),\hfill \\ \hfill F_1^{\left(\gamma \right)}& \equiv & \sum _{t=0}^{T-1}\sum _{i_t{i}_{t+1}}{J}_{i_t{i}_{t+1}}^{\left(\gamma \right)}p(i_t,i_{t+1};t+1)-T{J}_0^{\left(\gamma \right)}\hfill \\ & =& 0,(\gamma =1,...,N_2)\hfill \end{array}$$ (6)

and a normalization condition

$$\begin{array}{c}\hfill \sum _{\{i_0,i_1,...,i_T\}}p(i_0,i_1,...,i_T)=1.\end{array}$$ (7)

These constraints are imposed using Lagrange multipliers. That is, the Lagrange multiplier terms are added to the path entropy as follows:

$$\begin{array}{ccc}& & -\sum _{\{i_0,i_1,...,i_T\}}p(i_0,i_1,...,i_T)\log p(i_0,i_1,...,i_T)\hfill \\ & & -\sum _{\alpha =1}^{N_1}{\beta }_{\alpha }\left(\sum _{t=0}^T\sum _{i_t}{\varepsilon }_{i_t}^{\left(\alpha \right)}p(i_t;t)-(T+1)E_0^{\left(\alpha \right)}\right)\hfill \\ & & +\sum _{\gamma =1}^{N_2}{\nu }_{\gamma }\left(\sum _{t=0}^{T-1}\sum _{i_t{i}_{t+1}}{J}_{i_t{i}_{t+1}}^{\left(\gamma \right)}p(i_t,i_{t+1};t+1)-T{J}_0^{\left(\gamma \right)}\right)\hfill \\ & & +(\rho +1)\left(\sum _{\{i_0,i_1,...,i_T\}}p(i_0,i_1,...,i_T)-1\right).\hfill \end{array}$$ (8)

Extremizing Eq. (8) with respect to p(i₀, i₁, …, i_T), we obtain

$$\begin{array}{ccc}& -& \log p(i_0,i_1,...,i_T)-\sum _{\alpha }{\beta }_{\alpha }\sum _{t=0}^T{\varepsilon }_{i_t}^{\left(\alpha \right)}\hfill \\ & & +\sum _{\gamma }{\nu }_{\gamma }\sum _{t=0}^{T-1}{J}_{i_t{i}_{t+1}}^{\left(\gamma \right)}+\rho =0.\hfill \end{array}$$ (9)

The Lagrange multipliers introduced in Eq. (8) are determined by additional equations which come from taking the variation of Eq. (8) with respect to these Lagrange multipliers. The solution to Eq. (9) is expressed in terms of the Lagrange multipliers as follows

$$\begin{array}{ccc}& & p(i_0,i_1,...,i_T)\hfill \\ & & =\exp \left(\rho -\sum _{\alpha }{\beta }_{\alpha }\sum _{t=0}^T{\varepsilon }_{i_t}^{\left(\alpha \right)}+\sum _{\gamma }{\nu }_{\gamma }\sum _{t=0}^{T-1}{J}_{i_t{i}_{t+1}}^{\left(\gamma \right)}\right)\hfill \\ & & =\exp \left(\rho \right)v\left(i_0\right)G(i_0,i_1)G(i_1,i_2)...G(i_{T-1},i_T)v\left(i_T\right),\hfill \end{array}$$ (10)

where the elements of the vector v, v(i), and the elements of the transfer matrix G, G(i, j), are defined as follows

$$\begin{array}{ccc}\hfill v\left(i\right)& =& \exp \left(-\sum _{\alpha }{\beta }_{\alpha }{\varepsilon }_i^{\left(\alpha \right)}/2\right),\hfill \\ \hfill G(i,j)& =& \exp \left(-\sum _{\alpha }{\beta }_{\alpha }{\varepsilon }_i^{\left(\alpha \right)}/2+\sum _{\gamma }{\nu }_{\gamma }{J}_{ij}^{\left(\gamma \right)}-\sum _{\alpha }{\beta }_{\alpha }{\varepsilon }_j^{\left(\alpha \right)}/2\right).\hfill \end{array}$$ (11)

The m-point joint probability distribution, Eq. (2), is obtained from Eq. (10) by summing over indices i₀, …, i_{t − m}, i_{t + 1}, …, i_T as follows:

$$\begin{array}{ccc}\hfill p(a_1,...,a_m;t)& =& \sum _{i_0,...,i_{t-m},i_{t+1},...,i_T}p(i_0,i_1,...,i_{t-m},a_1,...,a_m,i_{t+1},...,i_T)\hfill \\ & =& \exp \left(\rho \right)\left[{\mathbf{v}}^{†}{\mathbf{G}}^{t-m+1}\right]\left(a_1\right)G(a_1,a_2)G(a_2,a_3)...G(a_{m-1},a_m)\left[{\mathbf{G}}^{T-t}\mathbf{v}\right]\left(a_m\right)\hfill \\ & =& \frac{\left[{\mathbf{v}}^{†}{\mathbf{G}}^{t-m+1}\right]\left(a_1\right)G(a_1,a_2)G(a_2,a_3)...G(a_{m-1},a_m)\left[{\mathbf{G}}^{T-t}\mathbf{v}\right]\left(a_m\right)}{{\mathbf{v}}^{†}{\mathbf{G}}^T\mathbf{v}},\hfill \end{array}$$ (12)

where [v†Gⁿ](a) and [Gⁿv](a) denote the ath components of the row and column vectors v†Gⁿ and Gⁿv, respectively. (Similarly, [Gⁿ](a, b) denotes the (a, b) component of the matrix Gⁿ throughout the paper.) Therefore combining Eqs. (4) and (12), we have

$$\begin{array}{ccc}\hfill p(a_1,...,a_m\to a_{m+1};t)& =& \frac{\exp \left(\rho \right)\left[{\mathbf{v}}^{†}{\mathbf{G}}^{t-m}\right]\left(a_1\right)G(a_1,a_2)...G(a_m,a_{m+1})\left[{\mathbf{G}}^{T-t}\mathbf{v}\right]\left(a_{m+1}\right)}{\exp \left(\rho \right)\left[{\mathbf{v}}^{†}{\mathbf{G}}^{t-m}\right]\left(a_1\right)G(a_1,a_2)...G(a_{m-1},a_m)\left[{\mathbf{G}}^{T-t+1}\mathbf{v}\right]\left(a_m\right)}\hfill \\ & =& \frac{G(a_m,a_{m+1})\left[{\mathbf{G}}^{T-t}\mathbf{v}\right]\left(a_{m+1}\right)}{\left[{\mathbf{G}}^{T-t+1}\mathbf{v}\right]\left(a_m\right)}=p(a_m\to a_{m+1};t).\hfill \end{array}$$ (13)

This shows that a conditional probability of transition in fact depends only on the last two states, those right before and after the transition. Therefore, the process is indeed a first order Markov one. However, it should be noted that the transition probability has explicit time dependence.

The first order Markov property was also derived in Ref. 25 for the special case of constraining one-point and two-point statistics which we now define. One-particle statistics corresponds to $F_0^{\left(\alpha \right)}$ with

$$\begin{array}{c}\hfill {\varepsilon }_i^{\left(\alpha \right)}={\delta }_{i,\alpha }(\alpha =1,...,N),\end{array}$$ (14)

where the index α of the constraint now goes over each state of the system, N being their total number of such states. This constraint simply counts the number of times state α is visited over the course of the trajectory. Likewise, two-point statistics corresponds to imposing $F_1^{(\tau ,\sigma )}$ with

$$\begin{array}{c}\hfill J_{i,j}^{(\tau ,\sigma )}={\delta }_{i,\tau }{\delta }_{j,\sigma }(\tau ,\sigma =1,...,N),\end{array}$$ (15)

where we labelled the constraint by double indices (τ, σ) instead of the single index γ for notational convenience. This again simply counts the number of transitions from state τ to σ over the course of the trajectory.

IV. DERIVATION OF THE TIME-HOMOGENEOUS MASTER EQUATION

Recall that a master equation requires time-dependent state occupation probabilities and time-independent transition probabilities. Under what conditions are such approximations valid? To answer this question we apply the Perron-Frobenius theorem^26–29 to the G transfer matrix of Sec. III—a square matrix which by construction is of size N × N and has positive elements. According to the theorem, G satisfies the following properties:

• (1)It has a positive real eigenvalue r, called the Perron-Frobenius eigenvalue, such that any other eigenvalue λ is strictly smaller than r in absolute value, |λ| < r.
• (2)There is a left eigenvector y^† = (y₁, …, y_N) for r with positive components. That is, y^†G = ry^† and y_i > 0 for all i. Similarly, there is a right eigenvector z with positive components, such that Gz = rz and z_i > 0 for all i.
• (3)Left and right eigenvectors with eigenvalue r are non-degenerate.
• (4)${\lim }_{T\to \infty }\frac{{\mathbf{G}}^T}{r^T}=\mathbf{z}{\mathbf{y}}^{†}$

Now reconsider Eq. (13) where

$$\begin{array}{c}\hfill p(a_m\to a_{m+1};t)=\frac{G(a_m,a_{m+1})\left[{\mathbf{G}}^{T-t}\mathbf{v}\right]\left(a_{m+1}\right)}{\left[{\mathbf{G}}^{T-t+1}\mathbf{v}\right]\left(a_m\right)}.\end{array}$$ (16)

Since the vector v has only non-negative elements, both ${\mathbf{G}}^{\mathbf{T}}\mathbf{v}/r^T$ and ${\mathbf{v}}^{†}{\mathbf{G}}^{\mathbf{T}}/r^T$ have well-defined non-zero limits for T → ∞,

$$\begin{array}{c}\hfill \underset{T\to \infty }{\lim }\frac{{\mathbf{G}}^T\mathbf{v}}{r^T}=\mathbf{z}\left({\mathbf{y}}^{†}\mathbf{v}\right);\underset{T\to \infty }{\lim }\frac{{\mathbf{v}}^{†}{\mathbf{G}}^T}{r^T}=\left({\mathbf{v}}^{†}\mathbf{z}\right){\mathbf{y}}^{†}.\end{array}$$ (17)

Therefore, taking the limit T − t → ∞ of Eq. (16) and using Eq. (17), we find

$$\begin{array}{c}\hfill p(a\to b)=\frac{G(a,b)z\left(b\right)}{rz\left(a\right)}.\end{array}$$ (18)

That is, the transition probability is time-independent in this limit. However, from Eq. (12), the m-point joint probabilities are still explicitly time-dependent when T − t is large

$$\begin{array}{ccc}& & p(a_1,...,a_m;t)\hfill \\ & & =\frac{\left[{\mathbf{v}}^{†}{\mathbf{G}}^{t-m+1}\right]\left(a_1\right)G(a_1,a_2)G(a_2,a_3)...G(a_{m-1},a_m)z\left(a_m\right)}{r^t{\mathbf{v}}^{†}\mathbf{z}}\hfill \end{array}$$ (19)

and, in particular, this is true for the one-point occupation probability

$$\begin{array}{c}\hfill p(a;t)=\frac{\left[{\mathbf{v}}^{†}{\mathbf{G}}^t\right]\left(a\right)z\left(a\right)}{r^t{\mathbf{v}}^{†}\mathbf{z}}.\end{array}$$ (20)

Thus maximizing the path entropy under the linear constraint Eq. (6) up to two-point probabilities, which are imposed for infinite duration into the future (T − t → ∞), we obtain a time-homogeneous Markov process which is described by (1) time-independent transition probabilities and (2) time-dependent one-point occupation probabilities. From Eqs. (18) and (20), we now obtain the evolution equation for the time-homogeneous Markov process

$$\begin{array}{c}\hfill p(a;t+1)=\sum _{b}p(b;t)p(b\to a),\end{array}$$ (21)

which is the celebrated master equation.

Note the asymmetry in time: the transition probabilities as well as the joint probabilities are time dependent when the limit of t → ∞ is taken but T − t is kept finite. This is simply due to the fact that the transition probability p(b → a) is defined in a time-asymmetric manner.

The last limit to consider is the stationary case, when both T − t and t are large. Then the m-point joint probability of Eq. (12) reduces to

$$\begin{array}{ccc}& & p(a_1,...,a_m)\hfill \\ & & =\frac{y\left(a_1\right)G(a_1,a_2)G(a_2,a_3)...G(a_{m-1},a_m)z\left(a_m\right)}{r^{m-1}{\mathbf{y}}^{†}\mathbf{z}},\hfill \end{array}$$ (22)

which is independent of time as are the state occupation probability or any conditional probability derived from Eq. (22). This is to be expected, since we have time translation invariance in the stationary limit. Stationarity also trivially follows when the constraints themselves are stationary, which are much stronger conditions than those in Eq. (6).

Equation (22) was also derived in the large T limit with (m = T) in Ref. 31 though the stationary Markov process was assumed from the onset therein. In contrast, in the current work m is finite and can be as small as 1, even in the large T limit, and stationarity is derived rather than being an a priori assumption. Likewise, the first order Markov process was derived in Ref. 25 from path entropy maximization for the special case of pair statistics constraints, but neither conditions for the time-homogeneous process nor stationarity were discussed.³²

V. TIME-HOMOGENEOUS MARKOV PROCESSES WITH AN ARBITRARY INITIAL CONDITION

We have discussed how data can come in the form of state occupation probabilities (e.g., how long during the course of a single molecule fluorescence experiment did a protein dwell in a low fluorescent state) or transition probabilities. However, data may also be available in the form of conditions at different points in time (e.g., the sample is pumped into a photoexcited state at time t = 0). Are our conclusions on time-homogeneity from Sec. IV robust to initial, final, or other such conditions? In this section, we briefly show when the time-homogeneity of transition probability depends on such conditions.

Consider an arbitrary condition imposed at time τ

$$\begin{array}{c}\hfill p(a;t=\tau )=\pi \left(a\right).\end{array}$$ (23)

We then add the term ∑_aλ(a)(p(a; τ) − π(a)) with Lagrange multipliers λ(a) (a = 1, …, N) to the constrained entropy, Eq. (8). As before, setting the variation with respect to p(i₀, i₁, …, i_T) to zero yields

$$\begin{array}{ccc}\hfill p(i_0,i_1,...,i_T)& =& \exp \left(\rho +\lambda \left(i_{\tau }\right)-\beta \sum _{t=0}^T{\varepsilon }_{i_t}+\nu \sum _{t=0}^{T-1}{J}_{i_t{i}_{t+1}}\right)\hfill \\ & =& \exp (\rho +\lambda \left(i_{\tau }\right))v\left(i_0\right)G(i_0,i_1)G(i_1,i_2)...G(i_{T-1},i_T)v\left(i_T\right)\hfill \\ & =& \frac{v\left(i_0\right)\pi \left(i_{\tau }\right)G(i_0,i_1)G(i_1,i_2)...G(i_{T-1},i_T)v\left(i_T\right)}{{\sum }_{j_0...,j_T}v\left(j_0\right)\pi \left(j_{\tau }\right)G(j_0,j_1)G(j_1,j_2)...G(j_{T-1},j_T)v\left(j_T\right)},\hfill \end{array}$$ (24)

where in the last line we used the normalization condition Eq. (7) to eliminate ρ and the initialization constraint Eq. (23) to eliminate λ. We now have

$$\begin{array}{ccc}& & \tau \le t-m+1:\hfill \\ & & p(a_1,...,a_m;t)=\frac{{\sum }_a\left[{\mathbf{v}}^{†}{\mathbf{G}}^{\tau }\right]\left(a\right)\pi \left(a\right)\left[{\mathbf{G}}^{t-\tau -m+1}\right](a,a_1)G(a_1,a_2)...G(a_{m-1},a_m)\left[{\mathbf{G}}^{T-t}\mathbf{v}\right]\left(a_m\right)}{{\sum }_b\left[{\mathbf{v}}^{†}{\mathbf{G}}^{\tau }\right]\left(b\right)\pi \left(b\right)\left[{\mathbf{G}}^{T-\tau }\mathbf{v}\right]\left(b\right)},\hfill \\ & & t-m+1<\tau \le t:\hfill \\ & & p(a_1,...,a_m;t)\hfill \\ & & =\frac{\left[{\mathbf{v}}^{†}{\mathbf{G}}^{t-m+1}\right]\left(a_1\right)G(a_1,a_2)...G(a_{\tau -t+m-1},a_{\tau -t+m})\pi \left(a_{\tau -t+m}\right)}{{\sum }_b\left[{\mathbf{v}}^{†}{\mathbf{G}}^{\tau }\right]\left(b\right)\pi \left(b\right)\left[{\mathbf{G}}^{T-\tau }\mathbf{v}\right]\left(b\right)}\hfill \\ & & \times G(a_{\tau -t+m},a_{\tau -t+m+1})...G(a_{m-1},a_m)\left[{\mathbf{G}}^{T-t}\mathbf{v}\right]\left(a_m\right),\hfill \\ & & t<\tau :\hfill \\ & & p(a_1,...,a_m;t)=\frac{{\sum }_a\left[{\mathbf{v}}^{†}{\mathbf{G}}^{t-m+1}\right]\left(a_1\right)G(a_1,a_2)...G(a_{m-1},a_m)}{{\sum }_b\left[{\mathbf{v}}^{†}{\mathbf{G}}^{\tau }\right]\left(b\right)\pi \left(b\right)\left[{\mathbf{G}}^{T-\tau }\mathbf{v}\right]\left(b\right)}\hfill \\ & & \times \left[{\mathbf{G}}^{\tau -t}\right](a_m,a)\pi \left(a\right)\left[{\mathbf{G}}^{T-\tau }\mathbf{v}\right]\left(a\right).\hfill \end{array}$$ (25)

Using the definition of the transition probability from Eq. (4) we find

$$\begin{array}{ccc}& & \tau <t:\hfill \\ & & p(a_1,...,a_m\to a_{m+1};t)=\frac{G(a_m,a_{m+1})\left[{\mathbf{G}}^{T-t}\mathbf{v}\right]\left(a_{m+1}\right)}{\left[{\mathbf{G}}^{T-t+1}\mathbf{v}\right]\left(a_m\right)}\hfill \\ & & \tau \ge t:\hfill \\ & & p(a_1,...,a_m\to a_{m+1};t)\hfill \\ & & =\frac{G(a_m,a_{m+1}){\sum }_a\left[{\mathbf{G}}^{\tau -t}\right](a_{m+1},a)\pi \left(a\right)\left[{\mathbf{G}}^{T-\tau }\mathbf{v}\right]\left(a\right)}{{\sum }_b\left[{\mathbf{G}}^{\tau -t+1}\right](a_m,b)\pi \left(b\right)\left[{\mathbf{G}}^{T-\tau }\mathbf{v}\right]\left(b\right)}.\hfill \end{array}$$ (26)

We notice that the indices a₁, …, a_{m − 1} have dropped out from the right-hand side of Eq. (26). We can therefore write

$$\begin{array}{c}\hfill p(a_1,...,a_m\to a_{m+1};t)=p(a_m\to a_{m+1};t),\end{array}$$ (27)

showing that, once more, we have a first order Markov process. Furthermore, the transition probability for t > τ has exactly the same form as Eq. (13), independent of the initial condition π. It is therefore time-homogeneous under the limit of large T − t. The same is not true of t ⩽ τ, where the transition probability always depends on the specified condition and time-homogeneity requires both large T − τ and τ − t. As noted earlier, this time-asymmetry is a natural consequence of the fact that the definition of the transition probability itself is time-asymmetric.

VI. GENERAL DERIVATION OF nth-ORDER MARKOV PROCESS FROM PATH ENTROPY MAXIMIZATION

In this section, we generalize the arguments of Sec. III in two important ways: (1) we consider constraints on the data up to n + 1-point probabilities

$$\begin{array}{ccc}& & F^{\left(\alpha \right)}(\left\{p(i;t)\right\},\left\{p(i\to j;t)\right\},...,\hfill \\ & & \left\{p(i_0,...,i_{n-1}\to i_n;t)\right\})=0,\hfill \end{array}$$ (28)

and (2) we do not assume that the constraints F^(α) are linear functions of their arguments (as was the case for Eq. (6)).

Provided constraints are linear—as was the case in Eq. (6)—the arguments in Sec. III are generalizable to nth-order Markov processes. Indeed, the path probability would be described by the multiplication of rank-(n + 1) tensors rather than matrices, such as Eq. (12). The nth-order Markov process would follow immediately, though the derivation of the time-homogeneity of various transition probabilities, as in Secs. IV and V, would require the difficult task of applying an analogue of the Perron-Frobenius theorem for general tensors.

Since we want to derive the nth-order Markov process for fully general constraints, as given by Eq. (28), we take a different route. We first express the path probability p(i₁, i₂, …, i_T) in terms of the conditional probabilities:

$$\begin{array}{ccc}\hfill p(i_0,i_1,...,i_T)& =& p(i_0;0)p(i_0\to i_1;1)p(i_0,i_1\to i_2;2)...\hfill \\ & & \times p(i_0,i_1...,i_{T-1}\to i_T;T).\hfill \end{array}$$ (29)

Substituting this expression into Eq. (5), we get

$$\begin{array}{ccc}\hfill H& =& -\sum _{\{i_0,i_1,...,i_T\}}p(i_0,i_1,...,i_T)\hfill \\ & & \times \left(\log p(i_0;0)+\sum _{t=0}^{T-1}\log p(i_0,...,i_t\to i_{t+1};t+1)\right)\hfill \\ & =& -\sum _{i}p(i;0)\log p(i;0)\hfill \\ & & -\sum _{t=0}^{T-1}\sum _{\{i_0,i_1,...,i_{t+1}\}}p(i_0,i_1,...,i_{t+1};t+1)\hfill \\ & & \times \log p(i_0,...,i_t\to i_{t+1};t+1),\hfill \end{array}$$ (30)

where, in getting from first to second line, we invoked the relation between joint and marginal probabilities; $p(i_0,...,i_m;m)={\sum }_{i_{m+1},...,i_T}p(i_1,i_2,...,i_T).$

Now reconsider the constraints given by Eq. (28) imposed from p(i; t) to p(i₀, …, i_{n − 1} → i_n; t). We will maximize the entropy, Eq. (30), in two steps:

• (1)We maximize the entropy with respect to {p(i₀, …, i_k; t)} (k > n), for given values of {p(i₀, …, i_k; t)} with k ⩽ n.
• (2)We then vary the entropy over the remaining variables, {p(i₀, …, i_k; t)} (k ⩽ n).

By assumption, constraints on the data only matter in step 2. Furthermore, as we now show, step 1 (the unconstrained maximization) is sufficient to show that the general path probability reduces to that of an nth-order Markov process.

In order to perform step 1, we first invoke the equality

$$\begin{array}{c}\hfill -\sum _i{q}_i\log q_i\le -\sum _i{q}_i\log p_i\end{array}$$ (31)

for arbitrary probability distributions p_i and q_i.³³ It follows from Eq. (31) that

$$\begin{array}{ccc}& & -\sum _{j}p(i_0,...,i_{m-1}\to j;t)\log p(i_0,...,i_{m-1}\to j;t)\hfill \\ & \le & -\sum _{j}p(i_0,...,i_{m-1}\to j;t)\log p(i_{m-n},...,i_{m-1}\to j;t).\hfill \end{array}$$ (32)

Multiplying both sides of Eq. (32) by p(i₀, …, i_{m − 1}; t) and summing over i₀, …, i_{m − 1}, we find

$$\begin{array}{ccc}\hfill & =& -\sum _{i_0,...,i_{m-1},j}p(i_0,...,i_{m-1},j;t)\log p(i_0,...,i_{m-1}\to j;t)\hfill \\ & & \le -\sum _{i_0,...,i_{m-1},j}p(i_0,...,i_{m-1},j;t)\log p(i_{m-n},...,i_{m-1}\to j;t),\hfill \end{array}$$ (33)

where Eq. (4) was used. The above sets a bound on the last term of the path entropy, Eq. (30). Therefore, for given values of {p(i₀, …, i_k; t)} with k ⩽ n, we see that H is maximized for

$$\begin{array}{ccc}& & p(i_0,...,i_{m-1}\to j;t)\hfill \\ & & =p(i_{m-n},...,i_{m-1}\to j;t)(m>n),\hfill \end{array}$$ (34)

the system now being described by a nth-order Markov model where the probability p(i; t) is determined only by previous n steps of history.

Now the transition probability Eq. (34) for nth-order Markov process can be substituted into the path entropy formula, Eq. (30). Step 2 can then be carried forward: the resulting path entropy can be maximized with respect to the remaining variables p(i₀; t), p(i₀ → i₁; t), …, p(i₀, i₁, …, i_{n − 1} → i_n; t) under the constraints, Eq. (28).

In summary, we have just shown that nth-order Markov processes follow under very general constraints provided by Eq. (28). Markov models emerge from the entropy maximization method—and these provide immediate and principled generalizations of the ubiquitous master equation.

VII. DISCUSSION

Markov processes and master equations—the evolution equation describing a first order time-homogeneous Markov process—are standard stochastic modeling tools invoked across disciplines. Such models are usually justified mechanistically by coarse-graining arguments or by assuming quick randomization in space of reactants and products (the “well-stirred” approximation). Yet it is challenging to ascertain a priori whether any of these conditions actually hold. Just like maximum entropy has provided an alternative to ergodic theory for the justification of the equilibrium probability distribution,¹⁷ we believe that the path entropy techniques of Filyukov and Karpov,¹⁴ and later Jaynes,¹⁵ provide a compelling axiomatic basis for the Markov process and the master equation. Here the Markov process emerges as a solution to the following inverse problem: given measurable n-point constraints on a trajectory, what is the least biased model for a probability distribution of paths? By least biased, we mean one that, for instance, does not impose correlations in a model when such correlations are not otherwise warranted by the data (technically these are the logical consistency axioms of Shore and Johnson). The unique solution to this problem is that which maximizes the entropy subject to constraints from the data.

With this formalism, we justify generalizations of the master equation on rigorous mathematical grounds. It is tempting to conjecture whether the nth-order Markov process can lead to a time-homogeneous process so long as the constraints are imposed for a time much longer than that of one time step. The proof would require an analogue of the Perron-Frobenius theorem for general tensors, an interesting subject for further investigation.