Transition probabilities for general birth-death processes with applications in ecology, genetics, and evolution

Abstract

A birth-death process is a continuous-time Markov chain that counts the number of particles in a system over time. In the general process with n current particles, a new particle is born with instantaneous rate λ_n and a particle dies with instantaneous rate μ_n. Currently no robust and efficient method exists to evaluate the finite-time transition probabilities in a general birth-death process with arbitrary birth and death rates. In this paper, we first revisit the theory of continued fractions to obtain expressions for the Laplace transforms of these transition probabilities and make explicit an important derivation connecting transition probabilities and continued fractions. We then develop an efficient algorithm for computing these probabilities that analyzes the error associated with approximations in the method. We demonstrate that this error-controlled method agrees with known solutions and outperforms previous approaches to computing these probabilities. Finally, we apply our novel method to several important problems in ecology, evolution, and genetics.

1 Introduction

Birth-death processes (BDPs) have a rich history in probabilistic modeling, including applications in ecology, genetics, and evolution (Thorne et al 1991; Krone and Neuhauser 1997; Novozhilov et al 2006). Traditionally, BDPs have been used to model the number of organisms or particles in a system, each of which reproduce and die in continuous time. A general BDP is a continuous-time Markov chain on the non-negative integers in which instantaneous transitions from state n ≥ 0 to either n+1 or n−1 are possible. These transitions are called “births” and “deaths”. Starting at state n, jumps to n + 1 occur with instantaneous rate λ_n and jumps to n − 1 with instantaneous rate μ_n. The simplest BDP has linear rates λ_n = nλ and μ_n = nμ with no state-independent terms (Kendall 1948; Feller 1971). This model is the most widely-used BDP since there exist closed-form expressions for its transition probabilities (Bailey 1964; Novozhilov et al 2006). Many applications of BDPs require convenient methods for computing the probability P_m,n(t) that the system moves from state m to state n in finite time t ≥ 0. These probabilities exhibit their usefulness in many modeling applications since the probabilities do not depend on the possibly unobserved path taken by the process from m to n and hence make possible analyses of discretely sampled or partially observed processes. Despite the relative simplicity of specifying the rates of a general BDP, it can be remarkably difficult to find closed-form solutions for the transition probabilities even for simple models (Renshaw 1993; Mederer 2003; Novozhilov et al 2006).

In a pioneering series of papers, Karlin and McGregor develop a formal theory of general BDPs that expresses their transition probabilities in terms of a sequence of orthogonal polynomials and a spectral measure (Karlin and McGregor 1957a,b, 1958b). While the work of Karlin and McGregor yields valuable theoretical insights regarding the existence of unique solutions and properties of recurrence and transience for a given process, there remains no clear recipe for determining the orthogonal polynomials and measure corresponding to an arbitrary set of birth and death rates. Additionally, even when the polynomials and measure are known, the transition probabilities may not have an analytic representation or a convenient computational form.

Possibly due to the difficulty of finding computationally useful formulas for transition probabilities in general BDPs, many applied researchers resort to easier analyses using moments, first passage times, equilibrium probabilities, and other tractable quantities of interest. Referring to the system of Kolmogorov forward differential equations for transition probabilities that we give below, Novozhilov et al (2006, page 73) write,

“The problem with exact solutions of system (1) is that, in many cases, the expressions for the state probabilities, although explicit, are intractable for analysis and include special polynomials. In such cases, it may be sensible to solve more modest problems concerning the birth-and-death process under consideration, without the knowledge of the time-dependent behavior of state probabilities p_n(t).”

Indeed, closed-form analytic expressions for transition probabilities of general BDPs are only known for a few types of processes. Some examples include constant birth and death rates (Bailey 1964), zero birth or death rates (pure-death and pure-birth) (Yule 1925; Taylor and Karlin 1998), and certain linear rates (Karlin and McGregor 1958a). As a seemingly straightforward example, in the BDP with linear birth and death rates λ_n = nλ + ν and μ_n = nμ + γ including state-independent terms, Ismail et al (1988) offer the orthogonal polynomials and associated measure, but still no closed form is available for the transition probabilities.

Despite the difficulty in obtaining analytic expressions, several authors have made progress in approximate numerical methods for solution of transition probabilities in general BDPs. Murphy and O’Donohoe (1975) develop an appealing numerical method for the transition probabilities based on a continued fraction representation of Laplace-transformed transition probabilities. They invert these transformed probabilities by first truncating the continued fraction. Several other authors give similar expressions derived from truncation of the state space (Grassmann 1977b,a; Rosenlund 1978; Sharma and Dass 1988; Mohanty et al 1993). However, Klar et al (2010) find that methods based on continued fraction truncation and then subsequent analytical transformation can suffer from instability. As an alternative, Parthasarathy and Sudhesh (2006a) express the infinite continued fraction representation given by Murphy and O’Donohoe as a power series. Unfortunately, the small radius of convergence of this series makes it less useful for numerical computation.

We also note that for general BDPs that take values on a finite state space (usually n ∈ {0, 1,…, N}), it is possible to write a finite-dimensional stochastic transition rate matrix and solve for the matrix of transition probabilities. If the rate matrix is diagonal-izable, computation of transition probabilities in this manner can be computationally straightforward. To illustrate, let Q be a finite-dimensional stochastic rate matrix with Q = UΛU⁻¹ where U is an orthogonal matrix and Λ is diagonal. The matrix of transition probabilities P satisfies the matrix differential equation P′ = PQ with initial condition P(0) = I. The solution is P(t) = exp[Qt] = U diag(e^z₁t, e^z₂t,…, e^z_Nt) U⁻¹, where z₁,…, z_N are the eigenvalues of Q. However, it is possible to specify reasonable rate parameters in a general BDP that satisfy requirements for the existence of a unique solution, but do not result in a diagonalizable rate matrix. Also, if the state space over which the BDP takes values is large, numerical eigendecomposition of Q may be computationally expensive and could introduce serious roundoff errors.

To our knowledge, no robust computational method currently exists for finding the finite-time transition probabilities of general BDPs with arbitrary rates. Such a technique would allow rapid development of rich and sophisticated ecological, genetic, and evolutionary models. Additionally, in statistical applications, transition probabilities can serve as observed data likelihoods, and are thus often useful in estimating transition rate parameters from partially observed BDPs. We believe more sophisticated BDPs can be very useful for applied researchers. In spite of the numerical difficulties presented by approximant methods, we are surprised that continued fraction methods like that of Murphy and O’Donohoe (1975) are not more widely explored. This may be due to omission of important details in their derivation of continued fraction expressions for the Laplace transform of the transition probabilities.

In this paper, we build on continued fraction expressions for the Laplace transforms of the transition probabilities of a general BDP using techniques similar to those introduced by Murphy and O’Donohoe, and we fill in the missing details in the proof of this representation. We then apply the Laplace inversion formulae of Abate and Whitt (1992a,b) to obtain an efficient and robust method for computation of transition probabilities in general BDPs. Our method relies on three observations: 1) it is possible to find exact expressions for Laplace transforms of the transition probabilities of a general BDP using continued fractions (Murphy and O’Donohoe 1975); 2) evaluation of continued fractions is typically very fast, requires far fewer evaluations than equivalent power series, and there exist robust algorithms for evaluating them efficiently (Bankier and Leighton 1942; Wall 1948; Blanch 1964; Lorentzen and Waadeland 1992; Craviotto et al 1993; Abate and Whitt 1999; Cuyt et al 2008); and 3) recovery of probability distributions by Laplace inversion using a Riemann sum approximation is often more computationally stable than analytical methods of inversion (Abate and Whitt 1992a,b, 1995). Finally, we demonstrate the advantages of our error-controlled method through its application to several birth-death models in ecology, genetics, and evolution whose solution remains unavailable by other means.

2 Transition probabilities

2.1 Background

A general birth-death process is a continuous-time Markov process X = {X(t), t ≥ 0} counting the number of arbitrarily defined “particles” in existence at time t ≥ 0, with X(0) = m ≥ 0. To characterize the process, we define non-negative instantaneous birth rates λ_n and death rates μ_n for n ≥ 0, with μ₀ = 0 and transition probabilities P_m,n(t) = Pr(X(t) = n | X(0) = m). While λ_n and μ_n are time-homogeneous constants, they may depend on n. We refer to the classical linear BDP in which λ_n = nλ and μ_n = nμ as the “simple birth-death process” (Kendall 1948; Feller 1971). The general BDP transition probabilities satisfy the infinite system of ordinary differential equations

$$\begin{array}{l}\frac{{\text{d}P}_{m,0}(t)}{\text{d}t}={\mu }_1{P}_{m,1}(t)-{\lambda }_0{P}_{m,0}(t),\text{and}\\ \frac{{\text{d}P}_{m,n}(t)}{\text{d}t}={\lambda }_{n-1}{P}_{m,n-1}(t)+{\mu }_{n+1}{P}_{m,n+1}(t)-({\lambda }_n+{\mu }_n)P_{m,n}(t)\text{for}n\ge 1,\end{array}$$ (1)

with boundary conditions P_m,m(0) = 1 and P_m,n(0) = 0 for n ≠ m (Feller 1971).

Karlin and McGregor (1957b) show that for arbitrary starting state m, transition probabilities can be represented in the form

$$P_{m,n}(t)={\pi }_n{\int }_0^{\infty }{e}^{-xt}{Q}_m(x)Q_n(x)\psi (\text{d}x),$$ (2)

where π₀ = 1 and π_n = (λ₀ ··· λ_n−1)/(μ₁ ··· μ_n) for n ≥ 1. Here, {Q_n(x)} is a sequence of polynomials satisfying the three-term recurrence relation

$$\begin{array}{l}{\lambda }_0{Q}_1(x)={\lambda }_0+{\mu }_0-x,\text{and}\\ {\lambda }_n{Q}_{n+1}(x)=({\lambda }_n+{\mu }_n-x)Q_n(x)-{\mu }_n{Q}_{n-1}(x),\end{array}$$ (3)

and ψ is the spectral measure of X with respect to which the polynomials {Q_n(x)} are orthogonal. The system (1) has a unique solution if and only if

$$\sum _{k=0}^{\infty }\left({\pi }_k+\frac{1}{{\lambda }_k{\pi }_k}\right)=\infty .$$ (4)

In what follows, we assume that the rate parameters {λ_n} and {μ_n} satisfy (4). Closed-form solutions to (1) are available for a surprisingly small number of choices of {λ_n} and {μ_n}. We therefore need another approach to find useful formulae for computation of the transition probabilities.

2.2 Continued fraction representation of Laplace transform

To find an expression that is useful for computing P_m,n(t) for an arbitrary general BDP, a fruitful approach is often to Laplace transform each equation of the system (1) and form a recurrence relationship relating back to the Laplace transform of P_m,n(t). We base our presentation on that of Murphy and O’Donohoe (1975). Denote the Laplace transform of P_n,m(t) as

$$f_{m,n}(s)=\mathcal{L}[P_{m,n}(t)](s)={\int }_0^{\infty }{e}^{-st}{P}_{m,n}(t)\text{d}t.$$ (5)

Applying the Laplace transform to (1), with the starting state m = 0, we arrive at

$$\begin{array}{l}{sf}_{0,0}(s)-P_{0,0}(0)={\mu }_1{f}_{0,1}(s)-{\lambda }_0{f}_{0,0}(s),\text{and}\\ {sf}_{0,n}(s)-P_{0,n}(0)={\lambda }_{n-1}{f}_{0,n-1}(s)+{\mu }_{n+1}{f}_{0,n+1}(s)-({\lambda }_n+{\mu }_n)f_{0,n}(s)\end{array}$$ (6)

for n ≥ 1. Rearranging and recalling that P_0,0(0) = 1 and P_0,n(0) = 0 for n ≥ 1, we simplify (6) to

$$\begin{array}{l}{f}_{0,1}(s)=\frac{1}{{\mu }_1}[(s+{\lambda }_0)f_{0,0}(s)-1],\text{and}\\ f_{0,n}(s)=\frac{1}{{\mu }_n}\left[(s+{\lambda }_{n-1}+{\mu }_{n-1})f_{0,n-1}(s)-{\lambda }_{n-2}{f}_{0,n-2}(s)\right]\text{for}n\ge 2.\end{array}$$ (7)

Some rearranging of (7) yields the forward system of recurrence relations

$$\begin{array}{l}{f}_{0,0}(s)=\frac{1}{s+{\lambda }_0-{\mu }_1\left(\frac{f_{0,1}(s)}{f_{0,0}(s)}\right)},\text{and}\\ \frac{f_{0,n}(s)}{f_{0,n-1}(s)}=\frac{{\lambda }_{n-1}}{s+{\mu }_n+{\lambda }_n-{\mu }_{n+1}\left(\frac{f_{0,n+1}(s)}{f_{0,n}(s)}\right)}.\end{array}$$ (8)

Then combining these expressions, we arrive at the generalized continued fraction

$$f_{0,0}(s)=\frac{1}{s+{\lambda }_0-\frac{{\lambda }_0{\mu }_1}{s+{\lambda }_1+{\mu }_1-\frac{{\lambda }_1{\mu }_2}{s+{\lambda }_2+{\mu }_2-\cdots }}}.$$ (9)

This is an exact expression for the Laplace transform of the transition probability P_0,0(t). Let the partial numerators in (9) be a₁ = 1 and a_n = −λ_n−2μ_n−1, and the partial denominators b₁ = s + λ₀ and b_n = s + λ_n−1 + μ_n−1 for n ≥ 2. Then (9) becomes

$$f_{0,0}(s)=\frac{a_1}{b_1+\frac{a_2}{b_2+\frac{a_3}{b_3+\cdots }}}.$$ (10)

To express (10) in more typographically economical notation, we write

$$f_{0,0}(s)=\frac{a_1}{b_1+}\frac{a_2}{b_2+}\frac{a_3}{b_3+}\cdots .$$ (11)

We denote the kth convergent (approximant) of f_0,0(s) as

$$f_{0,0}^{(k)}(s)=\frac{a_1}{b_1+}\frac{a_2}{b_2+}\cdots \frac{a_k}{b_k}=\frac{A_k(s)}{B_k(s)}.$$ (12)

There are deep connections between the orthogonal polynomial representation (3), Laplace transforms (7), and continued fractions of the form (9) that are beyond the scope of this paper (Karlin and McGregor 1957b; Bordes and Roehner 1983; Guillemin and Pinchon 1999). Interestingly, Flajolet and Guillemin (2000) demonstrate a close relationship between the Laplace transforms of transition probabilities and state paths of the underlying Markov chain.

Before stating a theorem supporting this representation, we give two lemmas that will be useful in what follows.

Lemma 1

Both the numerator A_k and denominator B_k of (12) satisfy the same recurrence, due to Wallis (1695):

$$\begin{array}{l}{A}_k=b_k{A}_{k-1}+a_k{A}_{k-2},\mathit{and}\\ B_k=b_k{B}_{k-1}+a_k{B}_{k-2},\end{array}$$ (13)

with A₀ = 0, A₁ = a₁, B₀ = 1, and B₁ = b₁.

Lemma 2

By repeated application of Lemma 1, we arrive at the determinant formula

$$\begin{array}{l}{A}_k{B}_{k-1}-A_{k-1}{B}_k=(b_k{A}_{k-1}+a_k{A}_{k-2})B_{k-1}-A_{k-1}(b_k{B}_{k-1}+a_k{B}_{k-2})\\ =-a_k(A_{k-1}{B}_{k-2}-A_{k-2}{B}_{k-1})\\ ={(-1)}^{k-1}\prod _{i=1}^k{a}_i.\end{array}$$ (14)

Now we state and prove a theorem giving expressions for the Laplace transform of P_m,n(t). Although Murphy and O’Donohoe (1975) first report this result, they do not provide a detailed derivation in their paper.

Theorem 1

The Laplace transform of the transition probability P_m,n(t) is given by

$$f_{m,n}(s)=\{\begin{array}{ll}\left(\prod _{j=n+1}^m{\mu }_j\right)\frac{B_n(s)}{B_{m+1}(s)+}\frac{B_m(s)a_{m+2}}{b_{m+2}+}\frac{a_{m+3}}{b_{m+3}+}\cdots \hfill & \mathit{for}n\le m,\hfill \\ \left(\prod _{j=m}^{n-1}{\lambda }_j\right)\frac{B_m(s)}{B_{n+1}(s)+}\frac{B_n(s)a_{n+2}}{b_{n+2}+}\frac{a_{n+3}}{b_{n+3}+}\cdots \hfill & \mathit{for}m\le n,\hfill \end{array}$$ (15)

where a_n, b_n, and B_n are as defined above.

Proof

Proof To simplify notation, we sometimes omit the dependence of f_k, A_k, and B_k on the Laplace variable s. Suppose the process starts at X(0) = m. We can re-write the Laplace-transformed equations (6) with P_m,m(0) = 1 and P_m,n(0) = 0 for all n ≠ m as

$${sf}_{m,0}(s)-{\delta }_{m0}={\mu }_1{f}_{m,1}(s)-{\lambda }_0{f}_{m,0}(s),$$ (16a)

$${sf}_{m,n}(s)-{\delta }_{mn}={\lambda }_{n-1}{f}_{m,n-1}(s)+{\mu }_{n+1}{f}_{m,n+1}(s)-({\lambda }_n+{\mu }_n)f_{m,n}(s),$$ (16b)

where δ_mn = 1 if m = n and zero otherwise. We first derive the expression for n ≤ m. If m = 0, f_0,0(s) is given by (11), so we assume in what follows when n ≤ m, that m ≥ 1. Rearranging (16a), we see that since B₀ = 1 and s + λ₀ = b₁ = B₁,

$$f_{m,0}=\frac{B_0}{B_1}{\mu }_1{f}_{m,1}.$$ (17)

Now, to show the general case by induction, assume that for n ≤ m,

$$f_{m,n-1}=\frac{B_{n-1}}{B_n}{\mu }_n{f}_{m,n}.$$ (18)

Substituting (18) into (16b) when n < m, we have

$$b_{n+1}{f}_{m,n}={\lambda }_{n-1}\frac{B_{n-1}}{B_n}{\mu }_n{f}_{m,n}+{\mu }_{n+1}{f}_{m,n+1}$$ (19)

$$\left(b_{n+1}+a_{n+1}\frac{B_{n-1}}{B_n}\right)f_{m,n}={\mu }_{n+1}{f}_{m,n+1}$$ (20)

$$f_{m,n}=\frac{B_n}{B_{n+1}}{\mu }_{n+1}{f}_{m,n+1}$$ (21)

and so (18) is true for any n < m. Letting n = m, we have by (18) and (16b),

$$b_{m+1}{f}_{m,m}=1+{\lambda }_{m-1}\left(\frac{B_{m-1}}{B_m}{\mu }_m{f}_{m,m}\right)+{\mu }_{m+1}{f}_{m,m+1}.$$ (22)

Recalling that s + λ_m + μ_m = b_m+1 and using Lemma 1,

$${\mu }_{m+1}{f}_{m,m+1}=1-\frac{B_{m+1}}{B_m}{f}_{m,m}.$$ (23)

Rearranging the previous equation, we find that

$$f_{m,m}=\frac{1}{\frac{B_{m+1}}{B_m}+{\mu }_{m+1}\frac{f_{m,m+1}}{f_{m,m}}}.$$ (24)

Likewise, we can write (16b) as a continued fraction recurrence:

$$\frac{f_{m,n}}{f_{m,n-1}}=\frac{{\lambda }_{n-1}}{s+{\mu }_n+{\lambda }_n+{\mu }_{n+1}\frac{f_{m,n+1}}{f_{m,n}}}.$$ (25)

Then plugging (25) into (24) and iterating, we obtain the continued fraction for f_m,m:

$$\begin{array}{l}{f}_{m,m}=\frac{1}{\frac{B_{m+1}}{B_m}+}\frac{a_{m+2}}{b_{m+2}+}\frac{a_{m+3}}{b_{m+3}+}\cdots \\ =\frac{B_m}{B_{m+1}+}\frac{B_m{a}_{m+2}}{b_{m+2}+}\frac{a_{m+3}}{b_{m+3}+}\cdots .\end{array}$$ (26)

This is an exact formula for the Laplace transform of P_m,m(t), and proves the case m = n. For n ≤ m, we iterate (18) to get

$$\begin{array}{l}{f}_{m,n}=\frac{B_n}{B_{n+1}}{\mu }_{n+1}{f}_{m,n+1}\\ =\frac{B_n}{B_{n+1}}\frac{B_{n+1}}{B_{n+2}}{\mu }_{n+1}{\mu }_{n+2}{f}_{m,n+2}\\ =\frac{B_n}{B_{n+1}}\frac{B_{n+1}}{B_{n+2}}\cdots \frac{B_{m-1}}{B_m}{\mu }_{n+1}{\mu }_{n+2}\cdots {\mu }_m{f}_{m,m}\\ =\left(\prod _{j=n+1}^m{\mu }_j\right)\frac{B_n}{B_m}{f}_{m,m}.\end{array}$$ (27)

Substituting (26) for f_m,m completes the proof for n ≤ m.

To find the formula for f_m,n when n > m, we adopt a similar approach. From (24) we arrive at

$$B_{m+1}{f}_{m,m}=B_m-B_m{\mu }_{m+1}{f}_{m,m+1}.$$ (28)

We proceed inductively. Assume that for n > m,

$$B_{n+1}{f}_{m,n}=\left(\prod _{j=m}^{n-1}{\lambda }_j\right)B_m+{\mu }_{n+1}{B}_n{f}_{m,n+1}.$$ (29)

From (16b), we have

$$b_{n+2}{f}_{m,n+1}={\lambda }_n{f}_{m,n}+{\mu }_{n+2}{f}_{m,n+2}.$$ (30)

Solving for f_m,n in (29) and plugging this into the above equation, we have

$$b_{n+2}{f}_{m,n+1}={\lambda }_n\left(\prod _{j=m}^{n-1}{\lambda }_j\right)\frac{B_m}{B_{n+1}}+{\lambda }_n{\mu }_{n+1}\frac{B_n}{B_{n+1}}{f}_{m,n+1}+{\mu }_{n+2}{f}_{m,n+2}.$$ (31)

Recalling that −λ_nμ_n+1 = a_n+2,

$$(b_{n+2}{B}_{n+1}+a_{n+2}{B}_m)f_{m,n+1}=\left(\prod _{j=m}^n{\lambda }_j\right)B_n+{\mu }_{n+2}{B}_{n+1}{f}_{m,n+2},$$ (32)

and by Lemma 1,

$$B_{n+2}{f}_{m,n+1}=\left(\prod _{j=m}^n{\lambda }_j\right)B_m+{\mu }_{m+2}{B}_{n+1}{f}_{m,m+2}.$$ (33)

This establishes the recurrence (29). Then for any n ≥ m, we can rearrange (29) to obtain

$$f_{m,n}=\left(\prod _{j=m}^{n-1}{\lambda }_j\right)\frac{B_m}{B_{n+1}-B_n{\mu }_{n+1}\frac{f_{m,n+1}}{f_{m,n}}}.$$ (34)

This completes the proof.

2.3 Obtaining transition probabilities

Murphy and O’Donohoe (1975) find transition probabilities by truncating (15) at a pre-specified depth, forming a partial fractions sum, and inverse transforming. Parthasarathy and Sudhesh (2006a) give a series solution for transition probabilities based on an equivalence between continued fractions like (15) and power series. However, both of these approaches suffer from serious drawbacks, as we explore in detail in the Appendix.

We instead seek an efficient and robust numerical method for evaluating and inverting (15). We first note that continued fractions typically converge rapidly, and in our experience, evaluation of (15) is very fast and stable using the Lentz algorithm and its subsequent improvements (Lentz 1976; Thompson and Barnett 1986; Press 2007). We therefore invert (15) numerically by a summation formula.

To do this, we treat the continued fraction representation (15) of the Laplace transform of P_m,n(t) as an unknown but computable function of the complex Laplace variable s. We base our presentation on that of Abate and Whitt (1992a). If ε is a positive real number such that all singularities of f_m,n(s) lie to the left of ε in the complex plane, the inverse Laplace transform of f_m,n(s) is given by the Bromwich integral

$$P_{m,n}(t)={\mathcal{L}}^{-1}(f_{m,n}(s))=\frac{1}{2\pi i}{\int }_{\epsilon -i\infty }^{\epsilon +i\infty }{e}^{st}{f}_{m,n}(s)\text{d}s.$$ (35)

Letting s = ε + iu,

$$\begin{array}{l}{P}_{m,n}(t)=\frac{1}{2\pi }{\int }_{-\infty }^{\infty }{e}^{(\epsilon +iu)t}{f}_{m,n}(\epsilon +iu)\text{d}u\\ =\frac{e^{\epsilon t}}{2\pi }{\int }_{-\infty }^{\infty }[cos(ut)+isin(ut)]f_{m,n}(\epsilon +iu)\text{d}u\\ =\frac{e^{\epsilon t}}{2\pi }[{\int }_{-\infty }^{\infty }\left[Re(f_{m,n}(\epsilon +iu))cos(ut)-Im(f_{m,n}(\epsilon +iu))sin(ut)\right]\text{d}u+i{\int }_{-\infty }^{\infty }\left[Im(f_{m,n}(\epsilon +iu))cos(ut)+Re(f_{m,n}(\epsilon +iu))sin(ut)\right]\text{d}u],\end{array}$$ (36)

but P_m,n(t) is real-valued, so the imaginary part of the last equality in (36) is zero.

Then

$$P_{m,n}(t)=\frac{e^{\epsilon t}}{2\pi }{\int }_{-\infty }^{\infty }\left[Re(f_{m,n}(\epsilon +iu))cos(ut)-Im(f_{m,n}(\epsilon +iu))sin(ut)\right]\text{d}u.$$ (37)

But since P_m,n(t) = 0 for t < 0, we also have that

$${\int }_{-\infty }^{\infty }\left[Re(f_{m,n}(\epsilon +iu))cos(ut)+Im(f_{m,n}(\epsilon +iu))sin(ut)\right]\text{d}u=0.$$ (38)

Then applying (38) to (37), we obtain

$$P_{m,n}(t)=\frac{e^{\epsilon t}}{\pi }{\int }_{-\infty }^{\infty }Re(f_{m,n}(\epsilon +iu))cos(ut)\text{d}u.$$ (39)

Finally, we note that since

$$Re(f(\epsilon -iu))={\int }_0^{\infty }{e}^{-\epsilon t}cos(ut)P_{m,n}(t)\text{d}t=Re(f(\epsilon +iu)),$$ (40)

it must be the case that Re f_m,n(ε + iu) is even in u for every ε. Therefore,

$$P_{m,n}(t)=\frac{2e^{\epsilon t}}{\pi }{\int }_0^{\infty }Re(f_{m,n}(\epsilon +iu))cos(ut)\text{d}u.$$ (41)

Following Abate and Whitt (1992a), we approximate the integral above by a discrete Riemann sum via the trapezoidal rule with step size h:

$$\begin{array}{l}{P}_{m,n}(t)\approx \frac{{he}^{\epsilon t}}{\pi }Re(f_{m,n}(\epsilon ))+\frac{2{he}^{\epsilon t}}{\pi }\sum _{k=1}^{\infty }Re(f_{m,n}(\epsilon +\mathit{ikh}))cos(\mathit{kht})\\ =\frac{e^{A/2}}{2t}Re\left(f_{m,n}\left(\frac{A}{2t}\right)\right)+\frac{e^{A/2}}{t}\sum _{k=1}^{\infty }{(-1)}^{k}Re\left(f_{m,n}\left(\frac{A+2k\pi i}{2t}\right)\right),\end{array}$$ (42)

where the second line is obtained by setting h = π/(2t) and ε = A/(2t); this change of variables eliminates the cosine term.

2.4 Numerical considerations

While (42) presents a method for numerical solution of the transition probabilities P_m,n(t) for a BDP with arbitrary birth and death rates, it is not yet an algorithm for reliable evaluation of these probabilities. In order to develop a reliable numerical method, we must: 1) characterize the error introduced by discretization of the integral in (41); 2) determine a suitable method to evaluate this nearly alternating sum while controlling the error; and 3) accurately and rapidly evaluate the infinite continued fraction in (15).

Abate and Whitt show that the discretization error that arises in (42) is

$$e_d=\sum _{k=1}^{\infty }{e}^{-kA}{P}_{m,n}((2k+1)t),$$ (43)

and when P_m,n(t) ≤ 1,

$$e_d\le \sum _{k=1}^{\infty }{e}^{-kA}=\frac{e^{-A}}{1-e^{-A}}\approx e^{-A},$$ (44)

when e^−A is small. Then to obtain e_d ≤ 10^−γ, we set A = γ log(10). As Abate and Whitt point out, the terms of the series (42) alternate in sign when

$$Re\left(f_{m,n}\left(\frac{A+2k\pi i}{2t}\right)\right)$$ (45)

has constant sign. This suggests that a series acceleration method may be helpful in keeping the terms of the sum manageable and avoiding roundoff error due to summands of alternating sign. We opt to use the Levin transform for this purpose (Levin 1973; Press 2007; Numerical Recipes Software 2007).

Evaluation of rational approximations to continued fractions by repeated application of Lemma 1 is appealing, but suffers from roundoff error when denominators are small (Press 2007). To evaluate the infinite continued fraction in the summand of (42), we use the modified Lentz method (Lentz 1976; Thompson and Barnett 1986; Press 2007). To demonstrate, suppose we wish to approximate the value of f_0,0(s), given by (9) by truncating at depth k. Then

$$f_{0,0}^{(k)}(s)=\frac{A_k(s)}{B_k(s)}$$ (46)

is the kth rational approximant to the infinite continued fraction f_0,0(s). In the modified Lentz method, we stabilize the computation by finding the ratios

$$C_k=\frac{A_k}{A_{k-1}}\text{and}{D}_k=\frac{B_{k-1}}{B_k}$$ (47)

so that $f_{0,0}^{(k)}$ can be found iteratively by

$$f_{0,0}^{(k)}=f_{0,0}^{(k-1)}{C}_k{D}_k.$$ (48)

Using Lemma 1, we can iteratively compute C_k and D_k via the updates

$$C_k=b_k+\frac{a_k}{C_{k-1}}\text{and}{D}_k=\frac{1}{b_k+a_k{D}_{k-1}}.$$ (49)

In practice, we must evaluate the continued fraction to only a finite depth, but we must evaluate to a depth sufficient to control the error. Suppose we wish to evaluate the infinite continued fraction f_0,0(s) given by (9) at some complex number s. Intuitively, we wish to terminate the Lentz algorithm when the difference between successive convergents is small. However, it is not immediately clear how the difference between convergents $f_{0,0}^{(k)}(s)-f_{0,0}^{(k-1)}(s)$ is related to the absolute error $f_{0,0}(s)=f_{0,0}^{(k)}$.Craviotto et al (1993) make this relationship clear by furnishing an a posteriori truncation error bound for Jacobi fractions of the same form as (9) in this paper. Assuming that $f_{0,0}^{(k)}(s)=A_k(s)/B_k(s)$ converges to f0,0(s) as k → ∞, Craviotto et al (1993) give the bound

$$\left|f_{0,0}(s)-f_{0,0}^{(k)}(s)\right|\le \frac{\left|\frac{B_k(s)}{B_{k-1}(s)}\right|}{\left|\text{Im}\left(\frac{B_k(s)}{B_{k-1}(s)}\right)\right|}|f_{0,0}^{(k)}(s)-f_{0,0}^{(k-1)}(s)|,$$ (50)

that is valid when Im(s) is nonzero. Note that B_k(s)/B_k−1(s) = 1/D_k(s), so (50) is easy to evaluate during iteration under the Lentz algorithm. Therefore, we stop at depth k in the Lentz algorithm when

$$\frac{\mid 1/D_k(s)\mid }{\mid Im(1/D_k(s))\mid }\left|f_{0,0}^{(k)}(s)-f_{0,0}^{(k-1)}(s)\right|$$ (51)

is small.

2.5 Numerical results

Although our error-controlled method is designed to be used when an analytic solution cannot be found, we seek to validate our numerical results by comparison to available analytic and numerical solutions. For the simple BDP with λ_n = nλ and μ_n = nμ, our numerical results agree with the values from the well-known closed-form solution given explicitly in Bailey (1964) as

$$\begin{array}{l}{P}_{m,n}(t)=\sum _{j=0}^{min(m,n)}\left(\begin{array}{c}m\\ j\end{array}\right)\left(\begin{array}{c}m+n-j-1\\ m-1\end{array}\right){\alpha }^{m-j}{\beta }^{n-j}{(1-\alpha -\beta )}^j\\ P_{m,0}(t)={\alpha }^m\end{array}$$ (52)

where

$$\alpha =\frac{\mu \left(e^{(\lambda -\mu )t}-1\right)}{\lambda e^{(\lambda -\mu )t}-\mu }\text{and}\beta =\frac{\lambda \left(e^{(\lambda -\mu )t}-1\right)}{\lambda e^{(\lambda -\mu )t}-\mu }.$$ (53)

Murphy and O’Donohoe (1975) give numerical probabilities for four general birth-death models: a) immigration-death with λ_n = 0.2 and μ_n = 0.4n; b) immigration-emigration with μ_n = 0.3, μ₀ = 0, and μ_n = 0.1; c) queue with λ_n = 0.6, μ₀ = 0, μ₁ = μ₂ = 0.2, μ₃ = μ₄ = 0.4, and μ_n = 0.6 for n ≥ 5; and d) λ_n = 0.4, ${\mu }_n=0.1\sqrt{n}$ Our results agree with those computed by Murphy and O’Donohoe for each of the four models given in Tables 2 through 7 in their paper (Murphy and O’Donohoe 1975). We note that Murphy and O’Donohoe did not report probabilities for m > 2 or n > 5 in any of their four models. In our experience, their method performs poorly when n + m + k is greater than approximately 20.

As a demonstration of the instability of the approximant method, we contrast the numerical results given by our error-controlled method with those obtained using the approximant method, that we implemented as described in Murphy and O’Donohoe (1975), except for some rescaling of intermediate quantities to avoid obvious sources of roundoff error. Figure 1 shows this comparison, using model (a) above, for three values of the truncation index k. Note that increasing the truncation depth k in the approximant method does not improve the error.

Fig. 1.

Comparison of transition probabilities P_10;n(t = 1) computed by our error-controlled method and that of Murphy and O’Donohoe (1975) for the immigration-death model with λ_n = 0=2 and μ_n = 0.4n. The open circles are the values given by our method. The solid line corresponds with the approximant method of Murphy and O’Donohoe with k = 2 (solid line), k = 3 (dashed line), and k = 4 (dotted line). In our experience, the approximant method fails whenever n +m+ k is greater than approximately 20. It is interesting to note that increasing the depth of truncation k in the approximant method actually worsens the approximation.

3 Applications

Drawing on the robustness and generality of our error-controlled method, we conclude with four models in ecology, genetics, and evolution whose analytic solutions remain elusive and where past numerical approaches have fallen short. Using our approach, computation of transition probabilities is straightforward, and the techniques outlined above may be used without modification. Some of the examples are well-known models, and others are novel. In some cases, the orthogonal polynomials satisfying (3) are known, and hence a solution could be numerically computed using (2), provided there are good ways of evaluating the polynomials. Often, a severe drawback of using known orthogonal polynomials to compute a solution based on (2) is that the polynomials are model-specific. This makes experimentation and model selection difficult, since computation of transition probabilities depends on a priori analytic information about the polynomials and measure associated with the BDP. Our method does not rely on a priori information about the process, other than the birth and death rates for each state.

3.1 Immigration and emigration

Consider a population model for the number of organisms in an area, and suppose new immigrants arrive at rate ν, and emigrants leave at rate γ. Organisms living in the area reproduce with per-capita birth rate λ and die with rate μ. Define the linear rates

$${\lambda }_n+n\lambda +\nu \text{and}{\mu }_n=n\mu +\gamma .$$ (54)

For the case γ = 0, an analytic expression for the orthogonal polynomials is known (Karlin and McGregor 1958a). For nonzero γ, orthogonal polynomials are available from which a solution of the form (2) may be computed (Karlin and McGregor 1958a; Ismail et al 1988). However, using our error-controlled method, we can easily find the transition probabilities without additional analytic information. Figure 2 shows an example of the time-evolution of P_10,n(t) for various times t and states n, with the parameters λ = 0.5, ν = 0.2, μ = 0.3, and γ= 0.1. The approximant method method of Murphy and O’Donohoe fails to produce useful probabilities for n > 10 (not shown).

Fig. 2.

Transition probabilities for the immigration/emigration model with λ = 0.5, ν = 0.2, μ = 0.3, and γ= 0.1. The top panel shows P_10,n(t) with t = 1 (solid line), t = 2 (dashed line), t = 3 (dotted line), and t = 4 (dash-dotted line) for n = 0,…, 50. The bottom panel shows P_10,n(t) with n = 15 (solid line), n = 20 (dashed line), n = 25 (dotted line), and n = 30 (dash-dotted line) for t ∈ (0, 20).

3.2 Logistic growth with Allee effects

Populations of organisms that occupy a finite space may be subject to various constraints on their growth. The per-capita birth rate may decline when there are more organisms than the ecosystem can sustain (Tan and Piantadosi 1991). This can happen when there are too many organisms competing for the same food supply. The decay of population size above some carrying capacity is usually called logistic growth by ecologists. Another density-dependent constraint is known as the Allee effect, in which per-capita birth rate increases superlinearly with n once a small population has been established, due to favorable consequences of density, such as cooperation and mutual protection from predators (Allee et al 1949). As a realistic example of a general BDP that has no obvious solution by orthogonal polynomials, we seek a model that both transiently supports growth above the carrying capacity, and reflects these two density-dependent constraints, similar in spirit to models described by Tan and Piantadosi (1991) and Dennis (2002).

Qualitatively, if the per-capita birth rate with no density effects is λ, then the total birth rate should rise faster than nλ when n is small, slower than nλ for intermediate n near the carrying capacity, and should decay toward zero for n greater than the carrying capacity. Tan and Piantadosi introduce a logistic birth rate ${\lambda }_n=n\lambda (1-\frac{n}{N})$ for a finite state space model that takes values {0; 1; : : : ;N}. However, to allow for temporary growth beyond the carrying capacity, we choose λ_n α λn²e^−&agr;n for intermediate and large n. To achieve attenuated growth for small n as well, we scale this rate by a logistic function, yielding

$${\lambda }_n=\frac{\lambda n^2{e}^{-\alpha n}}{1+e^{\beta (n-M)}}\text{and}{\mu }_n=n\mu ,$$ (55)

where M is the population size with highest birth rate, and the death rate is assumed to be proportional only to the number of existing individuals. Figure 3 shows the resulting rates for various states n, with the different phases of population change shaded. To illustrate that the model produces the desired behavior, several realizations of the process are given in the lower panel for various starting values. The shaded regions correspond with the three phases of growth. Note that most paths in the lower panel of Figure 3 center near n = 27, where the birth rate and death rate are equal. The lower panel corresponds with Figure 1 in Dennis (2002). Figure 4 demonstrates the success of the error-controlled method in computing time-dependent extinction probabilities P_m,0 for various starting values with λ = 1, α = 0.2, β = 0.3, M = 20, and μ = 0.1.

Fig. 3.

Behavior of logistic/Allee model. The upper panel shows a plot of birth (solid line) and death (dashed line) rates for states n = 0; : : : ; 60, and parameters λ = 1, μ = 0.1, M = 20, α = 0.2, and β = 0.3. The different phases of growth are labeled in the shaded regions. The lower panel shows stochastic realizations of the logistic/Allee model for various starting values. The shaded regions correspond with the shaded phases of growth in the upper panel.

Fig. 4.

Logistic/Allee model probabilities of extinction P_m;0(t) for initial population sizes m = 1 (solid line), m = 5 (dashed line), m = 10 (dotted line), and m = 15 (dash-dotted line). The full model parametrization is found in the text.

3.3 Moran models with mutation and selection

The probability of fixation or extinction of an allele in finite populations is frequently of interest to researchers in genetics. However, publications often rely on the probability of eventual extinction P_m,0(t → ∞), or the probability of fixation of a novel mutation in a population of constant size N, P_1,N (t → ∞). While these asymptotic probabilities do reveal important properties of the underlying models, the information they provide about the distribution of time to fixation/extinction is incomplete. In practice, researchers may observe that m organisms in a sample exhibit a certain trait at a certain time. Then P_m,0(t), the probability of extinction of that trait at finite times t in the future should presumably be of great interest, since researchers cannot reliably observe the process for infinitely long times. Additionally, the finite-time probability of fixation/extinction may exhibit threshold effects or unexpected dynamics that are not revealed by the asymptotic probability of such an event.

Moran (1958) introduces a model for the time-evolution of a biallelic locus when the population size is constant through time. A biallelic locus is a location in an organism’s genome in which two different genetic variants or alleles exist in a population. We are interested in how the number of individuals carrying each allele changes from generation to generation. Krone and Neuhauser (1997) exploit the Moran model to derive a BDP counting the number of individuals with a certain allele in the context of ancestral genealogy reconstruction in which one allele offers a selective advantage to individuals that carry it. Selection greatly complicates the problem and remains an active area of research. In a limiting case, this process corresponds to Kingman’s coalescent process when there is no mutation or selection (Kingman 1982a,b).

To construct the Moran process with mutation and selection, suppose a finite population of N haploid organisms has 2 alleles at a certain locus: A₁ and A₂. Individuals that carry A₁ reproduce at rate α and A₂ individuals reproduce at rate β. Suppose further that individuals carrying the A₁ allele have a selective advantage over individuals carrying A₂, so α > β. When an individual dies, it is replaced by the offspring of a random parent chosen from all N individuals, including the one that dies. This parent contributes a gamete carrying its allele that is also subject to mutation. Mutation from A₁ to A₂ happens with probability u and in reverse with rate υ. The new offspring receives the possibly mutated haplotype and the process continues.

Let X(t) be a BDP counting the number of A₁ individuals on the state space n ε {0,…,N}. To construct the transition rates of the process, suppose there are currently n individuals of type A₁. We first consider the addition of a new individual of type A₁, so that n → n + 1. For this to happen, the individual that dies must be of type A₂. If the parent of the replacement is one of the n of type A₁, the parent contributes its allele without mutation, and this happens with probability 1 − u. If the parent of the replacement is one of the N − n of type A₂, the parent contributes its allele, which then mutates with probability υ. Therefore, the total rate of addition is

$${\lambda }_n=\frac{N-n}{N}\left[\alpha \frac{n}{N}(1-u)+\beta \frac{N-n}{N}v\right],$$ (56)

for n = 0,…,N with λ_n = 0 when n > N. Likewise, the removal of an individual of type A₁ can happen when one of the n individuals of type A₁ is chosen for replacement. If the parent of the replacement is one of the N − n of type A₂, the parent contributes A₂ without mutation, with probability 1 − υ. If the parent is one of the n of type A₁, the allele must mutate to A₂ with probability u. The total rate of removals becomes

$${\mu }_n=\frac{n}{N}\left[\beta \frac{N-n}{N}(1-v)+\alpha \frac{n}{N}u\right],$$ (57)

for n = 1,…,N with μ₀ = λ_N = 0 and μ_n = 0 when n > N. Note that if υ > 0, then λ₀ > 0 so the A₁ allele cannot go extinct. Also, if u > 0, then μ_N > 0, so the A₁ allele cannot be fixed in the population.

Karlin and McGregor (1962) derive the relevant polynomials and measure for the Moran process described above, but without selection, so that α = β. Donnelly (1984) gives expressions for the transition probabilities in the case where α = β = 1, noting that when selection is introduced (via differing α and α), his approach is no longer fruitful. Using our technique, computation of the transition probabilities under selection is straightforward. The upper panel of Figure 5 shows the probability of fixation by time t. The lower panel shows the finite-time fixation probability of A₁, P_m,100(t), with u = 0 so the state n = 100 is absorbing.

Fig. 5.

Transition probabilities for the Moran model with selection. The upper panel shows the probability of n individuals having allele A₁ at time t, P_50;n(t) for the Moran model with N = 100, starting from m = 50 with u = 0:02, v = 0:01, α = 60, and β = 10. We show the probabilities for t = 1 (solid line), t = 3 (dashed line), t = 5 (dotted line), t = 8 (dash-dotted line). Note that although the states 0 and 100 are not absorbing, the mutation rates u and v are small enough that probability accumulates significantly in these end states. Note also the asymmetry in the distribution at longer times. The lower panel reports the probability of fixation by time t, P_m;100(t), for the same model, but with u = 0 so the state n = 100 is absorbing. The probabilities shown are for m = 70 (solid line), m = 50 (dashed line), m = 20 (dotted line), and m = 1 (dash-dotted line). Note the starkly different time-dynamics for different starting values.

Since the state space in the Moran model is finite, it is natural to consider the matrix exponentiation method discussed in the Introduction. We write the stochastic transition matrix as

$$Q=\left[\begin{array}{cccccc}-{\lambda }_0& {\lambda }_0& & & & \\ {\mu }_1& -({\lambda }_1+{\mu }_1)& {\lambda }_1& & & \\ & {\mu }_2& -({\lambda }_2+{\mu }_2)& {\lambda }_2& & \\ & & \ddots & \ddots & \ddots & \\ & & & {\mu }_N& -({\lambda }_N+{\mu }_N)& {\lambda }_N\end{array}\right]$$ (58)

where λ_n and μ_n are defined by (56) and (57), respectively. In our experience, the matrix exponentiation method often works well, and its computational cost is similar to that of our error-controlled method. However, it is highly sensitive to rate matrix conditioning. For example, Figure 6 shows a comparison of transition probabilities from the error-controlled method and the matrix exponentiation method for the Moran model with N = 100, α = 210, β = 20, u = 0:002, and υ = 0. In evolutionary terms, this means that mutation from A₁ to A₂ is impossible, and the A₂ haplotype suffers from low fitness. Computationally, this has the effect of making μ_n small for most n, and hence the rate matrix grows ill-conditioned.

Fig. 6.

Comparison of Moran model transition probabilities P_50;n(t = 0.2) computed by two methods with N = 100, α = 210, β = 20, u = 0:002, and v = 0. The open circles correspond with our error-controlled method, and the solid line corresponds with the matrix exponentiation method. This choice of parameters causes wild fluctuations in probabilities reported by the matrix exponentiation method since the stochastic rate matrix becomes nearly singular.

Although the rate matrix in this example is nearly defective, this choice of parameter values is not unreasonably extreme. For example, researchers in population genetics often wish to test the hypothesis that selection occurs in a dataset. They fit parameters for models with selection (full model) and without selection (restricted model) and perform a likelihood ratio test of this hypothesis. If the estimates of β and u in the full model are small, they may be unable to reliably compute the probability (likelihood) of the data, given the estimated parameter values under the full model.

3.4 A frameshift-aware indel model

Thorne et al (1991) introduce a BDP modeling insertion and deletion of nucleotides in DNA for applications in molecular evolution. The authors model the process of sequence length evolution by assuming that a new nucleotide can be inserted adjacent to every existing nucleotide, and every existing nucleotide is subject to deletion, at a constant per-nucleotide rate. This corresponds to the simple BDP with λ_n = nλ and μ_n = nμ. If a sequence has m nucleotides at time 0 and there are n nucleotides at time t later, the probability of this event is P_m,n(t).

However, an important aspect of biological sequence evolution is conservation of the structure and biophysical properties of proteins that result from transcription and translation of DNA sequences. After coding DNA is transcribed into RNA, ribosomes translate 3-nucleotide chunks (codons) of the RNA into a single amino acid residue, that is then joined to the end of a growing protein polymer. Insertions or deletions (indels) in a DNA sequence that result in a shift in this triplet code are called “frame-shift” mutations. It is likely that a frame-shift indel occurring in a protein-coding DNA sequence results in a protein that is prematurely terminated or possesses structural and chemical characteristics unlike the ancestral protein. Insertions or deletions whose length is a multiple of three should be more common. We seek to model this behavior in a novel way: suppose the indel process is a BDP similar in spirit to the one presented by Thorne et al (1991), and the rate of insertion and deletion of nucleotides depends on the number of nucleotides already inserted, modulo (mod) 3:

$${\lambda }_n=\{\begin{array}{lll}n{\beta }_0\hfill & \text{if}n-1=0\hfill & mod3\hfill \\ n{\beta }_1\hfill & \text{if}n-1=1\hfill & mod3\hfill \\ n{\beta }_2\hfill & \text{if}n-1=2\hfill & mod3\hfill \end{array}\text{and}{\mu }_n=\{\begin{array}{lll}n{\gamma }_0\hfill & \text{if}n-1=0\hfill & mod3\hfill \\ n{\gamma }_1\hfill & \text{if}n-1=1\hfill & mod3\hfill \\ n{\gamma }_2\hfill & \text{if}n-1=2\hfill & mod3\hfill \end{array}.$$ (59)

Here we assume that β₂ > β_0, β₁, and γ₁ > γ_0, γ₂ so that transitions to state n such that n − 1 = 0 mod 3 occur at a faster rate per nucleotide. The linear-periodic nature of these birth and death rates make solution of the orthogonal polynomials and measure corresponding with this BDP difficult. The approximant method of Murphy and O’Donohoe also fails here for large n. However, using our error-controlled method, numerical results are readily available. Figure 7 shows P_1,n(t) for n = 0,…,50 at various times t. Note that the distribution of the number of inserted bases has peaks at the integers mod three. Finally, it is worth noting that the dearth of tractable BDPs for indel events has been a major deterrent in statistical sequence alignment and we are actively exploring solutions to this problem using our error-controlled method.

Fig. 7.

Frameshift-aware indel model probability of observing n inserted DNA bases, given starting at m = 1. The transition probability P_1;n(t) is shown for t = 5 (solid line), t = 7 (dashed line), t = 9 (dotted line), and t = 11 (dash-dotted line), with parameters β₀ = 0.3, β₁ = 1, β₂ = 4, γ₀ = 2, γ₁ = 0.2, and γ₂ = 0.2.

4 Conclusion

Traditionally the simple BDP with linear rates has dominated modeling applications, since its transition probabilities and other quantities of interest find analytic expressions. However, increasingly sophisticated models in ecology, genetics, and evolution, among other fields, may necessitate more advanced computational methods to handle processes whose birth and death rates do not easily yield analytic solutions. We have demonstrated a flexible method for finding transition probabilities of general BDPs that works for arbitrary sets of birth and death rates {λ_n} and {μ_n}, and does not require additional analytic information. This should prove useful for rapid development and testing of new models in applications. For simple models whose solution is available, we find that our method agrees with known solutions and remains robust for large starting and ending states and long times t. It is our hope that the method presented here will assist researchers in understanding the properties of increasingly rich and realistic models.

5 Appendix

5.1 Approximant method

Murphy and O’Donohoe (1975) approximate the inverse Laplace transform of (15) by first truncating the continued fraction as a rational approximant through a partial fractions sum. To illustrate the pitfalls of this approach, we derive the inversion expressions presented by Murphy and O’Donohoe and analyze their properties. We provide an example to show that this technique can become numerically unstable. We first seek to uncover the truncation error in the time domain of the transition probabilities. If we truncate the continued fractions (15) at depth k, we have

$$\begin{array}{l}{f}_{m,n}^{(k)}(s)=\left(\prod _{j=n+1}^m{\mu }_j\right)\frac{B_n}{B_{m+1}+}\frac{B_m{a}_{m+2}}{b_{m+2}+}\frac{a_{m+3}}{b_{m+3}+}\cdots \frac{a_{m+k}}{b_{m+k}}\text{for}n\le m,\text{and}\\ f_{m,n}^{(k)}(s)=\left(\prod _{j=m}^{n-1}{\lambda }_j\right)\frac{B_m}{B_{n+1}+}\frac{B_n{a}_{n+2}}{b_{n+2}+}\frac{a_{n+3}}{b_{n+3}+}\cdots \frac{a_{n+k}}{b_{n+k}}\text{for}n\ge m.\end{array}$$ (60)

For concreteness, suppose in what follows that n ≥ m. Note that the denominator of the second equation is simply B_n+k. Let $A_k^{(n)}$ be the numerator of the continued fraction in the second equation in (60), so

$$f_{m,n}^{(k)}=\left(\prod _{j=m}^{n-1}{\lambda }_j\right)\frac{A_k^{(n)}}{B_{n+k}},$$ (61)

where $A_k^{(n)}$ satisfies $A_k^{(0)}=A_k,A_1^{(n)}={\prod }_{j=1}^{n+1}{a}_j$, and

$$A_k^{(n)}=a_{n+k}{A}_{k-2}^{(n)}+b_{n+k}{A}_{k-1}^{(n)}.$$ (62)

Note also that the difference between truncated estimates in the Laplace domain (s) is

$$\begin{array}{l}\frac{A_{n+k}}{B_{n+k}}-\frac{A_n}{B_n}=\frac{A_{n+k}{B}_n-A_n{B}_{n+k}}{B_{n+k}{B}_n}\\ =\frac{{(-1)}^n{A}_k^{(n)}}{B_{n+k}{B}_n}.\end{array}$$ (63)

This yields the generalized determinant formula

$$A_{n+k}{B}_n-A_n{B}_{n+k}={(-1)}^n{A}_k^{(n)},$$ (64)

and at a root s_i of B_n+k(s), we have

$$A_k^{(n)}(s_i)={(-1)}^n{A}_{n+k}(s_i)B_n(s_i).$$ (65)

Now if s₁, s_2,…,s_n are the roots of B_n(s), we have, using the previous line and a partial fractions decomposition of (60), the formula for the Laplace transform of the transition probability P_m,n(t), truncated at k,

$$\begin{array}{l}{f}_{m,n}^{(k)}(s)=\left(\prod _{j=m}^{n-1}{\lambda }_j\right)\frac{B_m(s)A_k^{(n)}(s)}{B_{n+k}(s)}\\ =\left(\prod _{j=m}^{n-1}{\lambda }_j\right)\frac{B_m(s)A_k^{(n)}(s)}{{\prod }_{i=1}^{n+k}(s-s_i)}\\ =\left(\prod _{j=m}^{n-1}{\lambda }_j\right)\sum _{i=1}^{n+k}\frac{B_m(s)B_n(s_i)A_{n+k}(s_i)}{{\prod }_{j\ne i}(s_j-s_i)}\left(\frac{1}{s-s_i}\right),\end{array}$$ (66)

since we only require the values of A_n+k(s) and B_n(s) at the zeros of B_n+k(s). Then inverse transforming, an approximate formula for the transition probability P_m,n(t) is

$$P_{m,n}^{(k)}(t)\approx \left(\prod _{j=m}^{n-1}{\lambda }_j\right)\sum _{i=1}^{n+k}\frac{B_m(s_i)B_n(s_i)A_{n+k}(s_i)}{{\prod }_{j\ne i}(s_j-s_i)}{e}^{-s_{i}t}.$$ (67)

The roots of B_n(s), used in (66) and (67), are often found numerically as follows. Consider the characteristic polynomial det (B̃_n + sI) of the matrix

$${\stackrel{\sim }{B}}_n=\left(\begin{array}{cccccc}{\lambda }_0& 1& & & & \\ {\lambda }_0{\mu }_1& {\lambda }_1+{\mu }_1& 1& & & \\ & {\lambda }_1{\mu }_2& {\lambda }_2+{\mu }_2& 1& & \\ & & \ddots & \ddots & \ddots & \\ & & & {\lambda }_{n-3}{\mu }_{n-2}& {\lambda }_{n-2}+{\mu }_{n-2}& 1\\ & & & & {\lambda }_{n-2}{\mu }_{n-1}& {\lambda }_{n-1}+{\mu }_{n-1}\end{array}\right).$$ (68)

It is clear that the nth partial denominator B_n(s) = det(B̃_n + sI), and this quantity is zero when −s is an eigenvalue of the matrix B̃_n. Therefore, the negatives of the eigenvalues of B̃_n are the roots of B_n(s). Furthermore, B̃_n can be transformed into a real symmetric matrix via a similarity transform and hence B_n(s) has precisely n roots, all of which are simple, real, and negative. One usually finds these eigenvalues via the QR algorithm or similar numerical techniques (Press 2007). However, the iterative eigendecomposition of (68) generates small errors in the eigenvalues for large n + k. These errors are amplified in the product in the denominator of each summand in (67), resulting in a sum with both positive and negative terms that may be very large. Klar et al (2010) encounter similar instability in this algorithm. Their solution is to find the roots of the terms in the numerator and compute each summand as a product of individual numerators and denominators in an attempt to keep roundoff error in the product from accumulating. So, if z_1,…_,z_n+k are the roots of A_n+k then (67) becomes

$$P_{m,n}^{(k)}(t)\approx \left(\prod _{j=m}^{n-1}{\lambda }_j\right)\sum _{i=1}^{n+k}{B}_m(s_i)B_n(s_i)(z_i-s_i)\prod _{j\ne i}\left(\frac{z_j-s_i}{s_j-s_i}\right)e^{-s_{i}t}.$$ (69)

This procedure does improve the numerical stability of the computation, but requires two eigendecompositions of possibly large matrices for every evaluation of P_m,n(t), increasing the computational cost and, for large m and n, the roundoff error. In our opinion, it is more advantageous to avoid truncation of the continued fraction (9) at a pre-specified index, and instead evaluate the continued fraction until convergence during numerical inversion. Figure 1 shows how approximant methods fail for large n.

5.2 A power series method

Parthasarathy and Sudhesh (2006a) present exact solutions by transforming continued fractions such as (9) into an equivalent power series. Wall (1948) shows that Jacobi fractions of this type can always be represented by an equivalent power series. However, the small radius of convergence of power series expressions for transition probabilities can limit their usefulness for long times or large birth or death rates. Parthasarathy and Sudhesh show that P_0,n(t) has a power series representation given by

$$P_{m,n}(t)=\left(\prod _{k=0}^{n-1}{a}_{2k}\right)\sum _{m=0}^{\infty }{(-1)}^{m}A(m,2n)\frac{t^{m+n}}{(m+n)!},$$ (70)

where

$$A(m,n)=\sum _{i_1=0}^n{a}_{i_1}\sum _{i_2=0}^{i_1+1}{a}_{i_2}\sum _{i_3=0}^{i_2+1}{a}_{i_3}\cdots \sum _{i_m=0}^{i_{m-1}+1}{a}_{i_m},$$ (71)

with A(0, n) = 1 (Parthasarathy and Sudhesh 2006a,b). Here, a_2n = λ_n and a_2n+1 = μ_n in the notation used in their papers. This approach is unique because it yields an exact analytic expression for the transition probabilities of a general BDP. However, the radius of convergence of the power series depends on the specified rates, and this radius may be quite small. To illustrate the pitfalls of this approach, consider a_n = (n + 1) λ, corresponding to the BDP with λ_n = (2n + 1) λ and μ_n = 2nλ (Parthasarathy and Sudhesh 2006a, Example 4.6). The power series for the transition probability in this process becomes

$$P_{0,n}(t)=\sum _{m=0}^{\infty }{(-1)}^m\frac{(2n+2m)!}{m!n!}\frac{{(\lambda t/2)}^{n+m}}{(n+m)!}.$$ (72)

Then the radius of convergence R of the power series is given by

$$\begin{array}{l}1/R=\underset{m\to \infty }{lim}\left|\frac{(2n+2m+2)!{(\frac{\lambda }{2})}^{n+m+1}}{(m+1)!n!(n+m+1)!}\times \frac{m!n!(n+m)!}{(2n+2m)!{(\frac{\lambda }{2})}^{n+m}}\right|\\ =\underset{m\to \infty }{lim}\frac{(2m+2n+1)(2n+2m+2)}{(m+1)(n+m+1)}\left(\frac{\lambda }{2}\right)\\ =\underset{m\to \infty }{lim}\frac{2m+2n+1}{m+1}\lambda \\ =2\lambda .\end{array}$$ (73)

And so the series diverges when 2λt > 1. To illustrate the limitations of the power series approach, note that in this process, the transition intensity from 0 to 1 is λ, so the expected first-passage time from 0 to 1 is (T_0,1) = 1/λ. Therefore, we cannot evaluate (72) when t is greater than (T_0,1)=2. If n is much greater than 1, we may be unable to reliably evaluate P_0,n(t) for times near (T_0,n).