An estimator for the recombination rate from a continuously observed diffusion of haplotype frequencies

Abstract

Recombination is a fundamental evolutionary force, but it is difficult to quantify because the effect of a recombination event on patterns of variation in a sample of genetic data can be hard to discern. Estimators for the recombination rate, which are usually based on the idea of integrating over the unobserved possible evolutionary histories of a sample, can therefore be noisy. Here we consider a related question: how would an estimator behave if the evolutionary history actually was observed? This would offer an upper bound on the performance of estimators used in practice. In this paper we derive an expression for the maximum likelihood estimator for the recombination rate based on a continuously observed, multi-locus, Wright–Fisher diffusion of haplotype frequencies, complementing existing work for an estimator of selection. We show that, contrary to selection, the estimator has unusual properties because the observed information matrix can explode in finite time whereupon the recombination parameter is learned without error. We also show that the recombination estimator is robust to the presence of selection in the sense that incorporating selection into the model leaves the estimator unchanged. We study the properties of the estimator by simulation and show that its distribution can be quite sensitive to the underlying mutation rates.

Introduction

Recombination is a fundamental evolutionary force which shuffles genetic variation along a chromosome and gives rise to new haplotypes not previously seen in a population. It is a major goal of population genetics to infer rates of recombination along the genome and to disentangle its effects from other evolutionary forces such as mutation, selection, migration, and genetic drift. However, the effects of recombination can be difficult to detect; generally the signal of recombination is weak and a single recombination event may leave no discernible trace in a sample of genetic data (Hayman et al. 2022). Typically one observes a sample from the state of the population only at the present day, while the evolutionary history of the population, which can be much more informative for recombination, is a latent, unobserved variable. A wide range of inferential methods tackle this problem by positing a generative reproductive model for the population and integrating over all possible evolutionary histories, or by approximating this idea. A popular model is the diffusion limit of a Cannings-type model for recombination, genetic drift, and mutation. Under this limit the evolution of haplotype frequencies follows the Wright–Fisher diffusion with recombination (Ohta and Kimura 1969a, b) while the genealogical history of a sample is known as the ancestral recombination graph (ARG) (Griffiths and Marjoram 1997). Reconstruction of ARGs is a major current endeavour (see Peñalba and Wolf 2020, for recent review), and with the very large samples available in recent datasets it becomes ever more necessary to introduce computational and/or model heuristics.

In this paper we address a related question: in the idealised situation in which one observes the entire evolutionary history of a population, as defined via the trajectory of haplotype frequencies in the diffusion limit, can we define an estimator for the recombination rate based on this observation and derive its properties? Although observing the entire sample path of a diffusion is unrealistic in practice, we may regard the corresponding estimator as setting an upper bound on the information about recombination available to us. We note that statistical inference from a continuously observed diffusion is by now a standard problem; see Kutoyants (2004) for textbook treatment for scalar diffusions (though regularity conditions imposed throughout that work preclude most of it applying to the Wright–Fisher diffusion even in one dimension). Further, advances in sequencing technologies are leading to growing availability of genetic data sampled from a population across different times, sometimes over very long timescales, and providing great potential for improved statistical inference (Dehasque et al. 2020); such datasets can be considered as discrete, noisy versions of the idealised setting studied in this paper.

A motivation for this work is Watterson (1979) who derived the maximum likelihood estimator $\widehat{s}$ for natural selection from an observation of the trajectory of a Wright–Fisher diffusion (here a diallelic, one-locus model comprising only selection and genetic drift). He found the complete distribution of the estimator. It is worth noting that in this model $\widehat{s}$ does not enjoy the usual desirable asymptotic properties such as consistency, since one of the alleles will almost surely go extinct in finite time and thus the total information available about the parameter up to time T remains finite as $T\to \infty$. If we introduce bidirectional recurrent mutation to the model then it becomes ergodic, and Sant et al. (2022) have recently shown that in this situation the estimator enjoys the properties of consistency (uniformly over compact subsets of the parameter space) as well as asymptotic normality and asymptotic efficiency. We will see that, with or without mutation, the estimator for recombination behaves very differently to that of selection because the ‘information’ (defined formally below) can become infinite in finite time. Essentially, in a model of selection the signal-to-noise ratio for the selection parameter remains finite on hitting a boundary of the simplex of possible frequencies, while for the recombination parameter it may not. We will see that if the information becomes infinite then the maximum likelihood estimator (MLE) for recombination becomes exact, ${\widehat{\rho }}_{\text{MLE}}=\rho$.

The paper is structured as follows. In Sect. 2 we summarise likelihood theory for a continuously observed diffusion and specialise it to the infinitesimal variance of a Wright–Fisher diffusion. In Sect. 3 we derive the MLE for a general Wright–Fisher diffusion with arbitrary infinitesimal drift subject only to the constraint that the drift is linear in its unknown parameters. We then specialise this to the model of our primary interest, a multi-locus model with unknown recombination rate. Throughout we focus on the two-locus case which illustrates the main ideas without complicating the notation. Our main results are to derive an expression for the MLE and to show that if the information explodes then it is possible to learn the recombination parameter without error. Section 4 studies the impact of the presence of selection on this estimator, and in Sect. 5 we conduct a simulation study to investigate the empirical properties of the MLE. We discuss some potential directions for future work in Sect. 6.

Likelihood in diffusion paths

General case

We first give a summary of general likelihood-based inference for the parameters of a diffusion before specialising to the Wright–Fisher diffusion. Let $\{X(t):t\ge 0\}$ be a d-dimensional diffusion process and suppose its path $\{X(t):t\in [0,T]\}$ is observed up to time T. The generator of the diffusion has a form

$$\begin{array}{c}\hfill \mathcal{L}=\frac{1}{2}\sum _{i,j=1}^d{V}_{\mathit{ij}}(x)\frac{{\partial }^2}{\partial x_i\partial x_j}+\sum _{i=1}^d{\mu }_i(x;\phi )\frac{\partial }{\partial x_i},\end{array}$$ 1

where the model has r parameters $\phi ={({\phi }_1,\cdots ,{\phi }_r)}^{\top }$ in a parameter space $\mathrm{\Theta }$. We assume the drift $\mu ={({\mu }_1\cdots ,{\mu }_d)}^{\top }$ can be written in the form

$$\begin{array}{c}\hfill \mu (x;\phi )=c(x)+a(x;\phi ),\end{array}$$ 2

with $a(x;{\phi }_0)\equiv 0$ for a fixed reference parameter ${\phi }_0$, and $c(·)$ does not contain any parameters to be estimated. (For example, later we will estimate the rate of recombination in the presence of recurrent mutation with the latter having rates fixed and known. Then $a(·;\phi )$ will correspond to the contribution of recombination while $c(·)$ will correspond to the contribution of mutation, containing known mutation parameters.)

We will denote the corresponding path measure on continuous functions from [0, T] to ${\mathbb{R}}^d$ by ${\mathbb{P}}_{\phi }^{(T)}$.

If the $d\times d$ matrix $V=(V_{\mathit{ij}})$ is non-singular for almost all $t\in [0,T]$ and, for each $\phi \in \mathrm{\Theta }$,

$$\begin{array}{c}\hfill {\mathbb{P}}_{\phi }^{(T)}(I_{\mathit{ij}}<\infty ,i,j=1,\cdots ,r)=1,\end{array}$$ 3

where $I_T=(I_{\mathit{ij}})$ is the $r\times r$ observed information matrix

$$\begin{array}{c}\hfill I_T={\int }_0^{T}Z{(X(t);\phi )}^{\top }{V}^{-1}(X(t))Z(X(t);\phi )\text{d}t,Z_{\mathit{ij}}(x;\phi )=\frac{\partial a_i(x;\phi )}{\partial {\phi }_j},\end{array}$$ 4

then the likelihood for $\phi$ takes the form of a Radon–Nikodym derivative

$$\begin{array}{c}\hfill L_T(\phi )=\frac{\text{d}{\mathbb{P}}_{\phi }^{(T)}}{\text{d}{\mathbb{P}}_{{\phi }_0}^{(T)}}\end{array}$$

given with respect to a dominating measure which here we have chosen to be the model parametrised by ${\phi }_0$ so that ${\mathbb{P}}_{{\phi }_0}^{(T)}$ is the distribution over paths with drift c. Under these conditions, the likelihood takes the form

$$\begin{array}{cc}\hfill & L_T(\phi )=exp\left({\int }_0^T,a,{(X(t);\phi )}^{\top },V,{(X(t))}^{-1},,,\text{d},\tilde{X},(t)\right)\hfill \\ \hfill & \left(-,\frac{1}{2},{\int }_0^T,a,{(X(t);\phi )}^{\top },V,{(X(t))}^{-1},a,(X(t);\phi ),,,\text{d},t\right),\hfill \end{array}$$ 5

where

$$\begin{array}{c}\hfill \tilde{X}(t)=X(t)-{\int }_0^{t}c(X(s))\text{d}s.\end{array}$$ 6

The first integral in (5) is with respect to the path $\{\tilde{X}(t):t\in [0,T]\}$ and the second is with respect to t.

A heuristic way to understand Eq. (5) is as follows. Let $\mathrm{\Delta }X(t)=X(t+\mathrm{\Delta }t)-X(t)$. The distribution of $\mathrm{\Delta }X(t)$ given $X(t)=x$ is taken as approximately normal with mean $\mu (x)\mathrm{\Delta }t$ and covariance matrix $V(x)\mathrm{\Delta }t$ as $\mathrm{\Delta }t\to 0$. If V(x) is non-singular, then the quadratic form in the exponent of the normal density of $\mathrm{\Delta }X(t)$ is

$$\begin{array}{c}\hfill [\mathrm{\Delta }X(t)-\mu (x)\mathrm{\Delta }t{]}^{\top }[\mathrm{\Delta }tV(X){]}^{-1}[\mathrm{\Delta }X(t)-\mu (x)\mathrm{\Delta }t]=\\ \hfill \mu {(x)}^{\top }V{(x)}^{-1}\mu (x)\mathrm{\Delta }t-2\mu {(x)}^{\top }V{(x)}^{-1}\mathrm{\Delta }X(t)+\mathcal{O}({(\mathrm{\Delta }t)}^2).\end{array}$$

We are expressing this density with respect to another normal density with mean $c(x)\mathrm{\Delta }t$ and covariance matrix $V(x)\mathrm{\Delta }t$, and thus we subtract the corresponding quadratic form

$$\begin{array}{c}\hfill c{(x)}^{\top }V{(x)}^{-1}c(x)\mathrm{\Delta }t-2c{(x)}^{\top }V{(x)}^{-1}\mathrm{\Delta }X(t)+\mathcal{O}({(\mathrm{\Delta }t)}^2).\end{array}$$

After some rearrangement, letting $\mathrm{\Delta }t\to 0$, and integrating from 0 to T, we recover the quadratic form appearing in (5). Note that the likelihood ignores the terms |V(x)| in the diffusion, since we assume that the likelihood is with respect to a parametric form only for $\mu$. (Statistical inference for parameters of V would be trivial in this setting, since V is identifiable from the path via its quadratic variation.) See Basawa and Prakasa Rao (1980, Ch. 9) and Kloeden et al. (2003, Ch. 6) for further details on the general case.

Wright–Fisher diffusion

The family of Wright–Fisher diffusions has generator (1) with diffusion coefficient of the form

$$\begin{array}{c}\hfill V_{\mathit{ij}}(x)=x_i({\delta }_{\mathit{ij}}-x_j),\end{array}$$

where ${\delta }_{\mathit{ij}}$ denotes the Kronecker delta (i.e. ${\delta }_{\mathit{ij}}=1$ if $i=j$ and ${\delta }_{\mathit{ij}}=0$ if $i\ne j$). The diffusion takes values in the simplex

$$\begin{array}{c}\hfill {\mathrm{\Delta }}_{d-1}:=\left\{x,\in ,{[0,1]}^d,:,,\sum _{i=1}^d,x_i,=,1\right\},\end{array}$$

and the domain of $\mathcal{L}$ is $\mathcal{D}(\mathcal{L})=C^2({\mathrm{\Delta }}_{d-1})$, twice continuously differentiable functions with domain ${\mathrm{\Delta }}_{d-1}$. For now we continue to leave the drift in the form (2) but otherwise unspecified.

The matrix V(x) is singular since ${\sum }_{i=1}^d{x}_i=1$. Our first task, then, is to modify the results from Sect. 2.1 to accommodate this issue. We achieve this by studying the first $d-1$ coordinates of X, whose infinitesimal covariance matrix $V^{\ast }(x)$ is non-singular. Fortunately, its inverse $V^{\ast }{(x)}^{-1}$ takes on a particularly simple form, as we now show.

Theorem 1

Assume (3) holds for a Wright–Fisher diffusion with drift coefficient $\mu (x;\phi )=c(x)+a(x;\phi )$ and diffusion coefficient $V=(V_{\mathit{ij}})$, $V_{\mathit{ij}}(x)=x_i({\delta }_{\mathit{ij}}-x_j)$. Then the likelihood is

$$\begin{array}{c}\hfill L_T(\phi )=exp\left({\int }_0^T,\sum _{i=1}^d,\frac{a_i(X(t);\phi )}{X_i(t)},,,\text{d},{\tilde{X}}_i,(t),-,\frac{1}{2},{\int }_0^T,\sum _{i=1}^d,\frac{a_i{(X(t);\phi )}^2}{X_i(t)},,,\text{d},t\right),\end{array}$$ 7

with $\tilde{X}$ given by (6).

Proof

We consider the diffusion $(X_1(t),\cdots ,X_{d-1}(t))$ with drift ${\mu }^{\ast }(x;\phi )={({\mu }_1(x;\phi ),\cdots ,{\mu }_{d-1}(x;\phi ))}^{\top }$ and non-singular $(d-1)\times (d-1)$ covariance matrix $V^{\ast }(x)$. Define $X_d(t)=1-{\sum }_{i=1}^{d-1}{X}_i(t)$ and ${\mu }_d(x;\phi )=-{\sum }_{i=1}^d{\mu }_i(x;\phi )$. It follows from standard normal theory, for example Kendall et al. (1994, p520–521), that

$$\begin{array}{c}\hfill {[V^{\ast }{(x)}^{-1}]}_{\mathit{ij}}=(x_d^{-1}+x_i^{-1}{\delta }_{\mathit{ij}}).\end{array}$$ 8

We know that ${\sum }_{i=1}^d{a}_i(x;\phi )=0$ and ${\sum }_{i=1}^d\text{d}{\tilde{X}}_i(t)=0$ (since both ${\sum }_{i=1}^d\text{d}{X}_i(t)=0$ and ${\sum }_{i=1}^d{c}_i(X(t))=0$, the latter required for X to take values in ${\mathrm{\Delta }}_{d-1}$ when $\phi ={\phi }_0$), so

$$\begin{array}{cc}\hfill a^{\ast }{(X(t);\phi )}^{\top }{V}^{\ast }{(X(t))}^{-1}\text{d}\tilde{X}(t)& =\sum _{i=1}^{d-1}\sum _{j=1}^{d-1}{a}_i(X(t);\phi )(X_d{(t)}^{-1}+{\delta }_{\mathit{ij}}{X}_i{(t)}^{-1})\text{d}{\tilde{X}}_j(t)\hfill \\ \hfill & =\frac{a_d(X(t);\phi )}{X_d(t)}\text{d}{\tilde{X}}_d(t)+\sum _{i=1}^{d-1}\frac{a_i(X(t);\phi )}{X_i(t)}\text{d}{\tilde{X}}_i(t)\hfill \\ \hfill & =\sum _{i=1}^d\frac{a_i(X(t);\phi )}{X_i(t)}\text{d}{\tilde{X}}_i(t),\hfill \end{array}$$ 9

with a similar calculation for

$$\begin{array}{cc}\hfill a^{\ast }{(X(t);\phi )}^{\top }{V}^{\ast }{(X(t))}^{-1}{a}^{\ast }(X(t);\phi )& =\sum _{i=1}^{d-1}\sum _{j=1}^{d-1}{a}_i(X(t);\phi )a_j(X(t);\phi ){V_{\mathit{ij}}^{\ast }(X(t))}^{-1}\hfill \\ \hfill & =\sum _{i=1}^d\frac{a_i{(X(t);\phi )}^2}{X_i(t)}.\hfill \end{array}$$ 10

Substituting (9) and (10) into (5) yields (7). $\square$

Theory for maximum likelihood estimators

General Wright–Fisher diffusion

Our next goal is to derive an MLE for the parameters $\phi$ of a Wright–Fisher diffusion. This is found by differentiating the log-likelihood with respect to the parameters. In all the examples we encounter, the drift is a linear function of the parameters so for the remainder of this article we assume $a(x;\phi )$ to be of the form

$$\begin{array}{c}\hfill a_i(x;\phi )=\sum _{k=1}^r{Z}_{\mathit{ik}}(x){\phi }_k,\end{array}$$ 11

where $Z_{\mathit{ik}}(x)=\frac{\partial a_i(x;\phi )}{\partial {\phi }_k}$ does not depend on $\phi$. To avoid issues of identifiability we suppose that the columns of $Z=(Z_{\mathit{ij}})$ are linearly independent functions. Then from Theorem 1 the log-likelihood is a quadratic function

$$\begin{array}{cc}\hfill log{L}_T(\phi )=& \sum _{k=1}^r{\phi }_k{\int }_0^T\sum _{i=1}^d\frac{Z_{\mathit{ik}}(X(t))}{X_i(t)}\text{d}{\tilde{X}}_i(t)\hfill \\ \hfill & -\frac{1}{2}\sum _{k=1}^r\sum _{l=1}^r{\phi }_k{\phi }_l{\int }_0^T\sum _{i=1}^d\frac{Z_{\mathit{ik}}(X(t))Z_{\mathit{il}}(X(t))}{X_i(t)}\text{d}t,\hfill \end{array}$$

with a unique maximum, $\widehat{\phi }$, in ${\mathbb{R}}^r$, which is the solution of the set of equations for $k=1,\cdots ,r$:

$$\begin{array}{c}\hfill 0={\int }_0^T\sum _{i=1}^d\frac{Z_{\mathit{ik}}(X(t))}{X_i(t)}\text{d}{\tilde{X}}_i(t)-\sum _{l=1}^r{\phi }_l{\int }_0^T\sum _{i=1}^d\frac{Z_{\mathit{ik}}(X(t))Z_{\mathit{il}}(X(t))}{X_i(t)}\text{d}t.\end{array}$$ 12

The set of Eq. (12) are familiar in regression theory. Now denote the ($r\times 1$) vector $Y=(Y_k)$ with elements

$$\begin{array}{c}\hfill Y_k={\int }_0^T\sum _{i=1}^d\frac{Z_{\mathit{ik}}(X(t))}{X_i(t)}\text{d}{\tilde{X}}_i(t),\end{array}$$

and let $\mathrm{\Sigma }(X(t))=\text{diag}(X_i(t))$. Then the Eq. (12) can be written

$$\begin{array}{c}\hfill [{\int }_0^{T}Z{(X(t))}^{\top }{\mathrm{\Sigma }}^{-1}(X(t))Z(X(t))\text{d}t]\widehat{\phi }=Y.\end{array}$$

Continuing to assume (3), the matrix on the left-hand side of the previous equation is non-singular (see Basawa and Prakasa Rao (1980, p223–224), Kloeden et al. (2003, p231)) and hence we arrive at the form

$$\begin{array}{c}\hfill \widehat{\phi }=[{\int }_0^{T}Z{(X(t))}^{\top }{\mathrm{\Sigma }}^{-1}(X(t))Z(X(t))\text{d}t{]}^{-1}Y.\end{array}$$ 13

Of course if $\mathrm{\Theta }\subset {\mathbb{R}}^r$ then it is not guaranteed that $\widehat{\phi }\in \mathrm{\Theta }$, and $\widehat{\phi }$ must be adjusted appropriately to ensure it is the MLE. An example of this adjustment is given later.

The observed information matrix (4) is a key quantity in telling us about how informative the data is for $\phi$. For this model, the observed information matrix $I_T$ has elements

$$\begin{array}{c}\hfill I_{\mathit{kl}}={\int }_0^T\sum _{i=1}^d\frac{1}{X_i(t)}{Z}_{\mathit{ik}}(X(t))Z_{\mathit{il}}(X(t))\text{d}t\end{array}$$ 14

using linearity of $a_i(x;\phi )$; the information matrix does not depend on $\widehat{\phi }$. The expression (13) for $\widehat{\phi }$ can be written

$$\begin{array}{c}\hfill \widehat{\phi }=I_T^{-1}Y.\end{array}$$

Deterministic model

As a check on the expression for $\widehat{\phi }$, we can ask for the estimator we would obtain if the observed trajectory is that of the deterministic model

$$\begin{array}{c}\hfill \frac{\text{d}{x}_i}{\text{d}t}={\mu }_i(x;\phi ),i=1,\cdots ,d.\end{array}$$ 15

Now from (2) and (6) we find

$$\begin{array}{c}\hfill \text{d}\tilde{x}=\text{d}x-c(x)\text{d}t=(c(x)+a(x;\phi ))\text{d}t-c(x)\text{d}t=a(x;\phi )\text{d}t,\end{array}$$

and so we can substitute this expression for $\text{d}\tilde{x}$ into the likelihood Eq. (12) to obtain that $\widehat{\phi }$ is a solution to

$$\begin{array}{c}\hfill 0={\int }_0^T\sum _{i=1}^d\frac{[a_i(x(t);\phi )-a_i(x(t);\widehat{\phi })]}{x_i(t)}\frac{\partial a_i(x(t);\widehat{\phi })}{\partial {\phi }_l}\text{d}t.\end{array}$$ 16

Owing to the factor $a_i(x(t);\phi )-a_i(x(t);\widehat{\phi })$, it is clear that a solution to the likelihood equation is given by $\widehat{\phi }=\phi$. It is reassuring that the estimator is well behaved even in this crude level of approximation; the trajectory defined by (15) is not a realisation from the assumed model since it is a path of bounded variation.

Neutral two-locus model

We now turn to our main result, an expression for the MLE for the recombination parameter $\rho \in [0,\infty )=:\mathrm{\Theta }$. Consider a neutral two-locus model in which there are K possible alleles at the first locus, locus A, and L possible alleles at the second, locus B. The haplotype of an individual is denoted $(i,j)\in \{1,\cdots ,K\}\times \{1,\cdots ,L\}$, and its frequency in the population is $x_{\mathit{ij}}$. Note that to reconcile this double-index notation with previous sections we must implicitly stack the KL possible haplotypes in some agreed order into a vector of length $d=KL$. We will switch between the two notations as required. To emphasise when haplotypes have been stacked we will use a bold index, so $x_{\mathit{i}}$ denotes the frequency of haplotype $\mathit{i}$, $\mathit{i}=1,\cdots ,d$.

The model is completed by specifying the drift. Here it is of the form

$$\begin{array}{c}\hfill a_{\mathit{ij}}(x;\rho )=\rho (x_{i·}{x}_{·j}-x_{\mathit{ij}}),i=1,\cdots ,K;j=1,\cdots ,L,\end{array}$$

where $x_{i·}:={\sum }_{l=1}^L{x}_{\mathit{il}}$ and $x_{·j}:={\sum }_{k=1}^K{x}_{\mathit{kj}}$. Recombination occurs between the two loci at rate $\rho$; specifically this is a model of the homologous crossing-over that takes place during meiosis. To simplify later results, we omit the conventional factor of 1/2 in the recombination rate parameter.

In much of what follows the choice for c(x) is immaterial, but for concreteness we will set

$$\begin{array}{c}\hfill c_{\mathit{ij}}(x)=\frac{{\theta }_A}{2}\sum _{k=1}^K{x}_{\mathit{kj}}(P_{\mathit{ki}}^A-{\delta }_{\mathit{ik}})+\frac{{\theta }_B}{2}\sum _{l=1}^L{x}_{\mathit{il}}(P_{\mathit{lj}}^B-{\delta }_{\mathit{jl}}).\end{array}$$

Here mutation takes place at locus A and B at respective rates ${\theta }_A/2$ and ${\theta }_B/2$ on the timescale of the diffusion. When a mutation occurs, the change in allele is governed by the $K\times K$ and $L\times L$ mutation transition matrices $P^A$ and $P^B$ (i.e. if a mutation occurs at locus A on haplotype (k, j) then it mutates to haplotype (i, j) with probability $P_{\mathit{ki}}^A$, $i=1,\cdots ,K$; similarly for $P^B$). We allow ${\theta }_A,{\theta }_B\ge 0$, so the model may or may not be ergodic.

Note the separate roles for the two components of the drift: here it is only $\rho$ to be estimated, with the other parameters appearing in $c(·)$ considered known. The likelihood is expressed with respect to the parametrisation ${\rho }_0=0$, a model in which the two loci are completely linked but the mutation parameters are the same.

Using the results of Sect. 3.1, for this model the log-likelihood is

$$\begin{array}{cc}\hfill log{L}_T(\rho )=& \rho {\int }_0^T\sum _{i=1}^K\sum _{j=1}^L(\frac{X_{i·}(t)X_{·j}(t)}{X_{\mathit{ij}}(t)}-1)\text{d}{\tilde{X}}_{\mathit{ij}}(t)\hfill \\ \hfill & -\frac{1}{2}{\rho }^2{\int }_0^T\sum _{i=1}^K\sum _{j=1}^L\frac{{(X_{\mathit{ij}}(t)-X_{i·}(t)X_{·j}(t))}^2}{X_{\mathit{ij}}(t)}\text{d}t\hfill \\ \hfill =& \rho {\int }_0^T\sum _{i=1}^K\sum _{j=1}^L\frac{X_{i·}(t)X_{·j}(t)}{X_{\mathit{ij}}(t)}\text{d}{\tilde{X}}_{\mathit{ij}}(t)\hfill \\ \hfill & -\frac{1}{2}{\rho }^2{\int }_0^T\sum _{i=1}^K\sum _{j=1}^L\frac{{(X_{\mathit{ij}}(t)-X_{i·}(t)X_{·j}(t))}^2}{X_{\mathit{ij}}(t)}\text{d}t,\hfill \end{array}$$ 17

where for the second equality we recall ${\sum }_{i=1}^K{\sum }_{j=1}^L\text{d}{\tilde{X}}_{\mathit{ij}}(t)=0$, with

$$\begin{array}{c}\hfill {\tilde{X}}_{\mathit{ij}}(t)=X_{\mathit{ij}}(t)-{\int }_0^t{c}_{\mathit{ij}}(X(s))\text{d}s,i=1,\cdots ,K;j=1,\cdots ,L.\end{array}$$

The estimator $\widehat{\rho }$ is therefore

$$\begin{array}{c}\hfill \widehat{\rho }=\frac{{\displaystyle {\int }_0^T\sum _{i=1}^K\sum _{j=1}^L\frac{X_{i·}(t)X_{·j}(t)}{X_{\mathit{ij}}(t)}\text{d}{\tilde{X}}_{\mathit{ij}}(t)}}{{\displaystyle {\int }_0^T\sum _{i=1}^K\sum _{j=1}^L\frac{{(X_{\mathit{ij}}(t)-X_{i·}(t)X_{·j}(t))}^2}{X_{\mathit{ij}}(t)}\text{d}t}},\end{array}$$ 18

and the observed information is

$$\begin{array}{c}\hfill I_T={\int }_0^T\sum _{i=1}^K\sum _{j=1}^L\frac{{(X_{\mathit{ij}}(t)-X_{i·}(t)X_{·j}(t))}^2}{X_{\mathit{ij}}(t)}\text{d}t.\end{array}$$ 19

The denominator in $\widehat{\rho }$ and the information can be simplified to

$$\begin{array}{c}\hfill I_T={\int }_0^T\sum _{i=1}^K\sum _{j=1}^L\frac{X_{i·}{(t)}^2{X}_{·j}{(t)}^2}{X_{\mathit{ij}}(t)}\text{d}t-T.\end{array}$$

It is worth remarking on the functional form (17) for $log{L}_T(\rho )$. This is a polynomial in $\rho$ and we can think of a trade-off between the order of the polynomial and the complexity of its coefficients. In this model we have a particularly simple quadratic polynomial in $\rho$, order only two, with the benefit of knowing that the function is convex with a unique finite maximum (since the coefficient of ${\rho }^2$ is negative). The price we pay is that the coefficients of the polynomial are highly cumbersome in the sense that they are given as integrals over the sample path of a diffusion. Contrast this with the dual coalescent model in which the likelihood for an observed sample path of an ARG would be a product of exponential waiting time densities times a product of rational functions for the transitions of the jump chain. With many possible jumps, these rational functions may be constructed from polynomials in $\rho$ of very high order, though their coefficients are much simpler than the stochastic integrals encountered here. In a coalescent model, the shape of the likelihood curve as a function of $\rho$ can be rather complicated, even exhibiting local minima when integrating over ARGs (Jenkins and Song 2009).

Deterministic model

Is $\widehat{\rho }$ in (18) a reasonable estimate? Again we can check what happens when X solves a deterministic model. Setting ${\theta }_A={\theta }_B=0$ for the moment, the deterministic model is

$$\begin{array}{c}\hfill \frac{\text{d}{x}_{\mathit{ij}}}{\text{d}t}=\rho (x_{i·}{x}_{·j}-x_{\mathit{ij}}),i=1,\cdots ,K;j=1,\cdots ,L.\end{array}$$ 20

and substituting $\text{d}{x}_{\mathit{ij}}$ directly into (18) again shows that $\widehat{\rho }=\rho$.

We can further describe the evolution of $I_T$. Note that in this deterministic model (summing (20) over j):

$$\begin{array}{c}\hfill \frac{\text{d}{x}_{i·}}{\text{d}t}=0,i=1,\cdots ,K.\end{array}$$

Therefore $x_{i·}(t)=x_{i·}(0)$ for all $t\ge 0$, and similarly for $x_{·j}(t)$. The solution to (20) is then

$$\begin{array}{c}\hfill x_{\mathit{ij}}(t)=x_{\mathit{ij}}(0)e^{-\rho t}+x_{i·}(0)x_{·j}(0)(1-e^{-\rho t}).\end{array}$$ 21

We have that

$$\begin{array}{c}\hfill log\left(\frac{x_{\mathit{ij}}(T)}{x_{\mathit{ij}}(0)}\right)={\int }_0^T\frac{\text{d}{x}_{\mathit{ij}}}{x_{\mathit{ij}}}=\rho \left({\int }_0^T,\frac{x_{i·}{x}_{·j}}{x_{\mathit{ij}}},,,\text{d},t,-,T\right),\end{array}$$

so (provided $\rho >0$):

$$\begin{array}{c}\hfill I_T={\rho }^{-1}\sum _{i=1}^K\sum _{j=1}^L{x}_{i·}(0)x_{·j}(0)log\left(\frac{x_{\mathit{ij}}(T)}{x_{\mathit{ij}}(0)}\right).\end{array}$$ 22

The limit information is therefore

$$\begin{array}{c}\hfill \underset{T\to \infty }{lim}{I}_T={\rho }^{-1}\sum _{i=1}^K\sum _{j=1}^L{x}_{i·}(0)x_{·j}(0)log\left(\frac{x_{i·}(0)x_{·j}(0)}{x_{\mathit{ij}}(0)}\right).\end{array}$$

As far as the deterministic model goes, the information is in the transient phase until the frequencies come to equilibrium. The accumulated information $I_T$ remains finite as $T\to \infty$. In a stochastic model on the other hand, we will see that the injection of noise allows $I_T\to \infty$ as $T\to \infty$. We note that one should regard this contrasting behaviour with caution: it does not mean that the estimator is consistent only in the stochastic setting. We have just seen that $\widehat{\rho }=\rho$ in the deterministic setting, which is trivially consistent, and creates a paradox when we try to reconcile this fact with the asymptotic finiteness of $I_T$. The paradox is resolved by noting that the data-generating mechanism differs from the one assumed in designing the estimator. Had we assumed a deterministic model throughout our analysis then, since the parameter is simply a rate appearing in an observed ODE, the ‘likelihood’ would be a point mass on the true rate and the MLE would be equal to that true rate. The ‘information’ in this setting, being the curvature of the log-likelihood, is immediately infinite. The fact that $\widehat{\rho }=\rho$ demonstrates that the estimator adapts automatically to a change in data-generating mechanism. The quantity $I_T$ could be regarded not as the information under the true model but as a way of quantifying ‘the informativeness of the deterministic trajectory under stochastic assumptions’. It is this quantity that remains finite as $T\to \infty$.

It is possible to repeat these calculations for a model with ${\theta }_A,{\theta }_B>0$; that is, to solve the deterministic mutation-recombination equation. Again $I_T$ converges to a finite limit; see Appendix A.

Stochastic differential equation interpretation

We can find an expression for the error associated with $\widehat{\rho }$ by regarding X(t) as the solution to a stochastic differential equation (SDE):

$$\begin{array}{c}\hfill \text{d}X(t)=[c(X(t))+a(X(t);\phi )]\text{d}t+\sigma (X(t))\text{d}W(t),X(0)=x(0),\end{array}$$ 23

where W is a $(d-1)$-dimensional Brownian motion and $\sigma (x)$ is a (non-unique) $d\times (d-1)$ matrix satisfying $\sigma (x)\sigma {(x)}^{\top }=V(x)$.

There are various ways to define $\sigma$ subject to this constraint. It is common to ask for $\sigma$ to be lower triangular by applying a Cholesky decomposition to V. In the case of the covariance matrix of the Wright–Fisher diffusion, the Cholesky decomposition is given analytically by Sato (1976), though it is one that explodes at the boundaries.

Proposition 1

The error associated with $\widehat{\rho }$ is

$$\begin{array}{c}\hfill \widehat{\rho }-\rho =-\frac{{\displaystyle {\int }_0^T\sum _{\mathit{i}=1}^d\frac{D_{\mathit{i}}(t)}{X_{\mathit{i}}(t)}\sum _{\mathit{j}=1}^{d-1}{\sigma }_{\mathit{i}\mathit{j}}(X(t))\text{d}{W}_{\mathit{j}}(t)}}{{\displaystyle {\int }_0^T\sum _{\mathit{i}=1}^d\frac{D_{\mathit{i}}{(t)}^2}{X_{\mathit{i}}(t)}\text{d}t}},\end{array}$$ 24

where $D_{\mathit{i}}(t)=X_{i_1{i}_2}(t)-X_{i_1·}(t)X_{·i_2}(t)$ is the coefficient of linkage disequilibrium for haplotype $\mathit{i}=(i_1,i_2)$.

Proof

Rearranging (18) slightly and using ${\sum }_{\mathit{i}=1}^d\text{d}{\tilde{X}}_{\mathit{i}}(t)=0$, we have

$$\begin{array}{c}\hfill \widehat{\rho }{I}_T=-{\int }_0^T\sum _{\mathit{i}=1}^d\frac{D_{\mathit{i}}(t)}{X_{\mathit{i}}(t)}\text{d}{\tilde{X}}_{\mathit{i}}(t).\end{array}$$

Now substituting for $\text{d}{\tilde{X}}_{\mathit{i}}(t)=\text{d}{X}_{\mathit{i}}(t)-c_{\mathit{i}}(X(t))\text{d}t$ using (23),

$$\begin{array}{c}\hfill \widehat{\rho }{I}_T=\rho I_T-{\int }_0^T\sum _{\mathit{i}=1}^d\frac{D_{\mathit{i}}(t)}{X_{\mathit{i}}(t)}\sum _{\mathit{j}=1}^{d-1}{\sigma }_{\mathit{i}\mathit{j}}(X(t))\text{d}{W}_{\mathit{j}}(t),\end{array}$$

which leads to (24). $\square$

Thus the bias and mean squared error of $\widehat{\rho }$ are given respectively by the expectation of the term on the right-hand side of (24) and the expectation of its square. Estimators of this form are not unbiased in general (Basawa and Prakasa Rao 1980, p 218).

Corrected MLE

There are two problems with the estimator $\widehat{\rho }$ defined in (18). First, the parameter space is $\mathrm{\Theta }=[0,\infty )$ but we cannot ensure $\widehat{\rho }\ge 0$. (Although $\rho <0$ is biologically unrealistic, mathematically it is nonetheless a valid model and a sample path may point to this region of the parameter space if $\text{d}{\tilde{X}}_{\mathit{ij}}(t)$ is sufficiently negative.) This is easily corrected by applying a rectified linear unit, $max\{0,\widehat{\rho }\}$. The second issue is more serious: it is not guaranteed that (3) holds. In other words, we have not ruled out the possibility that $I_T$ explodes in finite time. For observations for which $I_T<\infty$, we can still interpret (17) as a quasi-log-likelihood function (Kloeden et al. 2003, p231), but otherwise we must treat $L_T(\rho )$ as a generalized density valid only until the stopping time

$$\begin{array}{c}\hfill S:=inf\left\{t,\in ,[0,\infty ),:,,I_t,=,\infty \right\}.\end{array}$$ 25

See Liptser and Shiryaev (2001, Ch. 6) and Mijatović et al. (2012) for further discussion on this subtle point. Writing $\widehat{\rho }={\widehat{\rho }}_T$ for the estimator in (18), we define the following corrected estimator:

$$\begin{array}{c}\hfill {\widehat{\rho }}_{\text{MLE}}:={\mathbb{I}}_{[0,S)}(T)max\left\{0,,,{\widehat{\rho }}_T\right\}+{\mathbb{I}}_{[S,\infty )}(T)\underset{t\uparrow S}{lim}{\widehat{\rho }}_t.\end{array}$$ 26

A similar issue arises in the estimation of the immigration rate of the continuous branching with immigration (CBI) diffusion, where a related correction is proposed (Overbeck 1998, in particular Theorem 2(iv)). The subscript in (26) rather suggestively posits this quantity as the MLE; this is proven shortly, in Corollary 1. Although taking ${lim}_{t\uparrow S}{\widehat{\rho }}_t$ in (26) might seem to be unstable, that this is the appropriate correction to our estimator is justified by the following theorem.

Theorem 2

If $S\le T$ then ${\widehat{\rho }}_{\text{MLE}}=\rho$ with probability 1.

Proof

From the definition (26) of ${\widehat{\rho }}_{\text{MLE}}$ it suffices to show that $\widehat{\rho }\to \rho$ as $T\uparrow S$. Let

$$\begin{array}{c}\hfill N_T=-{\int }_0^{T\wedge S}\sum _{\mathit{i}=1}^d\frac{D_{\mathit{i}}(t)}{X_{\mathit{i}}(t)}\sum _{\mathit{j}=1}^{d-1}{\sigma }_{\mathit{i}\mathit{j}}(X(t))\text{d}{W}_{\mathit{j}}(t).\end{array}$$

This is a continuous, stopped martingale with $N_0=0$ and quadratic variation

$$\begin{array}{cc}\hfill {⟨N⟩}_T=& {〈-,{\int }_0^{T\wedge S},\sum _{\mathit{i}=1}^d,\frac{D_{\mathit{i}}(t)}{X_{\mathit{i}}(t)},\sum _{\mathit{j}=1}^{d-1},{\sigma }_{\mathit{i}\mathit{j}},(X(t)),,,\text{d},W_{\mathit{j}},(t)〉}_T\hfill \\ \hfill =& {\int }_0^{T\wedge S}\sum _{\mathit{i}=1}^d\sum _{\mathit{j}=1}^{d-1}\frac{D_{\mathit{i}}(t)}{X_{\mathit{i}}(t)}{\sigma }_{\mathit{i}\mathit{j}}(X(t))\sum _{\mathit{k}=1}^d\frac{D_{\mathit{k}}(t)}{X_{\mathit{k}}(t)}{\sigma }_{\mathit{k}\mathit{j}}(X(t))\text{d}t\hfill \\ \hfill =& {\int }_0^{T\wedge S}\sum _{\mathit{i}=1}^d\sum _{\mathit{k}=1}^d\frac{D_{\mathit{i}}(t)}{X_{\mathit{i}}(t)}\frac{D_{\mathit{k}}(t)}{X_{\mathit{k}}(t)}{V}_{\mathit{i}\mathit{k}}(X(t))\text{d}t\hfill \\ \hfill =& {\int }_0^{T\wedge S}\sum _{\mathit{i}=1}^d\frac{D_{\mathit{i}}{(t)}^2}{X_{\mathit{i}}(t)}\text{d}t\hfill \\ \hfill =& I_{T\wedge S}.\hfill \end{array}$$

where the second equality uses that $⟨·,·⟩$ is a bilinear form and $⟨\text{d}{W}_{\mathit{j}},\text{d}{W}_{\mathit{l}}⟩={\delta }_{\mathit{j}\mathit{l}}\text{d}t$. Thus by the law of large numbers for local martingales (Revuz and Yor 1999, Ch. V.1, Exercise 1.16, p186),

$$\begin{array}{c}\hfill \underset{T\to \infty }{lim}\frac{N_T}{I_{T\wedge S}}=0\text{with probability 1 on}\{I_{\infty }=\infty \}.\end{array}$$

The limit as $T\uparrow S$ is the same. But $N_T/I_{T\wedge S}$ is precisely the error $\widehat{\rho }-\rho$ given in Proposition 1, so $\widehat{\rho }\to \rho$ as $T\uparrow S$ with probability 1. $\square$

Corollary 1

${\widehat{\rho }}_{\text{MLE}}$ is the MLE for $\rho$.

Proof

This follows since we have separately verified that it is the MLE on $\{T<S\}$ and on $\{S\le T\}$. In the latter case $\rho$ is identifiable, so $L_T(\rho )$ is zero anywhere other than the true value. $\square$

Corollary 2

The error associated with ${\widehat{\rho }}_{\text{MLE}}$ is

$$\begin{array}{c}\hfill {\widehat{\rho }}_{\text{MLE}}-\rho =-{\mathbb{I}}_{[0,S)}(T)\times \left\{\begin{array}{cc}\frac{{\displaystyle {\int }_0^T{\sum }_{\mathit{i}=1}^d\frac{D_{\mathit{i}}(t)}{X_{\mathit{i}}(t)}{\sum }_{\mathit{j}=1}^{d-1}{\sigma }_{\mathit{i}\mathit{j}}(X(t))\text{d}{W}_{\mathit{j}}(t)}}{{\displaystyle {\int }_0^T{\sum }_{\mathit{i}=1}^d\frac{D_{\mathit{i}}{(t)}^2}{X_{\mathit{i}}(t)}\text{d}t}},\hfill & \widehat{\rho }\ge 0,\hfill \\ \rho ,\hfill & \widehat{\rho }<0,\hfill \end{array}\right)\end{array}$$

where we recall that $\widehat{\rho }$ is the uncorrected estimator given in (18).

Proof

This follows by combining Theorem 2 and Proposition 1. $\square$

The relevance of Theorem 2 is: If the sample path is such that $I_T=\infty$, then we learn $\rho$ without error. Inspecting the form of $I_T$ in (19), we see that its integrand is locally integrable in the interior of ${\mathrm{\Delta }}_{d-1}$. Thus for $I_T=\infty$ it is necessary to have at least one haplotype frequency $X_{\mathit{ij}}(t)\to 0$ before time T. We should expect the same phenomenon when inferring the mutation parameters in a one-locus model, where hitting one of the boundaries is completely informative for one of the mutation parameters.

The next result shows that having $I_T=\infty$ is not a hypothetical concern, and the proof makes it clear that explosion of $I_T$ is intimately related with hitting a boundary of ${\mathrm{\Delta }}_{d-1}$.

Theorem 3

Suppose that $\rho +\frac{{\theta }_A}{2}+\frac{{\theta }_B}{2}<\frac{1}{2}$, that mutation is parent-independent (i.e. $P^A$ and $P^B$ each have identical rows), and that x(0) lies in the interior of ${\mathrm{\Delta }}_{d-1}$. Then $\mathbb{P}(I_T=\infty )>0$.

Proof

It is clear from the form of $I_T$ in (19) that $\{I_T=\infty \}$ will occur if for some i, j,

• (i)For some $\delta >0$ and for all $t\in [0,T]$, X(t) lies in $A_1:=\{x\in {\mathrm{\Delta }}_{d-1}:{(x_{\mathit{ij}}-x_{i·}{x}_{·j})}^2>\delta \}$;
• (ii)$T_{\epsilon }(X_{\mathit{ij}}):=inf\{t\in [0,\infty ):X_{\mathit{ij}}(t)=\epsilon \}$, the first hitting time of $\epsilon$ by $X_{\mathit{ij}}$, satisfies $T_0(X_{\mathit{ij}})\in (0,T]$; and
• (iii)
  The integral

  $$\begin{array}{c}\hfill {\int }_0^{T_{\epsilon }(X_{\mathit{ij}})}\frac{1}{X_{\mathit{ij}}(t)}\text{d}t\end{array}$$

  diverges as $\epsilon \to 0$.

Condition (iii) extracts the explosion of $I_T$ from a denominator of its integrand, while condition (i) controls the corresponding numerator. Condition (ii) ensures that such explosion takes place before time T.

To study the finiteness or otherwise of the integral in (iii), choose a decomposition $\sigma (x)\sigma {(x)}^{\top }=V(x)$ so that the component of the SDE (23) corresponding to $X_{\mathit{ij}}(t)$ has the form

$$\begin{array}{c}\hfill \text{d}{X}_{\mathit{ij}}(t)={\mu }_{\mathit{ij}}(X_{\mathit{ij}}(t))\text{d}t+\sqrt{X_{\mathit{ij}}(t)(1-X_{\mathit{ij}}(t))}\text{d}W(t),X_{\mathit{ij}}(0)=x_{\mathit{ij}}(0),\end{array}$$

for a scalar Brownian motion W, where

$$\begin{array}{cc}\hfill {\mu }_{\mathit{ij}}(X_{\mathit{ij}}(t))=& \rho [X_{i·}(t)X_{·j}(t)-X_{\mathit{ij}}(t)]+\frac{{\theta }_A}{2}[X_{·j}(t)P_i^A-X_{\mathit{ij}}(t)]\hfill \\ \hfill & +\frac{{\theta }_B}{2}[X_{i·}(t)P_j^B-X_{\mathit{ij}}(t)].\hfill \end{array}$$

The idea is to show that this SDE behaves locally like a one-locus model of mutation only. More precisely we will compare $X_{\mathit{ij}}$ to another diffusion which solves the SDE

$$\begin{array}{c}\hfill \text{d}Z(t)=\frac{\vartheta }{2}[P-Z(t)]\text{d}t+\sqrt{Z(t)(1-Z(t))}\text{d}W(t),Z(0)=x_{\mathit{ij}}(0),\end{array}$$

for some $\vartheta \in [0,1)$, $P\in (0,1)$. Choose $\vartheta$ so that $\rho +\frac{{\theta }_A}{2}+\frac{{\theta }_B}{2}<\frac{\vartheta }{2}$ and choose P so that $x_{i·}(0)x_{·j}(0)<P$, $P_i^A<P$, and $P_j^B<P$. Then on the set $A_2:=\{x\in {\mathrm{\Delta }}_{d-1}:x_{i·}{x}_{·j}<P\}$ it is straightforward to verify we have

$$\begin{array}{c}\hfill {\mu }_{\mathit{ij}}(x)<\frac{\vartheta }{2}(P-x),\end{array}$$

and thus by a standard comparison theorem (see Theorem 1.1 and Remark 1.1 in Ikeda and Watanabe, 1977), we can construct a probability space on which $Z(t)\ge X_{\mathit{ij}}(t)$ for all $t\in [0,T_{A_2^{\complement }})$, where $T_A:=inf\{t\in [0,\infty ):X(t)\in A\}$. (For the comparison theorem to hold there is a required growth condition on the diffusion coefficient. That this holds follows from the fact that $\sqrt{x(1-x)}$ is 1/2-Hölder continuous; see also Remark 3.9 on p298 of Ethier and Kurtz (1986).) Thus condition (iii) is implied by the a.s. divergence of

$$\begin{array}{c}\hfill {\int }_0^{T_{\epsilon }(Z)}\frac{1}{Z(t)}\text{d}t\end{array}$$

as $\epsilon \to 0$, which in turn follows from Lemma 4.4 of Barton et al. (2004), noting that $\vartheta <1$ guarantees the 0-boundary for Z is accessible. [Some errors in the proof of Lemma 4.4 are corrected by Taylor (2007).] Tracing our steps backwards, we have shown that condition (iii) holds provided $0<T_0(X_{\mathit{ij}})\le T<T_{A_2^{\complement }}$. Since

$$\begin{array}{c}\hfill \mathbb{P}(T_{A_1^{\complement }}>T,0<T_0(X_{\mathit{ij}})\le T<T_{A_2^{\complement }})>0,\end{array}$$

we conclude $\mathbb{P}(I_T=\infty )>0$. $\square$

The conditions given in Theorem 3 simplify our proof, but it seems feasible to substantially weaken them.

Testing for the presence of recombination

It is possible to use ${\widehat{\rho }}_{\text{MLE}}$ to design a likelihood ratio test for the null hypothesis that ${\rho }_0=0$. Using (17), the appropriate likelihood ratio statistic is, for $I_T<\infty$,

$$\begin{array}{c}\hfill \mathrm{\Lambda }:=2log\frac{\text{d}{\mathbb{P}}_{{\widehat{\rho }}_{\text{MLE}}}^{(T)}}{\text{d}{\mathbb{P}}_{{\rho }_0}^{(T)}}={\widehat{\rho }}_{\text{MLE}}^2{I}_T,I_T<\infty .\end{array}$$

Under standard assumptions, noting that ${\rho }_0=0$ lies on the boundary of $\mathrm{\Theta }$, this has an asymptotic distribution which is an equal mixture between a ${\chi }_1^2$ distribution and a ${\chi }_0^2$ distribution under the null hypothesis (Self and Liang 1987). Denote the CDF of this distribution by $F_m$. In particular, to construct a level 5% test one should reject ${\rho }_0=0$ if $\mathrm{\Lambda }$ exceeds the 95th percentile of $F_m$; equivalently if it exceeds the 90th percentile of a ${\chi }_1^2$ distribution.

To account for the possibility that $I_T=\infty$ we set

$$\begin{array}{c}\hfill \mathrm{\Lambda }:=\left\{\begin{array}{cc}+\infty ,\hfill & {\widehat{\rho }}_{\text{MLE}}>0,\hfill \\ 0,\hfill & {\widehat{\rho }}_{\text{MLE}}=0.\hfill \end{array}\right),I_T=\infty .\end{array}$$

The asymptotic null distribution for $\mathrm{\Lambda }$ is now less clear, though we note that continuing to assume $F_m$ would be conservative. We study the power of this test empirically in Sect. 5.

Multiple loci

It is possible to extend the above results to a general multi-locus model. The extension is straightforward and we omit many of the lengthy but straightforward calculations.

In a multi-locus model of $\ell$ loci with $K_j$ possible alleles at locus j, haplotypes are of the form $\mathit{i}=(i_1,\cdots ,i_{\ell })\in {\prod }_{j=1}^{\ell }\{1,\cdots ,K_j\}=:E$ in a diffusion on ${\mathrm{\Delta }}_{d-1}$ with $d={\prod }_{j=1}^{\ell }{K}_j$ coordinates. Stacking the haplotypes, the diffusion coefficient has entries $V_{\mathit{i}\mathit{k}}(x)=x_{\mathit{i}}({\delta }_{\mathit{i}\mathit{k}}-x_{\mathit{k}})$ as usual, and the unknown component of the drift is

$$\begin{array}{c}\hfill a_{\mathit{i}}(x;{\rho }_1,\cdots ,{\rho }_{\ell -1})=\sum _{j=1}^{\ell -1}{\rho }_j(x_{{\mathit{i}}_{\le j}}{x}_{{\mathit{i}}_{>j}}-x_{\mathit{i}}),\end{array}$$

where ${\rho }_j$ is the recombination rate between locus j and $j+1$, with each ${\rho }_j$ to be estimated; $x_{\mathit{i}}$ is the frequency of haplotype $\mathit{i}$; and we marginalize over a contiguous subset of loci by writing

$$\begin{array}{cccc}\hfill x_{{\mathit{i}}_{\le j}}& =\sum _{i_{j+1}=1}^{K_{j+1}}\cdots \sum _{i_{\ell }=1}^{K_{\ell }}{x}_{(i_1,\cdots ,i_{\ell })},\hfill & \hfill x_{{\mathit{i}}_{>j}}& =\sum _{i_1=1}^{K_1}\cdots \sum _{i_j=1}^{K_j}{x}_{(i_1,\cdots ,i_{\ell })}.\hfill \end{array}$$

From (13) and (14) the joint estimator for $({\rho }_1,\cdots ,{\rho }_{\ell -1})$ is $\widehat{\varrho }=I_T^{-1}Y$ where $I_T$ is $(\ell -1)\times (\ell -1)$ and Y is $(\ell -1)\times 1$ with elements

$$\begin{array}{cc}\hfill I_{\mathit{jk}}& ={\int }_0^T\sum _{\mathit{i}\in E}\frac{\left(X_{{\mathit{i}}_{\le j}},(t),X_{{\mathit{i}}_{>j}},(t),-,X_{\mathit{i}},(t)\right)\left(X_{{\mathit{i}}_{\le k}},(t),X_{{\mathit{i}}_{>k}},(t),-,X_{\mathit{i}},(t)\right)}{X_{\mathit{i}}(t)}\text{d}t\hfill \\ \hfill & ={\int }_0^T\sum _{\mathit{i}\in E}\frac{X_{{\mathit{i}}_{\le j}}(t)X_{{\mathit{i}}_{>j}}(t)X_{{\mathit{i}}_{\le k}}(t)X_{{\mathit{i}}_{>k}}(t)}{X_{\mathit{i}}(t)}\text{d}t-T,\hfill \\ \hfill Y_j& ={\int }_0^T\sum _{\mathit{i}\in E}\frac{X_{{\mathit{i}}_{\le j}}(t)X_{{\mathit{i}}_{>j}}(t)-X_{\mathit{i}}(t)}{X_{\mathit{i}}(t)}\text{d}{\tilde{X}}_{\mathit{i}}(t)={\int }_0^T\sum _{\mathit{i}\in E}\frac{X_{{\mathit{i}}_{\le j}}(t)X_{{\mathit{i}}_{>j}}(t)}{X_{\mathit{i}}(t)}\text{d}{\tilde{X}}_{\mathit{i}}(t).\hfill \end{array}$$

An alternative model is to set ${\rho }_j=\rho$ for each $j=1,\cdots ,\ell$ and to construct a single scalar estimator. Then the estimator is

$$\begin{array}{c}\hfill \widehat{\varrho }=\frac{{\displaystyle \sum _{j=1}^{\ell -1}{\int }_0^T\sum _{\mathit{i}\in E}\frac{X_{{\mathit{i}}_{\le j}}(t)X_{{\mathit{i}}_{>j}}(t)}{X_{\mathit{i}}(t)}\text{d}{\tilde{X}}_{\mathit{i}}(t)}}{{\displaystyle \sum _{j,k=1}^{\ell -1}{\int }_0^T\sum _{\mathit{i}\in E}\frac{X_{{\mathit{i}}_{\le j}}(t)X_{{\mathit{i}}_{>j}}(t)X_{{\mathit{i}}_{\le k}}(t)X_{{\mathit{i}}_{>k}}(t)}{X_{\mathit{i}}(t)}\text{d}t-{(\ell -1)}^{2}T}}.\end{array}$$

These estimators should be corrected as in (26).

The effects of natural selection

Methods for inference of recombination can be confounded by natural selection (Reed and Tishkoff 2006; O’Reilly et al. 2008; Peñalba and Wolf 2020). In this section we investigate the effect of selection on ${\widehat{\rho }}_{\text{MLE}}$, for simplicity returning to a two-locus model, though it should be straightforward to extend these results to general multi-locus models.

Confounding by selection

First consider the following: Suppose that, unknown to the investigator, the two loci are under selection—possibly a complicated type with epistatic interaction. What effect does this have on our estimator for $\rho$? More precisely, consider a model in which the component of the drift with parameters to be estimated is still $a_{\mathit{ij}}(x;\rho )=\rho (x_{i·}{x}_{·j}-x_{\mathit{ij}})$, but the ‘known’ component of the drift is now

$$\begin{array}{cc}\hfill c_{\mathit{ij}}(x)=& \frac{{\theta }_A}{2}\sum _{k=1}^K{x}_{\mathit{kj}}(P_{\mathit{ki}}^A-{\delta }_{\mathit{ik}})+\frac{{\theta }_B}{2}\sum _{l=1}^L{x}_{\mathit{il}}(P_{\mathit{lj}}^B-{\delta }_{\mathit{jl}})\hfill \\ \hfill & +\frac{x_{\mathit{ij}}}{2}\left[\sum _{k=1}^K,\sum _{l=1}^L,\left(s_{ij,kl},x_{\mathit{kl}},-,\sum _{m=1}^K,\sum _{n=1}^L,s_{kl,mn},x_{\mathit{kl}},x_{\mathit{mn}}\right)\right],\hfill \\ \hfill & i=1,\cdots ,K;j=1,\cdots ,L.\hfill \end{array}$$

This is a very general diploid, epistatic model of selection in which the selective advantage of an individual carrying haplotypes (i, j) and (k, l), relative to other individuals, is parametrised by $s_{ij,kl}$. We are interested in the role of selection as a confounder, whereby inference is carried out using the incorrect selection parameters in the dominating measure.

From Eq. (18), $c(·)$ has an effect on $\widehat{\rho }$ only through the term $\text{d}{\tilde{X}}_{\mathit{ij}}(t)=\text{d}{X}_{\mathit{ij}}(t)-c_{\mathit{ij}}(X(t))\text{d}t$. Therefore, in a model with selection we should adjust (18) by defining a new estimator

$$\begin{array}{cc}\hfill {\widehat{\rho }}_{\text{sel}}=& \widehat{\rho }-\frac{1}{I_T}{\int }_0^T\sum _{i=1}^K\sum _{j=1}^L\frac{X_{i·}(t)X_{·j}(t)}{X_{\mathit{ij}}(t)}\hfill \\ \hfill & \times \frac{X_{\mathit{ij}}(t)}{2}\left[\sum _{k=1}^K,\sum _{l=1}^L,\left(s_{ij,kl},X_{\mathit{kl}},(t),-,\sum _{m=1}^K,\sum _{n=1}^L,s_{kl,mn},X_{\mathit{kl}},(t),X_{\mathit{mn}},(t)\right)\right]\text{d}t\hfill \\ \hfill =& \widehat{\rho }-\frac{1}{I_T}{\int }_0^T\sum _{i=1}^K\sum _{j=1}^L(X_{i·}(t)X_{·j}(t)-X_{\mathit{ij}}(t))\sum _{k=1}^K\sum _{l=1}^L\frac{s_{ij,kl}}{2}{X}_{\mathit{kl}}(t)\text{d}t.\hfill \end{array}$$ 27

The last term, which is linear in the selection parameters, quantifies the error introduced by ignoring selection. However, it demonstrates a remarkable property in the absence of epistasis. In that case we can write $s_{ij,kl}=s_{\mathit{ik}}^A+s_{\mathit{jl}}^B$, where $s_{\mathit{ik}}^A$ is the selection parameter associated with genotype ik at locus A, and similarly for $s_{\mathit{jl}}^B$. Then Eq. (27) simplifies to

$$\begin{array}{c}\hfill {\widehat{\rho }}_{\text{sel}}=\widehat{\rho }.\end{array}$$

That is, we have an attractive robustness property: if an investigator uses the incorrect model for selection then the estimator $\widehat{\rho }$ is unaffected provided selection is not epistatic. The observed information is also the same. Noting that Theorem 2 continues to hold when $c(·)$ is altered, we conclude that ${\widehat{\rho }}_{\text{MLE}}$ defined in Sect. 3.2.3 is still the MLE for $\rho$ in the presence of (non-epistatic) selection.

General confounding

Returning to the general inference problem of Sect. 3.1, we can generalise the previous observations by asking: when does a contribution c(x) to the drift leave the estimator $\widehat{\phi }$ unchanged? From (5), its contribution to the estimator via $\text{d}\tilde{X}(t)$ will be zero if and only if $a{(X(t);\phi )}^{\top }V{(X(t))}^{-1}c(X(t))$ does not depend explicitly on $\phi$. When the drift is linear in the parameters as in (11), this requirement becomes

• ($\ast$) ${\phi }^{\top }{Z}^{\top }{V}^{-1}c$ does not depend on $\phi$.

If ($\ast$) holds we will say that the estimation problem for $\widehat{\phi }$ is robust to the contribution of c(x). In the context of the Wright–Fisher diffusion we have the following result.

Proposition 2

For a Wright–Fisher diffusion with drift coefficient $\mu (x;\phi )=c(x)+Z(x)\phi$ and diffusion coefficient $V=(V_{\mathit{ij}})$, $V_{\mathit{ij}}(x)=x_i({\delta }_{\mathit{ij}}-x_j)$, the estimator $\widehat{\phi }$ in (13) is robust to c(x) if and only if

$$\begin{array}{c}\hfill \sum _{i=1}^d\frac{1}{x_i}\frac{\partial a_i}{\partial {\phi }_k}(x)c_i(x)=0,\end{array}$$ 28

for each $k=1,\cdots ,r$.

Proof

We determine ($\ast$) for the first $d-1$ coordinates of the Wright–Fisher diffusion, with ${[V^{\ast }(x)]}^{-1}$ as in (8):

$$\begin{array}{cc}\hfill {\phi }^{\top }Z{(x)}^{\top }{[V^{\ast }(x)]}^{-1}c(x)=& \sum _{k=1}^r{\phi }_k\sum _{i=1}^{d-1}\left(\frac{1}{x_i},\frac{\partial a_i}{\partial {\phi }_k},(x),-,\frac{1}{x_d},\frac{\partial a_d}{\partial {\phi }_k},(x)\right)c_i(x)\hfill \\ \hfill =& \sum _{k=1}^r{\phi }_k\sum _{i=1}^d\frac{1}{x_i}\frac{\partial a_i}{\partial {\phi }_k}(x)c_i(x),\hfill \\ \hfill & c_d(x):=-\sum _{i=1}^{d-1}{c}_i(x).\hfill \end{array}$$

Since $\frac{\partial a_i}{\partial {\phi }_k}(x)$ and $c_i(x)$ do not depend on $\phi$, the above quantity is a linear combination of the ${\phi }_k$. It does not depend on any ${\phi }_k$ if and only if each of its coefficients is zero, i.e. (28) holds. $\square$

To give another example of the applicability of Proposition 2, consider reversing the roles of selection and recombination, so that we are interested in designing an estimator for (non-epistatic) selection at locus A in the confounding presence of recombination. For simplicity we focus on a genic selection model without mutation:

$$\begin{array}{cc}\hfill c_{\mathit{ij}}(x)& =\rho (x_{i·}{x}_{·j}-x_{\mathit{ij}}),\hfill \\ \hfill a_{\mathit{ij}}(x;s_1^A,\cdots ,s_K^A)& =\frac{x_{\mathit{ij}}}{2}\left(s_i^A,-,\sum _{k=1}^K,s_k^A,x_{k·}\right),i=1,\cdots ,K;j=1,\cdots ,L.\hfill \end{array}$$

In this model we find

$$\begin{array}{cc}\hfill \sum _{i=1}^K\sum _{j=1}^L\frac{1}{x_{\mathit{ij}}}\frac{\partial a_{\mathit{ij}}}{\partial s_k^A}(x)c_{\mathit{ij}}(x)& =\sum _{i=1}^K\sum _{j=1}^L\frac{1}{x_{\mathit{ij}}}\frac{x_{\mathit{ij}}}{2}({\delta }_{\mathit{ik}}-x_{k·})\rho (x_{i·}{x}_{·j}-x_{\mathit{ij}})=0,\hfill \end{array}$$

and (28) holds; by Proposition 2 the estimator ${\widehat{s}}^A$ for $(s_1^A,\cdots ,s_K^A)$ is robust to recombination, as we might hope.

Using Proposition 2 it is also possible to show that the following problems are robust:

• (i)Estimation of mutation at locus A when there is selection at locus B,
• (ii)Estimation of genic selection at locus A when there is mutation at locus B;

while the following problems are not robust:

• (iii)Estimation of recombination when there is mutation at either locus,
• (iv)Estimation of mutation at locus A when there is mutation at locus B,
• (v)Estimation of mutation at locus A when there is recombination,
• (vi)Estimation of mutation at locus A when there is selection at locus A,
• (vii)Estimation of genic selection at locus A when there is mutation at locus A;

similarly for problems interchanging the two loci. We omit the straightforward calculations. We caution that in the estimation problems above, the likelihood, and thus the estimator $\widehat{\phi }$, will be valid only up to time S as in (25). It is possible to have $I_T=\infty$ even in models without recombination (in particular, we expect two path measures with different mutation parameters to be mutually singular if certain allele frequencies reach 0).

Joint estimation of recombination and selection

In contrast to Sect. 4.1, one might recognise the possible existence of selection and be interested in constructing a joint estimator for recombination and selection. How does the marginal estimator for $\rho$ from this compare to those already developed? To illustrate the idea, we consider a simple genic selection model at locus A in which only allele k is under selection; that is, $s_i^A=0$ for $i\ne k$:

$$\begin{array}{cc}\hfill c_{\mathit{ij}}(x)& =\frac{{\theta }_A}{2}\sum _{k=1}^K{x}_{\mathit{kj}}(P_{\mathit{ki}}^A-{\delta }_{\mathit{ik}})+\frac{{\theta }_B}{2}\sum _{l=1}^L{x}_{\mathit{il}}(P_{\mathit{lj}}^B-{\delta }_{\mathit{jl}}),\hfill \\ \hfill a_{\mathit{ij}}(x;\rho ,s_k^A)& =\rho (x_{i·}{x}_{·j}-x_{\mathit{ij}})+\frac{x_{\mathit{ij}}}{2}\left({\delta }_{\mathit{ik}},s_k^A,-,s_k^A,x_{k·}\right),\hfill \\ \hfill & i=1,\cdots ,K;j=1,\cdots ,L.\hfill \end{array}$$

From (13) and (14) we find

$$\begin{array}{cc}\hfill I_T& =\left(\begin{array}{cc}{\displaystyle {\int }_0^T\sum _{i=1}^K\sum _{j=1}^L\frac{{(X_{\mathit{ij}}(t)-X_{i·}(t)X_{·j}(t))}^2}{X_{\mathit{ij}}(t)}\text{d}t}& 0\\ 0& {\displaystyle \frac{1}{4}{\int }_0^T{X}_{k·}(t)(1-X_{k·}(t))\text{d}t}\end{array}\right),\hfill \\ \hfill Y& =\left(\begin{array}{c}{\displaystyle {\int }_0^T\sum _{i=1}^K\sum _{j=1}^L\frac{X_{i·}(t)X_{·j}(t)}{X_{\mathit{ij}}(t)}\text{d}{\tilde{X}}_{\mathit{ij}}(t)}\\ {\displaystyle \frac{1}{2}{\int }_0^T\text{d}{\tilde{X}}_{k·}(t)}\end{array}\right),\hfill \end{array}$$

and so $\widehat{\phi }=I_T^{-1}Y$ simplifies to

$$\begin{array}{c}\hfill \widehat{\phi }=\left(\begin{array}{c}\widehat{\rho }\\ {\widehat{s}}_k^A\end{array}\right),\end{array}$$

where $\widehat{\rho }$ is the same estimator as we found in (18) and

$$\begin{array}{c}\hfill {\widehat{s}}_k^A=\frac{2(\tilde{X}(T)-\tilde{X}(0))}{{\int }_0^T{X}_{k·}(t)(1-X_{k·}(t))\text{d}t}.\end{array}$$

Setting ${\theta }_A={\theta }_B=0$ recovers the estimator for selection found by Watterson (1979), up to a choice of timescale. The key point is that the presence of selection as a ‘known-unknown’ leaves the estimator for $\rho$ unaffected, as is clear from the diagonal nature of $I_T$. In fact this might have been predicted even earlier: requiring $I_{\mathit{kl}}=0$ for the off-diagonal entry of an observed information matrix, corresponding to two parameters ${\phi }_k$, ${\phi }_l$, is essentially equivalent to the robustness condition ($\ast$). (To see this we identify the $c_i(x)$ term in ($\ast$) with $Z_{\mathit{il}}{\phi }_l$, so ${\phi }_l$ parametrises what would have been a confounder in ($\ast$).)

Simulation study

In this section we conduct an empirical study of the properties of ${\widehat{\rho }}_{\text{MLE}}$ by simulation. Although there has been recent progress in the development of algorithms for exact simulation of certain classes of Wright–Fisher diffusion (Jenkins and Spanò 2017; Griffiths et al. 2018; García-Pareja et al. 2021), these algorithms do not cover the non-reversible diffusions considered in this paper. Instead we resort to simple Euler–Maruyama simulation; that is, to simulate small increments of the diffusion over a fixed, small timestep $\mathrm{\Delta }t$ using the approximation

$$\begin{array}{cc}\hfill X(t+\mathrm{\Delta }t)& =X(t)+[c(X(t))+a(X(t);\phi )]\mathrm{\Delta }t+\sigma (X(t))[W(t+\mathrm{\Delta }t)-W(t)],\hfill \\ \hfill & X(0)=x(0),\hfill \end{array}$$

where W(t) is the $(d-1)$-dimensional Brownian motion in (23). Integrals involving the sample path of X can be approximated using Riemann sums constructed from the same set of gridpoints.

Because of its singularities at the boundaries of ${\mathrm{\Delta }}_{d-1}$, the Cholesky decomposition of V(x) of Sato (1976) is perhaps not the best choice of $\sigma (x)$ for the purposes of simulation, a point also noted in He et al. (2020). Instead we use a decomposition found for instance in Pal (2011, 2013):

$$\begin{array}{c}\hfill {\sigma }_{\mathit{i}\mathit{j}}(x)=\sqrt{x_{\mathit{i}}}({\delta }_{\mathit{i}\mathit{j}}-\sqrt{x_{\mathit{i}}{x}_{\mathit{j}}}),\mathit{i},\mathit{j}=1,\cdots ,d.\end{array}$$

This formulation has the advantage of being simple, symmetric, bounded in x, and vectorising easily via

$$\begin{array}{c}\hfill \sigma (x)=(I_d-\text{diag}(x)1_d)\text{diag}(\sqrt{x}),\end{array}$$

where $I_d$ is the identity matrix and $1_d$ is the $d\times d$ matrix of ones. A disadvantage is that it uses a d-dimensional Brownian motion, one dimension more than is necessary for simulation.

Using Euler–Maruyama simulation it is possible to obtain a realisation with $X_{\mathit{ij}}(t+\mathrm{\Delta }t)\le 0$ for some (i, j). If this occurs we set $X_{\mathit{ij}}(t+\mathrm{\Delta }t)=0$, renormalize $X(t+\mathrm{\Delta }t)$ so that $X(t+\mathrm{\Delta }t)\in {\mathrm{\Delta }}_{d-1}$, and set $I_{t+\mathrm{\Delta }t}=\infty$.

In the following we posit a two-locus, diallelic ($K=L=2$) model with symmetric mutation ($P^A=P^B=\left(\begin{array}{cc}1/2& 1/2\\ 1/2& 1/2\end{array}\right)$) and initial condition $X(0)=\left(\begin{array}{cc}2/5& 1/5\\ 1/5& 1/5\end{array}\right)$. The stepsize is set to $\mathrm{\Delta }t={10}^{-6}$ and paths are simulated up to a time $T=1$. We consider two sets of mutation parameters: (i) ${\theta }_A={\theta }_B=1$, and (ii) ${\theta }_A={\theta }_B=5$, in order to distinguish models in which the boundaries can or cannot be approached. We explore a variety of recombination parameters, $\rho \in \{0,0.1,1,2.5,5,10,25\}$, and to estimate distributional properties of the estimator we repeat each experiment 100 times.

As an illustration and a check that our implementation is accurate, examples of individual sample paths for $\rho =5$ are shown in Figs. 1 and 2, together with the accumulated information, $I_t$, and the evolving error, ${\widehat{\rho }}_{\text{MLE}}-\rho$, as functions of time. As is clear from the Figures, the error is stochastically converging towards 0, with erratic jumps towards 0 in regions where the information accumulates most rapidly. In the second example, in which ${\theta }_A={\theta }_B=1$, the trajectory for $X_{22}(t)$ wanders sufficiently closely to 0 that $I_t=\infty$ for some $t<T$, whereupon ${\widehat{\rho }}_{\text{MLE}}=\rho$. The distribution of ${\widehat{\rho }}_{\text{MLE}}$ across 100 experiments using these parameters are shown in Fig. 3, with results for further experiments summarised in Table 1.

Fig. 1.

Example trajectories in a two-locus model with two alleles at each locus, $\rho =5$, and ${\theta }_A={\theta }_B=5$. Also shown in the lower plots are the trajectories of ${\widehat{\rho }}_{\text{MLE}}-\rho$ and $I_t$ for this sample path

Fig. 2.

As Fig. 1 but with ${\theta }_A={\theta }_B=1$

Fig. 3.

Distribution of ${\widehat{\rho }}_{\text{MLE}}$ estimated from 100 replicates. Mutation parameters are ${\theta }_A={\theta }_B=5$ (left) and ${\theta }_A={\theta }_B=1$ (right). The true recombination parameter, shown by a red line, is $\rho =5$

Table 1.

Distributional summaries of ${\widehat{\rho }}_{\text{MLE}}$ for ${\theta }_A={\theta }_B=5$ (top) and ${\theta }_A={\theta }_B=1$ (bottom): mean, variance, 5th percentile, median, 95th percentile, frequency of zero error, and power to reject ${\rho }_0=0$ at level 5%. Each estimate is based on 100 independent replicates

$\rho$	$\mathbb{E}({\widehat{\rho }}_{\text{MLE}})$	$\text{Var}({\widehat{\rho }}_{\text{MLE}})$	$\mathbb{E}({\widehat{\rho }}_{\text{MLE},(5)})$	$\text{Median}({\widehat{\rho }}_{\text{MLE}})$	$\mathbb{E}({\widehat{\rho }}_{\text{MLE},(95)})$	$\mathbb{P}({\widehat{\rho }}_{\text{MLE}}=\rho )$	$\mathbb{P}(\mathrm{\Lambda }>F_m^{-1}(0.95))$
${\theta }_A={\theta }_B=5$
0	2.69	13.52	0.00	1.06	9.81	0.11	0.10
0.1	2.58	13.57	0.00	0.92	9.99	0.09	0.19
1	2.69	11.15	0.00	1.45	9.55	0.09	0.18
2.5	4.75	20.25	0.00	3.36	14.98	0.08	0.32
5	6.74	20.18	1.41	5.89	16.76	0.07	0.38
10	12.44	39.94	4.26	10.94	26.49	0.04	0.79
25	28.38	69.79	16.89	27.36	44.70	0.02	1.00
${\theta }_A={\theta }_B=1$
0	0.21	1.07	0.00	0.00	0.60	0.95	0.03
0.1	0.34	1.61	0.10	0.10	1.07	0.95	0.98
1	1.22	1.38	1.00	1.00	1.96	0.95	0.98
2.5	3.07	2.78	2.50	2.50	6.69	0.77	0.92
5	5.79	6.55	4.09	5.00	11.00	0.71	1.00
10	10.73	4.88	9.74	10.00	17.29	0.75	1.00
25	26.49	27.72	20.64	25.00	36.30	0.65	1.00

As is clear from Table 1, ${\widehat{\rho }}_{\text{MLE}}$ is slightly upwardly biased, with the relative bias greater for $\rho$ close to 0. Even with our assumption that the entire sample path is observed, for ${\theta }_A={\theta }_B=5$ the distribution of ${\widehat{\rho }}_{\text{MLE}}$ is rather flat: for example, when $\rho =2.5$ the central 90% of its mass is approximately contained in the interval [0, 15]. The power to reject the hypothesis ${\rho }_0=0$ at level $5\%$ is consequently poor for small $\rho$, exceeding 0.5 only for the rows in the table with $\rho \ge 10$.

For ${\theta }_A={\theta }_B=1$ the picture is very different, demonstrating the sensitivity of ${\widehat{\rho }}_{\text{MLE}}$ to the mutation parameters. We can see that here there is high probability that $\mathbb{P}({\widehat{\rho }}_{\text{MLE}}=\rho )$, providing very high power to reject ${\rho }_0=0$ even for small $\rho >0$.

Discussion

In this article we have derived an expression for the maximum likelihood estimator of the recombination rate, ${\widehat{\rho }}_{\text{MLE}}$, from a continuously observed diffusion model of haplotype frequencies. As well as recombination, the diffusion model can incorporate mutation, selection, and genetic drift. We have investigated the empirical properties of the estimator and its robustness to the presence of other processes. We have shown that, contrary to a typical estimator, it is possible to have ${\widehat{\rho }}_{\text{MLE}}=\rho$ with positive probability, and this event is intimately associated with the hitting of the boundary by the diffusion (Theorem 3). Although in that theorem we made some convenient assumptions about the trajectory of X(t), we expect it is possible to refine this result further; indeed we conjecture that $\{I_T=\infty \}$ is equal to the event that one haplotype frequency reaches 0 by time T. This would provide an easy way to check whether $\{I_T=\infty \}$ has occurred.

Although Theorems 2 and 3 are written in statistical language, in terms of estimators and information, we can gain some further intuition by phrasing them in a more fundamental way: it is known that the non-explosion condition (4) holds if and only if ${\mathbb{P}}_{\phi }^{(T)}\ll {\mathbb{P}}_{{\phi }_0}^{(T)}$ (Hobson and Rogers 1998). Thus, when estimating recombination (or mutation, but not selection), hitting a boundary of the diffusion leads to the loss of absolute continuity of one path measure with respect to another. The information $I_T$ provides a natural measure of ‘signal-to-noise’. From the point of view of a finite population, although one usually thinks of stochastic effects as being more important when an allele is very rare compared to when it is common, on the contrary what matters in the diffusion limit here is that the variance in offspring distribution (noise) goes to zero at the boundary while the mean detectable effect of recombination (signal) does not. (The qualitatively different behaviour at a boundary between a finite population model and its diffusion limit is also remarked on by Ewens (2004, p180).) This also explains the effects of the mutation rate on estimation of $\rho$ as observed in Sect. 5: higher mutation rates act to push haplotype frequencies toward the interior of the simplex, where the accumulation of information is slower. It also matches biological intuition: if mutation rates are very small, we can reject a null of no recombination by using the four-gamete test on just a sample at a single time point. As mutation rates increase, it is harder to tell apart recurrent mutation from recombination.

Because of the unusual behaviour of the estimator for $\rho$, we have refrained from providing a detailed description of its asymptotic properties such as local asymptotic normality (LAN). Using the estimator for the immigration rate of the CBI diffusion as a guide, it should be possible to show that inference for $\rho$ exhibits LAN along the sequence of random times

$$\begin{array}{c}\hfill T_n:=inf\{t\in [0,T]:I_t=n\};\end{array}$$

see Overbeck (1998, §3.4). However, in the case $I_T<\infty$ for each T, finding the asymptotic behaviour of ${\widehat{\rho }}_{\text{MLE}}$ is more involved since we do not know the stationary distribution of X.

Finally, we observe that fundamental quantities appearing throughout this work are

$$\begin{array}{c}\hfill \sum _{i=1}^K\sum _{j=1}^L\frac{{(X_{\mathit{ij}}-X_{i·}{X}_{·j})}^2}{X_{\mathit{ij}}},\text{and}\sum _{i=1}^K\sum _{j=1}^L\frac{X_{\mathit{ij}}-X_{i·}{X}_{·j}}{X_{\mathit{ij}}}.\end{array}$$ 29

The role of $X_{\mathit{ij}}$ in the denominators has been to upweight the importance of those parts of the trajectories where the frequency of haplotype (i, j) is small, since these regions are more informative for $\rho$. In one sense this is unsatisfactory since the diffusion model is often regarded as an approximation of a discrete population of size N, and behaviour near the boundaries is inappropriate when true frequencies can only be a multiple of 1/N. One might prefer to replace the estimator $\widehat{\rho }$, which integrates each $X_{\mathit{ij}}(t)$ over [0, T], with one that integrates $X_{\mathit{ij}}(t)$ only over some sub-region

$$\begin{array}{c}\hfill \{t\in [0,T]:\epsilon \le X_{\mathit{ij}}(t)\le 1-\epsilon \}.\end{array}$$

Even under this restriction, the quantities in (29) tell us to focus our attention on those regions where the haplotype frequency is far from 1/2. The normalizations inherent in (29) seem to offer ‘natural’ new normalizations for the coefficient of linkage disequilibrium, which do not correspond to the usual normalizations found in $r^2$ and $D^{\prime }$, for example (Sved and Hill 2018). Exploring the properties of these new summaries of LD will be the subject of future work.

A Appendix

In this section we study the deterministic mutation-recombination equation

$$\begin{array}{cc}\hfill & \frac{\text{d}{x}_{\mathit{ij}}}{\text{d}t}=\rho (x_{i·}{x}_{·j}-x_{\mathit{ij}})+\frac{{\theta }_A}{2}(x_{·j}{P}_i^A-x_{\mathit{ij}})+\frac{{\theta }_B}{2}(x_{i·}{P}_j^B-x_{\mathit{ij}}),\hfill \\ \hfill & i=1,\cdots ,K;j=1,\cdots ,L.\hfill \end{array}$$ 30

Summing (30) over j yields:

$$\begin{array}{c}\hfill \frac{\text{d}{x}_{i·}}{\text{d}t}=\frac{{\theta }_A}{2}(P_i^A-x_{i·}),i=1,\cdots ,K,\end{array}$$

whose solution is

$$\begin{array}{c}\hfill x_{i·}(t)=P_i^A+(x_{i·}(0)-P_i^A)e^{-\frac{{\theta }_A}{2}t},i=1,\cdots ,K.\end{array}$$

Similarly,

$$\begin{array}{c}\hfill x_{·j}(t)=P_j^B+(x_{·j}(0)-P_j^B)e^{-\frac{{\theta }_B}{2}t},j=1,\cdots ,L.\end{array}$$

Therefore, (30) can be written

$$\begin{array}{c}\hfill \frac{\text{d}{x}_{\mathit{ij}}}{\text{d}t}+\left(\rho ,+,\frac{\theta }{2}\right)x_{\mathit{ij}}(t)=\rho x_{i·}(t)x_{·j}(t)+\frac{{\theta }_A}{2}{x}_{·j}(t)P_i^A+\frac{{\theta }_B}{2}{x}_{i·}(t)P_j^B=:F(t),\end{array}$$

with F(t) known and $\theta :={\theta }_A+{\theta }_B$. This can be solved via an integrating factor; the solution is

$$\begin{array}{cc}\hfill x_{\mathit{ij}}(t)=& e^{-\left(\rho ,+,\frac{\theta }{2}\right)t}\left[x_{\mathit{ij}},(0),+,{\int }_0^t,F,(s),e^{\left(\rho ,+,\frac{\theta }{2}\right)s},,,\text{d},s\right]\hfill \\ \hfill =& \left(1,-,e^{-\left(\rho ,+,\frac{\theta }{2}\right)t}\right)P_i^A{P}_j^B\hfill \\ \hfill & +(x_{·j}(0)-P_j^B)P_i^A\left(e^{-\frac{{\theta }_B}{2}t},-,e^{-\left(\rho ,+,\frac{\theta }{2}\right)t}\right)\hfill \\ \hfill & +(x_{i·}(0)-P_i^A)P_j^B\left(e^{-\frac{{\theta }_A}{2}t},-,e^{-\left(\rho ,+,\frac{\theta }{2}\right)t}\right)\hfill \\ \hfill & +(x_{i·}(0)-P_i^A)(x_{·j}(0)-P_j^B)\left(e^{-\frac{\theta }{2}t},-,e^{-\left(\rho ,+,\frac{\theta }{2}\right)t}\right)+x_{\mathit{ij}}(0)e^{-\left(\rho ,+,\frac{\theta }{2}\right)t}.\hfill \end{array}$$

Substituting the solutions for $x_{i·}(t)$, $x_{·j}(t)$, and $x_{\mathit{ij}}(t)$ into (19) gives an expression for the observed information. After some simplification we arrive at

$$\begin{array}{cc}\hfill & I_T\hfill \\ \hfill & ={\int }_0^T\sum _{i=1}^K\sum _{j=1}^L\frac{D_{\mathit{ij}}{(0)}^2{e}^{-2\left(\rho ,+,\frac{\theta }{2}\right)t}}{D_{\mathit{ij}}(0)e^{-\left(\rho ,+,\frac{\theta }{2}\right)t}+[P_i^A+(x_{i·}(0)-P_i^A)e^{-\frac{{\theta }_A}{2}t}][P_j^B+(x_{·j}(0)-P_j^B)e^{-\frac{{\theta }_B}{2}t}]}\text{d}t.\hfill \end{array}$$

where $D_{\mathit{ij}}(0)=x_{\mathit{ij}}(0)-x_{i·}(0)x_{·j}(0)$. Since the integrand of $I_T$ decays exponentially in t, clearly $I_{\infty }<\infty$ as before.

Data availability

Data and code used to conduct the simulation study is available at https://github.com/Paul-Jenkins/GriffithsJenkins2023.

Declarations

Conflict of interest

The authors have no competing interests to declare.