Bayesian selection of misspecified models is overconfident and may cause spurious posterior probabilities for phylogenetic trees

Significance

The Bayesian method is widely used to estimate species phylogenies using molecular sequence data. While it has long been noted to produce spuriously high posterior probabilities for trees or clades, the precise reasons for this overconfidence are unknown. Here we characterize the behavior of Bayesian model selection when the compared models are misspecified and demonstrate that when the models are nearly equally wrong, the method exhibits unpleasant polarized behaviors, supporting one model with high confidence while rejecting others. This provides an explanation for the empirical observation of spuriously high posterior probabilities in molecular phylogenetics.

Abstract

The Bayesian method is noted to produce spuriously high posterior probabilities for phylogenetic trees in analysis of large datasets, but the precise reasons for this overconfidence are unknown. In general, the performance of Bayesian selection of misspecified models is poorly understood, even though this is of great scientific interest since models are never true in real data analysis. Here we characterize the asymptotic behavior of Bayesian model selection and show that when the competing models are equally wrong, Bayesian model selection exhibits surprising and polarized behaviors in large datasets, supporting one model with full force while rejecting the others. If one model is slightly less wrong than the other, the less wrong model will eventually win when the amount of data increases, but the method may become overconfident before it becomes reliable. We suggest that this extreme behavior may be a major factor for the spuriously high posterior probabilities for evolutionary trees. The philosophical implications of our results to the application of Bayesian model selection to evaluate opposing scientific hypotheses are yet to be explored, as are the behaviors of non-Bayesian methods in similar situations.

The Bayesian method was introduced into molecular phylogenetics in the 1990s (1–3) and has since become one of the most popular methods for statistical analysis in the field, in particular, for estimation of species phylogenies (4–7). It has been noted that the method often produces very high posterior probabilities for trees or clades (nodes in the tree). In the first-ever Bayesian phylogenetic calculation, a biologically reasonable tree for five species of great apes was produced from a dataset of 11 mitochondrial tRNA genes (739 sites), but the posterior probability for that tree, at 0.9999, was uncomfortably high (1). In the past two decades, the Bayesian method has been used to analyze thousands of datasets, with the computation made possible through Markov chain Monte Carlo (MCMC) (4, 5). It has become a common practice to report posterior clade probabilities only if they are $<100\%$ (because most estimates are 100%). In some cases the high posterior probabilities are decidedly spurious. For example, conflicting trees may be inferred from the same data under different evolutionary models. Different trees may be inferred depending on the species sampled in the dataset (8) or on whether protein sequences or the encoding DNA sequences are analyzed (9). In such cases, the different trees cannot all be correct, even if the true tree is unknown. The concern is not so much that the inferred species relationships may be wrong but that they are supported by extremely high posterior probabilities.

In the star-tree paradox, large datasets were simulated using the star tree and then analyzed to calculate the posterior probabilities for the three binary trees (Fig. 1). Most biologists would want the posterior probabilities for the binary trees to converge to $(\frac{1}{3},\frac{1}{3},\frac{1}{3})$ when the amount of data increases (10–12). Instead they fluctuate among datasets according to a statistical distribution, sometimes producing strong support for a binary tree even though the data do not contain any information either for or against any binary tree (13–15).

Fig. 1.

(A) The three binary rooted trees for three species $T_1,T_2$, and $T_3$ and the star tree $T_0$. (B) The three binary unrooted trees for four species $T_1,T_2$, and $T_3$ and the star tree $T_0$. The branch length parameters are shown next to the branches, measured by the expected number of nucleotide changes per site. In the star-tree simulations, the star tree is used to generate data, which are analyzed to calculate the posterior probabilities for the three binary trees, with the star tree excluded.

Bayesian model selection is known to be consistent (16). When the data size $n\to \mathrm{\infty }$, the true model “dominates,” with its posterior probability approaching 1. If several models are equally right, the model with fewer parameters dominates. However, this theory applies only if the true model is included in the comparison. Given that a model is a simplified representation of the physical world, the more common situation in real data analysis should be the comparison of models that are all wrong. Not many theoretical results appear to exist concerning Bayesian comparison of misspecified models (17).

Here we study the asymptotic behavior of Bayesian model selection in a general setting where multiple misspecified models are compared. We are interested in how the posterior probabilities for models behave when the data size increases. Do the dynamics depend on whether there are any free parameters in the models? If one model is less wrong than another (in a certain sense appropriately defined), will the less wrong model always win? We present the proofs and mathematical analyses in General Theory for Equally Wrong Models with No Free Parameters (d=0) and General Theory for Equally Right or Equally Wrong Models with Free Parameters (d>0). In the main text, we summarize our results and illustrate them using three canonical simple problems. Our analysis suggests that the problem exposed by the star-tree paradox is actually far more troubling than discussed previously (11–15).

Results

Problem Description.

We consider independent and identically distributed (i.i.d.) models only. The data $x=\{x_1,\mathrm{\dots },x_n\}$ are an i.i.d. sample from the true model $g(\cdot )$. We consider two models as the case for more models is obvious. Model $H_k$ has density $f_k(x|{\theta }_k)$, with $d_k$ free parameters (${\theta }_k$), $k=\mathrm{1,2}$. We are in particular interested in models of the same dimension, with $d_1=d_2=d$. In the Bayesian analysis, we assign a uniform prior for the two models (${\pi }_1={\pi }_2=\frac{1}{2})$ and also a prior for the parameters within each model $H_k$: $f_k({\theta }_k)$. The posterior model probabilities, $P_k=\mathbb{P}(H_k|x)$, are then proportional to the marginal likelihoods: $M_k=f_k(x)=\int f_k({\theta }_k)f_k(x|{\theta }_k)\text{d}{\theta }_k$; that is, $P_1/P_2=({\pi }_1{M}_1)/({\pi }_2{M}_2)=M_1/M_2$. We are interested in the asymptotic behavior of $P_1$ in large datasets (as $n\to \mathrm{\infty }$).

The dynamics depend on how well the models fit the data. Let $\widehat{{\theta }_k}$ be the maximum-likelihood estimate (MLE) of ${\theta }_k$ under model $H_k$ from dataset $x$. Let ${\theta }_k^{\ast }$ be the limiting value of $\widehat{{\theta }_k}$ when the data size $n\to \mathrm{\infty }$. In other words ${\theta }_k^{\ast }$ minimizes the Kullback–Leibler (K-L) divergence from model $H_k$ to the true model,

$$D_k=D_{\text{KL}}(g,f_k)=\int g(x)\log g(x)\text{d}x-\int g(x)\log f_k(x|{\theta }_k^{\ast })\text{d}x,$$ [1]

and is known as the best-fitting or pseudotrue parameter value under the model (18). $D_k$ (calculated at ${\theta }_k^{\ast }$) measures the distance from $H_k$ to the true model, with $D_k\ge 0$. We say a model is “right” if it encompasses the true model, with $D=0$, and “wrong” if $D>0$. Model 1 is less wrong than model 2 if $D_1<D_2$. Both models are “equally right” if $D_1=D_2=0$ and “equally wrong” if $D_1=D_2>0$.

Characterization of Bayesian Model Selection.

The asymptotic behavior of $P_1=\mathbb{P}(H_1|x)$ when $n\to \mathrm{\infty }$ is analyzed in SI Text and summarized in Fig. 2. We identify three types of asymptotic behaviors: type 1 (“balanced”), type 2 (“volatile”), and type 3 (“polarized”), as defined below. We also refer to three types of inference problems that give rise to those behaviors.

Fig. 2.

Classification of Bayesian model-selection problems involving two equally right or equally wrong models, each with $d$ free parameters. Solid circles represent the true model, while the lines represent the parameter space of the compared models, with the open circles showing the best-fitting parameter value $({\theta }^{\ast })$. The two models are equally right (with $D_1=D_2=0$) if the solid and open circles coincide and equally wrong (with $D_1=D_2>0$) if they are separate. The models are “indistinct” if the two open cycles coincide (as in A and B) and are “distinct” if they are separate (as in C). The green, orange, and red boxes indicate the three different asymptotic behaviors of Bayesian model selection when the data size $n\to \mathrm{\infty }$.

Type 1 (balanced) is for the posterior model probability $P_1$ to converge (as $n\to \mathrm{\infty }$) to a single reasonable value that is different from 0 to 1, such as $\frac{1}{2}$. In other words, in essentially every large dataset, $P_1\approx \frac{1}{2}$. This behavior occurs when the two models are essentially identical. Examples include comparison of two identical models with no parameters, such as $H_1:p=0.5$ and $H_2:p=0.5$ irrespective of the true $p$ in a coin-tossing experiment (Fig. 2, cases $A_1$ and $A_2$), and overlapping models where the best-fitting parameter values lie in the region of overlap (Fig. 2, $A_3$ and $A_4$). Whether the two models are both right ($A_1$ and $A_3$) or both wrong ($A_2$ and $A_4$) does not affect the dynamics. The case of overlapping models is interesting. If the truth is $p=\frac{1}{2}$ while the two compared models are $H_1:0.4<p<0.6$ and $H_2:0<p<1$, and if we assign a uniform prior on $p$ in each model, then as $n\to \mathrm{\infty }$, $P_1\to \frac{1}{1+0.2}=\frac{5}{6}$, which appears more reasonable than $\frac{1}{2}$ as it favors the more-informative model $H_1$. At any rate, the comparison of identical or overlapping models is unusual for testing scientific hypotheses. This type of problem is not considered further.

Type 2 (volatile) is for $P_1$ to converge to a nondegenerate statistical distribution, such as $U(\mathrm{0,1})$. In other words, if we analyze different large datasets, all generated from the same true model, to compare two equally right or equally wrong models, $P_1$ varies among datasets according to a nondegenerate distribution. This behavior occurs when the two compared models become unidentifiable as the data size $n\to \mathrm{\infty }$. There are two scenarios. In the first one, both models are right, with $D_1=D_2=0$ (Fig. 2, $B_1$ and $B_2$). In the second one both models are equally wrong (with $D_1=D_2>0$) but indistinct (Fig. 2, $B_3$ and $B_4$). We say that two models are indistinct if and only if they, each at the best-fitting parameter values, are unidentifiable, with $f_1(x|{\theta }_1^{\ast })=f_2(x|{\theta }_2^{\ast })$ for essentially all $x$. In other words, in infinite data, the two models make essentially the same predictions about the data and are unidentifiable. In both scenarios of equally right and equally wrong models, $P_1$ varies among datasets according to a nondegenerate distribution.

Type 3 (polarized) is for $P_1$ to have a degenerate two-point distribution, at values 0 and 1. If we analyze large datasets to compare two models, we favor model 1 with total confidence in some datasets and model 2 with total confidence in others. This behavior is observed when the two models are equally wrong and also distinct.

It is remarkable that the asymptotic behavior is determined by whether or not the compared models are distinct and not by whether they are both right or both wrong or by whether the compared models have unknown parameters. For example, cases $B_1$ (two right models) and $B_3$ (two equally wrong models) in Fig. 2 show the same volatile behavior, while cases $C_1$ (no free parameters) and $C_2$ (with free parameters) show the same polarized behavior.

Problem 1. Fair-Coin Paradox (Equally Wrong Models with No Free Parameter).

Consider a coin-tossing experiment in which the coin is fair with the probability of heads $p=\frac{1}{2}$. We use the data of $x$ heads in $n$ tosses to compare two models: $H_1:p=0.4$ (tail bias) and $H_2:p=0.6$ (head bias). The two models are equally wrong. We assign a uniform prior for the two models ($\frac{1}{2}$ each) and calculate the posterior model probability $P_1=\mathbb{P}(H_1|x)$. This is a type-3 problem (Fig. 2, $C_1$).

As the models involve no free parameters, the likelihood $(L)$ and marginal likelihood $(M)$ are the same, given by the binomial probability for data $x$. The posterior odds are the likelihood-ratio

$$\frac{P_1}{1-P_1}=\frac{M_1}{M_2}=\frac{{0.4}^x\cdot {0.6}^{n-x}}{{0.6}^x\cdot {0.4}^{n-x}}={\left(\frac{0.4}{0.6}\right)}^{2x-n}.$$ [2]

When $n$ is large, $P_1$ tends to be extreme (close to 0 or 1). Indeed, $\alpha <P_1<1-\alpha$ if and only if $|2x-n|<B=\frac{\log \{\alpha /(1-\alpha )\}}{\log \{0.4/0.6\}}$. If $n$ is large, $2x-n$ is ∼$\mathbb{N}(0,n)$, so that

$$\mathbb{P}\{|2x-n|<B\}\approx 1-2\mathrm{\Phi }(-\frac{B}{\sqrt{n}})\approx \frac{2B}{\sqrt{2\pi n}},$$ [3]

where $\mathrm{\Phi }$ is the cumulative distribution function (CDF) for $\mathbb{N}(\mathrm{0,1})$. If $\alpha =1\%$, we have $B$ = 11.33296, so that only 11 data outcomes will give $P_1$ in the range (0.01, 0.99), with $x-\frac{n}{2}$ being $-5,-4,\mathrm{\cdots },5$. For $n={10}^3,{10}^4,{10}^5,{10}^6$, we have $\mathbb{P}\{0.01<P_1<0.99\}$ = 0.280, 0.090, 0.0286, and 0.0090 using the normal approximation of Eq. 3 or 0.272, 0.0876, 0.0277, and 0.0088 exactly by the binomial distribution. Thus, in large datasets, moderate posterior probabilities will be rare, and either $H_1$ or $H_2$ will be favored with posterior $>0.99$. When $n\to \mathrm{\infty }$, $P_1$ has a degenerate two-point distribution, taking the values 0 and 1, each half of the times. This is the type-3 polarized behavior. Note that there is no information either for or against either model in the data. Fig. 3 A, i shows the distribution of $P_1$ for $n={10}^3$.

Fig. 3.

The distribution of posterior model probability $P_1=\mathbb{P}\{H_1|x\}$ in three inference problems. (A) Problem 1 (fair-coin paradox) is for a coin-tossing experiment, where the true model is $p=0.5$ (a fair coin), and the compared models are (A, i) $H_1:p=0.4$ and $H_2:p=0.6$ so that the two models are equally wrong and (A, ii) $H_1:p=0.42$ and $H_2:p=0.6$ so that $H_1$ is less wrong than $H_2$. The data size (the number of coin tosses) is ${10}^3$. (B) Problem 2 (fair-balance paradox) is for a normal-distribution example in which the true model is $\mathbb{N}(\mathrm{0,1})$, and the two compared models are $H_1:\mathbb{N}(\mu ,1/\tau ),\mu <0$ and $H_2:\mathbb{N}(\mu ,1/\tau ),\mu >0$, with variance $1/\tau$ given. The two models are equally right when $\tau =1$ and equally wrong but indistinct when $\tau =1/9$ or 9. The data size is $n={10}^3$. The plots for $n=100$ or $\mathrm{\infty }$ are nearly the same. (C) Problem 3 (fair-balance paradox) is for a normal-distribution example in which the true model is $\mathbb{N}(\mathrm{0,1})$, and the two compared models are $H_1:\mathbb{N}(\mu ,1/{\tau }_1)$ and $H_2:\mathbb{N}(\mu ,1/{\tau }_2)$, with (C, i) ${\tau }_1=0.25$ and ${\tau }_2=2.58666$, so that the two models are equally wrong, and (C, ii) ${\tau }_1=0.3$ and ${\tau }_2=2.58666$, so that $H_1$ is less wrong than $H_2$. The prior is $\mu \sim \mathbb{N}(0,1/\xi )$ under each model, with $\xi =1$. The data size is $n=100$. All densities are estimated by simulating ${10}^5$ samples for $P_1$.

Fig. 3 A, ii shows the comparison of $H_1:p=0.42$ against $H_2:p=0.6$ when the truth is $p=0.5$. Here $H_1$ is less wrong and will eventually dominate. However, in large and finite datasets, the more wrong model $H_2$ can often receive high support. For example, for $n={10}^3$, nonextreme posterior probabilities in the range $0.01<P_1<0.99$ occur for only 13 data outcomes, with $x$ being 504–516, and in $14.8\%$ of datasets, $x$ is greater than those values so that $P_2>0.99$. Indeed over the whole range $36\le n\le 11,611$, the more wrong model $H_2$ is strongly favored too often, with $\mathbb{P}(P_2>0.99)>0.01$. The method becomes overconfident before it becomes reliable. It may be noted that such strong support for the more wrong model occurs only when the two models are opposing each other. It does not occur if both models are wrong in the same direction: In the comparison of $H_1:p=0.4$ and $H_2:p=0.42$ when the truth is $p=0.5$, the less wrong model $H_2$ dominates in the posterior.

Problem 2. Fair-Balance Paradox (Equally Right Models or Equally Wrong and Indistinct Models).

The true model is $\mathbb{N}(\mathrm{0,1})$, and we compare two models $H_1:\mathbb{N}(\mu ,1/\tau )$, $\mu <0$ and $H_2:\mathbb{N}(\mu ,1/\tau ),\mu >0$, with $\tau$ given. The data may represent measurement errors observed on a fair balance while the models claim that the balance has an unknown negative or positive bias. The best-fitting parameter value (the MLE when the data size $n\to \mathrm{\infty }$) is ${\mu }^{\ast }=0$ in each model, when the two models become identical (indistinct). Thus, the two models are equally right if $\tau =1$ (Fig. 2, $B_2$), and are equally wrong if $\tau =1/9$ or 9 (Fig. 2, $B_4$).

We assign a uniform prior on the two models ($\frac{1}{2}$ each), and $\mu \sim \mathbb{N}(0,1/\xi )$ with $\xi$ fixed, truncated to the appropriate range under each model. The data $(x)$, an i.i.d. sample from $\mathbb{N}(\mathrm{0,1})$, can be summarized as the sample mean $\overline{x}$. It can be shown that the posterior model probability $P_1=\mathbb{P}\{H_1|x\}$ varies among datasets according to the density

$$f(P_1)=\frac{\sqrt{\tau +\xi /n}}{\tau }\cdot \exp \left\{\frac{{[{\mathrm{\Phi }}^{-1}(P_1)]}^2}{2}\left[1-\frac{1}{\tau }-\frac{\xi }{n{\tau }^2}\right]\right\},$$ [4]

where ${\mathrm{\Phi }}^{-1}$ is the inverse CDF for $\mathbb{N}(\mathrm{0,1})$ (Analysis of Problem 2 (Two Equally Right Models or Equally Wrong but Indistinct Models)).

Fig. 3B shows the density of $P_1$ for different values of precision $(\tau )$, with $n={10}^3$. If $\tau =1$, the two models are equally right, and $f(P_1)\to 1$ when $n\to \mathrm{\infty }$ so that $P_1$ behaves like a $𝕌(\mathrm{0,1})$ random number (11, 12). If $\tau <1$, the assumed variance $(1/\tau )$ is larger than the true variance, so that the distribution has a mode at $\frac{1}{2}$. If $\tau >1$, the assumed variance is too small, and $P_1$ has a U-shaped distribution. If one overstates the precision of the experiment, one tends to overinterpret the data and generate extreme posterior model probabilities. In all three cases ($\tau <1,=\mathrm{\hspace{0.17em}1},>1$), $P_1$ has a nondegenerate distribution.

Problem 3. Fair-Balance Paradox (Equally Wrong and Distinct Models).

The true model is $\mathbb{N}(\mathrm{0,1})$, and the two compared models are $H_1:\mathbb{N}(\mu ,1/{\tau }_1)$ and $H_2:\mathbb{N}(\mu ,1/{\tau }_2)$, with ${\tau }_1<1<{\tau }_2$ given, while $\mu$ is a free parameter in each model. The best-fitting parameter value is ${\mu }^{\ast }=0$ in each model, irrespective of the value of $\tau$ assumed. Both models are wrong because of the misspecified variance: $H_1$ is overdispersed while $H_2$ is underdispersed. They are equally wrong, in the sense that $D_1=D_2$ in Eq. 1, if

$$\log \frac{{\tau }_1}{{\tau }_2}={\tau }_1-{\tau }_2$$ [5]

(Analysis of Problem 3 (Two Equally Wrong and Distinct Models, Gaussian with Incorrect Variances)). This is a type-3 problem (Fig. 2, $C_2$). We assign a uniform prior over the models ($\frac{1}{2}$ each), and $\mu \sim \mathbb{N}(0,1/\xi )$, with $\xi$ given, within each model. The dataset, an i.i.d. sample of size $n$ from $\mathbb{N}(\mathrm{0,1})$, can be summarized as the sample mean $\overline{x}$ and sample variance $s^2=\frac{1}{n}{\sum }_i{(x_i-\overline{x})}^2$. The posterior odds are given in Eq. S15 in Analysis of Problem 3 (Two Equally Wrong and Distinct Models, Gaussian with Incorrect Variances).

We use ${\tau }_1=0.25$ and ${\tau }_2=2.58666$, so that Eq. 5 holds and the two models are equally wrong, to generate independent variables $\overline{x}\sim \mathbb{N}(0,1/n)$ and $n{s}^2\sim {\chi }_{n-2}^2$ and to calculate $P_1$. Fig. 3 C, i shows the estimated density of $P_1$ for $n=100$, with $\xi =1$. When $n\to \mathrm{\infty }$, $P_1$ degenerates into a two-point distribution at 0 and 1, each with probability $\frac{1}{2}$. These are the same dynamics as in problem 1 (Fig. 3 A, i), even though in problem 1 the models do not involve any unknown parameters while here they do.

Fig. 3 C, ii shows the density of $P_1$ when ${\tau }_1=0.3$ (which is closer to the true $\tau =1$ than is 0.25), so that $H_1$ is less wrong than $H_2$ (with $D_1<D_2$). In this case when $n\to \mathrm{\infty }$, $P_1\to 1$. However, in large but finite datasets, $P_2$ for the more wrong model $H_2$ can be large in too many datasets: For example, with $n=100$, $\mathbb{P}\{P_2>0.99\}=0.0504$: in $5.04\%$ of datasets, the more wrong model $H_2$ has posterior higher than $99\%$.

Star-Tree Paradox and Bayesian Phylogenetics.

In Bayesian phylogenetics (1, 2), each model has two components: the phylogenetic tree describing the relationships among the species and the evolutionary model describing sequence evolution along the branches on the tree (19). Each tree $T_k$ has a set of time or branch-length parameters $(t_k)$, which measure the amount of evolutionary changes along the branches. The evolutionary model may also involve unknown parameters $(\psi )$. The tree and the evolutionary model together specify the likelihood (20), with $\theta =\{t,\psi \}$ being the unknown parameters. One of the trees is true, and all other trees are wrong, while the evolutionary model may be misspecified. The main objective is to infer the true tree. The data consist of an alignment of sequences from the modern species and have a multinomial distribution in which the categories correspond to the possible site patterns (configurations of nucleotides observed in the modern species) while the data size is the number of sites or alignment columns (21).

Here we consider three simple cases involving three or four species (Fig. 1). We use the general theory described above to predict the asymptotic behavior of posterior probabilities for trees and use computer simulation to verify the predictions.

Case A (Fig. 4 A and A´) involves equally right models. We use the rooted star tree $T_0$ for three species with $t=0.2$ (Fig. 1A) to generate datasets to compare the three binary trees. The Jukes–Cantor (JC) substitution model (22) is used both to generate and to analyze the data, which assumes that the rate of change between any two nucleotides is the same. The molecular clock (rate constancy over time) is assumed as well, so that the parameters in each binary tree are the two ages of nodes ($t_0,t_1$), measured by the expected number of nucleotide changes per site.

Fig. 4.

The distribution of posterior probabilities $(P_1,P_2,P_3)$ for the three binary trees $T_1,T_2$, and $T_3$ of Fig. 1, when datasets (sequence alignments of $n={10}^3$ or ${10}^5$ sites) are simulated using the star tree $T_0$ and analyzed to compare the three binary trees. In A and A´, the true tree is the star tree $T_0$ for three species of Fig. 1A, with $t=0.2$. Both the simulation and analysis models are JC, and the three binary trees are equally right models. In B and B´, the true tree is the star tree $T_0$ for three species of Fig. 1A, with $t=0.2$. The simulation model is JC+$\mathrm{\Gamma }$ (with $\alpha =1$), and the analysis model is JC. The three binary trees represent equally wrong and indistinct models. In C and C´, the true tree is the star tree $T_0$ for four species of Fig. 1B, with $t_1=t_2=t_3=t_4=0.2$. The simulation model is JC+$\mathrm{\Gamma }$ $(\alpha =1)$ and the analysis model is JC. The three binary trees represent equally wrong and distinct models. The three corners in the plots correspond to points (1, 0, 0), (0, 1, 0), and (0, 0, 1), while the center is $(\frac{1}{3},\frac{1}{3},\frac{1}{3})$.

The best-fitting parameter values are $t_0^{\ast }=0$ and $t_1^{\ast }=0.2$ for each of the three binary trees, in which case each binary tree converges to the true star tree. We assign uniform prior probabilities for the binary trees ($\frac{1}{3}$ each) and an exponential prior on branch lengths on each tree. According to our characterization, this is a type-2 problem of comparing equally right models (Fig. 2, $B_2$), so the posterior probabilities should have a nondegenerate distribution. This case was considered in previous studies (12, 14, 15), which generated numerically the limiting distribution of the posterior probabilities for the binary trees $(P_1,P_2,P_3)$ when $n\to \mathrm{\infty }$ and pointed out that they do not converge to $(\frac{1}{3},\frac{1}{3},\frac{1}{3})$ (11–13).

Case B (Fig. 4 B and B´) involves equally wrong models that are indistinct. This is similar to case A except that the JC+$\mathrm{\Gamma }$ model (22, 23) is used to generate data, with different sites in the sequence evolving at variable rates according to the gamma distribution with shape parameter $\alpha =1$. The data are then analyzed using JC (equivalently to JC+$\mathrm{\Gamma }$ with $\alpha =\mathrm{\infty }$). The best-fitting parameter values (i.e., the MLEs of branch lengths in infinite data) are $t_0^{\ast }=0$ and $t_1^{\ast }=0.16441$ under each of the three binary trees. The binary trees thus represent equally wrong models (with $D_1=D_2=D_3>0$ in Eq. 1) that are indistinct. The posterior tree probabilities have a nondegenerate distribution. This is the type-2 volatile behavior for equally wrong and indistinct models (Fig. 2, $B_4$).

Case C (Fig. 4 C and C´) involves equally wrong and distinct models. Like case B, the simulation model is JC+$\mathrm{\Gamma }$ with $\alpha =1$, and the analysis model is JC. However, we do not assume the molecular clock and consider unrooted trees for four species (Fig. 1B). The true tree is the unrooted star tree $T_0$ of Fig. 1B, with $t_1=t_2=t_3=t_4=0.2$. The best-fitting parameter values (the MLEs of branch lengths in infinite data) are $t_0^{\ast }=0.01037,t_i^{\ast }=0.16409,i=\mathrm{1,2,3,4}$, for each of the three binary trees (Fig. 1B). As $t_0^{\ast }>0$, the three binary trees are different from the star tree and represent equally wrong and distinct models (with $D_1=D_2=D_3>0$ in Eq. 1). As this is a type-3 problem (Fig. 2, $C_4$), our theory predicts that as $n\to \mathrm{\infty }$, the posterior probabilities for the three binary trees should degenerate into a three-point distribution, with probability $\frac{1}{3}$ each, for (1, 0, 0), (0, 1, 0), and (0, 0, 1). In other words, one of the binary trees will have posterior $\sim 100\%$ while the other two will have $\sim 0$. This is confirmed by simulation (Table S1).

We note that most phylogenetic analyses involve unrooted trees as the clock assumption is violated except for closely related species. Furthermore, because of the violation of the evolutionary model, all trees (or the joint tree-process models) represent wrong statistical models. Thus, among the three cases considered in Fig. 4, case C is the most relevant to analysis of real data, when Bayesian model selection exhibits type-3 polarized behavior. Previous analyses of the star-tree paradox (12, 14, 15) have deplored the volatile behavior of the Bayesian phylogenetic method, but those studies examined case A only, so the real situation is worse than previously realized.

A practically important scenario is where all binary trees are wrong because of violation of the evolutionary model but the true tree is less wrong than the other trees. We present such a case in Table S2, in which the data are simulated under JC+$\mathrm{\Gamma }$ (with $\alpha =1$) using a binary tree with a short internal branch ($t_0=0.002$) and then analyzed under JC. When the amount of data approaches infinity, the true tree will eventually win, but there exists a twilight zone in which high posterior probabilities for wrong trees occur too frequently; according to Table S2, this zone is wider than ${10}^3<n<{10}^5$. For example, at sequence length $n={10}^4$ and at the 1% nominal level, the error rate of rejecting the true tree is 25.0% and the error rate of accepting a wrong tree is 16.6% (Table S2).

Discussion

High Posterior Probabilities for Phylogenetic Trees.

This work has been motivated by the phylogeny problem and in particular by the empirical observation of spuriously high posterior probabilities for phylogenetic trees (9–14). We note that certain biological processes such as deep coalescence (24, 25), gene duplication followed by gene loss (26), and horizontal gene transfer (24, 26) may cause different genes or genomic regions to have different histories. However, as discussed in the Introduction, posterior probabilities for many trees or clades observed in real data analyses are decidedly spurious even if the true tree is unknown.

One explanation for the spuriously high posterior probabilities for phylogenetic trees is the failure of current evolutionary models to accommodate interdependence among sites in the sequence, leading to an exaggeration of the amount of information in the data. Interacting sites may carry much less information than independent sites. This explanation predicts the problem to be more serious in coding genes than in noncoding regions of the genome as noncoding sites may be evolving largely independently due to lack of functional constraints. However, empirical evidence points to the opposite, with noncoding regions having higher substitution rates and higher information content (if they are not saturated with substitutions), generating more extreme posteriors for trees.

Our results suggest that the problem may lie deeper and may be a consequence of the polarized nature of Bayesian model selection when all models under comparison are misspecified. As the assumptions about the process of sequence evolution are unrealistic, the likelihood model is wrong whatever the tree, although the true tree may be expected to be less wrong than the other trees. As the different trees constitute opposing models that are nearly equally wrong, the inference problem is one of type 3 (Fig. 2, $C_4$). Bayesian tree estimation may then be expected to produce extreme posterior probabilities in large datasets.

Bayesian Selection of Opposing Misspecified Models.

We have provided a characterization of model selection problems according to the asymptotic behavior of the Bayesian method as the data size $n\to \mathrm{\infty }$ [Fig. 2 and General Theory for Equally Wrong Models with No Free Parameters (d=0) and General Theory for Equally Right or Equally Wrong Models with Free Parameters (d>0)]. While all of the problems considered here involve comparison of two equally right or equally wrong models, three different asymptotic behaviors are identified, which we label as type 1, type 2, and type 3. The type-1 behavior is for the posterior model probability $P_1$ to converge to a sensible point value, such as $\frac{1}{2}$. We consider this to be a good balanced behavior, following phylogeneticists (10–12). The rationale is that one would like a sure answer given an infinite amount of data and the only reasonable sure answer should be $\frac{1}{2}$ for each model, since the data contain no information for or against either model. This behavior occurs only when the two models are identical or overlapping, a situation that does not appear relevant to scientific inference. With type-2 behavior, $P_1$ fluctuates among datasets (each of infinite size) like a random number, so that strong support may be attached to a particular model in some datasets. Biologists were surprised at this erratic behavior (10–12), which we label as volatile. This occurs when the models are equally right or equally wrong but indistinct. In theory, type-2 behavior may not pose a serious problem, because the parameter posteriors under the models, if examined carefully, should make it clear that the competing models essentially gave the same interpretation of the data and should lead to the same scientific conclusion. In data simulated in ref. 12 or in Fig. 4 A and A´, the estimates of $t_0$ should be very close to 0, and all binary trees are similar to the same star tree. Nevertheless this escaped our attention at the time.

With type-3 behavior, $P_1$ is $\sim 0$ in half of the datasets and $\sim 1$ in the other half. We describe this behavior as polarized. This occurs when the two models are equally wrong and distinct. Type-3 problems may be the most relevant to practical data analysis given that all models are simplified representations of reality and are thus wrong. A variation to type-3 problems is when one model is only slightly less wrong than another (Fig. 3 A, ii and C, ii and Table S2). While the less wrong model eventually wins in the limit of infinite data, Bayesian model selection is overconfident in large but finite datasets, supporting the more wrong model with high posterior too often.

Note that the question of how the posterior model probability should behave when large datasets are used to compare two equally wrong models is somewhat philosophical and may not have a simple answer. One position is to accept whatever behavior the Bayesian method exhibits. This may be legitimate given that Bayesian theory is the correct probability framework for summarizing evidence in the prior and likelihood. The polarized behavior in type-3 problems may then be seen as a consequence of “user error” (for not including the true model in the comparison), exacerbated by the large data size. In this regard we note that the posterior predictive distribution (27, 28) can be used to assess the general adequacy of any model or the compatibility between the prior and the likelihood, and indeed this has been widely used to assess the goodness of fit of models in phylogenetics (29, 30). Nevertheless, a number of sophisticated and parameter-rich models have been developed for Bayesian phylogenetic analysis, due to three decades of active research (31), and furthermore extreme sensitivity to the assumed model is not a desirable property of an inference method. Seven decades ago, Egon S. Pearson (ref. 32, p.142) wrote that “Hitherto the user has been accustomed to accept the function of probability theory laid down by the mathematicians; but it would be good if he could take a larger share in formulating himself what are the practical requirements that the theory should satisfy in application.” This stipulation may be relevant even today.

Two heuristic approaches have been suggested to remedy the high posterior model probabilities in the context of phylogenies. The first one is to assign nonzero probabilities to multifurcating trees (such as the star tree of Fig. 1) in the prior (11). This is equivalent to assigning some prior probability to the model $p=0.5$ in the fair-coin example of problem 1. While this resolves the star-tree paradox, it suffers from the conceptual difficulty that the multifurcating trees may not be plausible biologically. The second approach is to let the internal branch lengths in the binary trees become increasingly smaller in the prior when the data size increases (12, 14). This is non-Bayesian in that the prior depends on the size of the data. With both approaches, the posterior is extremely sensitive to the prior (9).

Non-Bayesian Methods.

The phylogeny problem was described by Jerzy Neyman (ref. 33, p. 1) as “a source of novel statistical problems.” In the frequentist framework, the test of phylogeny, or test of nonnested models in general, offers challenging inference problems. Note that in many model selection problems, the model itself is not the focus of interest. For example, when an experiment is conducted to evaluate the effect of a new fertilizer, the sensitivity of the inference to the assumed normal distribution with homogeneous variance may be of concern, but the focus is not on the normal distribution itself. In phylogenetics, the phylogeny (which is a model) is of primary interest, far more important than the branch lengths (which are parameters in the model). The test of phylogeny is thus more akin to significance/hypothesis testing than to model selection. Model-selection criteria such as Akaike information criteria (34) or Bayesian information criteria (35) simply rank the trees by their likelihood (maximized over branch lengths) and will not be useful for attaching a measure of significance or confidence in the estimated tree. The phylogeny problem (or the problem of comparing nonnested models in general) falls outside the Fisher–Neyman–Pearson framework of hypothesis testing, which involves two nested models, one of which is true (36, 37).

In principle Cox’s likelihood-ratio test (38), which conducts multiple tests with each model used as the null, can be used to compare nonnested models. For type-3 problems (Fig. 2, $C_1$–$C_4$), this test should lead to rejection of all models. Cox’s test has not been used widely in phylogenetics, apparently because of the existence of a great many possible trees and the heavy computation needed to generate the null distribution by simulation.

The most commonly used method for attaching a measure of confidence in the maximum-likelihood tree is the bootstrap (39), which samples sites (alignment columns) to generate bootstrap pseudodatasets and calculates the bootstrap support value for a clade (a node on the species tree) as the proportion of the pseudodatasets in which that node is found in the inferred ML tree. This application of bootstrap for model comparison appears to have important differences from the conventional bootstrap for calculating the standard errors and confidence intervals for a parameter estimate (40); a straightforward interpretation of the bootstrap support values for trees remains elusive (31, 41–43). At any rate, the asymptotic behavior of bootstrap support values under the different scenarios of Fig. 2 merits further research. For the fair-coin example of problem 1 (Fig. 2, $C_1$), the bootstrap support converges to $U(\mathrm{0,1})$, different from the posterior probability, although other cases are yet to be explored.

Materials and Methods

Star-Tree Simulations.

For Fig. 4 A, A´, B, and B´, the true tree is $T_0$ of Fig. 1A. The data of counts of five site patterns ($xxx$, $xxy$, $yxx$, $xyx$, and $xyz$) were simulated by multinomial sampling (21) and analyzed using a C program, which calculates the 2D integrals in the marginal likelihood by Gaussian-Legendre quadrature with 128 points (14). For Fig. 4 C and C´, the true tree is $T_0$ of Fig. 1B. Sequence alignments were simulated using EVOLVER and analyzed using MrBayes (4).