Estimating species richness by a Poisson-compound gamma model

Summary

We propose a Poisson-compound gamma approach for species richness estimation. Based on the denseness and nesting properties of the gamma mixture, we fix the shape parameter of each gamma component at a unified value, and estimate the mixture using nonparametric maximum likelihood. A least-squares crossvalidation procedure is proposed for the choice of the common shape parameter. The performance of the resulting estimator of N is assessed using numerical studies and genomic data.

1. Introduction

A typical dataset in the species problem comprises a series of counts {x_i : i = 1, . . ., D}, recording the numbers of individuals from a total of D observed distinct species. These counts are often summarized into the sufficient statistic n = (n₁, . . ., n_t), where n_j = ∑_i I {x_i = j}, t = max{x_i}. The parameter for estimation is the total number of the distinct species N in the underlying population. Research on this old but challenging problem is motivated by several novel genomic applications, and the following are two examples.

Serial analysis of gene expression data and expressed sequence tag data are two similar types of genomic data widely used for surveying transcribed genes. Both are collections of tags or signatures of genes, representing random samples from the entire gene expression population or mRNA pool (Thygesen & Zwinderman, 2006; Morris et al., 2003; Wang et al., 2005). We let n_j stand for the number of genes that have j tags in a sample; but owing to the limited sampling or sequencing effort, some expressed genes may not be observed. By knowing the total number of expressed genes, biologists can gain insights into the diversity of active genes in the given tissue.

The diversity of microbial species present can provide an interesting profile of a microbial community (Acinas et al., 2004; Hong et al., 2006). For example, using the 16S ribosomal RNA gene as the signature, researchers can identify distinct microbial species present in a soil sample. However, sampling is often shallow in practice, i.e. only a small fraction of the species is obtained, and the total diversity needs to be estimated.

In both examples, the gene expression level or the species abundance varies substantially. The first challenge in the estimation of N is how to model the heterogeneity. Ignoring it may result in a large downward bias (Chao & Lee, 1992; Bunge & Fitzpatrick, 1993; Norris & Pollock, 1998; Böhning & Schön, 2005). In a likelihood-based approach, one first specifies a probability distribution function f (x_i ; λ_i) (λ_i > 0), and then imposes a nondegenerate distribution Q(λ; θ), θ ∈ Θ ⊂ 𝕉^d, typically continuous, for the species abundance index λ_i. Thus, the marginal distribution of X becomes a mixture distribution, i.e. f (x; Q) = ∫ f (x; λ)dQ(λ). However, an inappropriate choice of Q can lead to systematic bias of the resulting estimator.

A more robust approach is to estimate Q by nonparametric maximum likelihood (Norris & Pollock, 1998). The nonparametric maximum likelihood estimator, Q̂, is in the form of a discrete distribution (Lindsay, 1995), and in the case of a Poisson mixture it is unique with a finite number of support points for any dataset. Wang & Lindsay (2005) proposed a penalized nonparametric maximum likelihood estimator, which is more stable than that of Norris & Pollock (1998). Nevertheless, severe underestimation can occur due to the interplay of factors including inadequate sampling effort, heterogeneity and skewness of the abundance curve. This weakness is also observed in other non-likelihood-based nonparametric approaches. In the motivating genomic applications, shallow sampling is typical. Thus, underestimation is the second major challenge.

In the motivating applications, a continuous Q can be appealing for interpretability of species abundance. More importantly, it could better capture the information of species abundance near zero, and hence improve the estimation of N when underestimation is a major concern. The objective of this study is to model Q using a smooth curve while estimating it using nonparametric maximum likelihood based on a Poisson-compound gamma model.

2. The model and maximum likelihood estimation

2.1. A general model

Consider that we have a mixture distribution f (x; H) defined based on a density kernel f (x; λ), λ ∈ 𝕉 and a mixing distribution H(λ; μ, ν), μ, ν ∈ 𝕉. Here, we assume that H is continuous and has location and dispersion parameters μ and ν, respectively. Furthermore, as ν → 0, H becomes degenerate, and thus, f (x; H) converges to a unicomponent distribution. If we fix ν and allow a mixing distribution G for μ, then the overall mixing distribution for λ becomes continuous, having a form Q(λ) = ∫ H(λ; μ, ν)dG(μ). The mixture density of X becomes $f_{\nu }(x;G)={\int }_{\mu }{f}_{\nu }^{*}(x;\mu )dG(\mu )$, where $f_{\nu }^{*}(x;\mu )={\int }_{\lambda }f(x;\lambda )dH(\lambda ;\mu ,\nu )$. If H is chosen to be conjugate for f, then $f_{\nu }^{*}$ can often be obtained in a simple closed form. For each given ν, the global maximum likelihood estimator of G, say Ĝ_ν, can be found by nonparametric maximum likelihood estimation (Lindsay, 1983, 1995). The estimator Ĝ_ν is a discrete distribution, while the resulting mixing distribution Q of λ, denoted by Q̂_ν(λ), is continuous. In this paper, we term Q̂_ν (λ) the smooth nonparametric maximum likelihood estimator (Magder & Zeger, 1996).

Alternatively, one can ignore the H structure and directly estimate Q(λ) in the defined mixture model f (x; Q) = ∫ f (x; λ)dQ by nonparametric maximum likelihood. The resulting discrete Q̂(λ) maximizes the likelihood over all Q; we call it the discrete nonparametric maximum likelihood estimator.

2.2. Properties of gamma mixtures

Let h(λ; α, β) = β^α{Γ(α)}⁻¹λ^α−1e^−βλ (λ, α, β > 0) be the gamma density. We consider two reparameterizations: (α, β) ↦ (α, μ) and (α, β) ↦ (μ, β), where μ = α/β. The following denseness property of the gamma mixture is cited from Fang & Chlamtac (1999, p. 1064):

Lemma 1. Let Q(λ) be the cumulative distribution function of a nonnegative random variable. Then it is possible to choose a sequence of distribution functions Q_m (λ), each of which corresponds to a gamma mixture, such that Q_m (λ) → Q(λ) as m → ∞, at any point where Q(λ) is continuous. For example, we may choose

$$Q_m(\lambda )=\sum _{k=1}^{\infty }[Q(k/m)-Q\{(k-1)/m\}]Q_m^k(\lambda ),$$

where $Q_m^k(\lambda )$ is a gamma distribution with mean k/m and variance k/m².

Intuitively, when m becomes large, each individual gamma component $Q_m^k$ approximates a delta function at k/m with a weight equal to the increase in the cumulative distribution function of Q from (k − 1)/m to k/m. Therefore, a gamma mixture model can be used to approximate any distribution defined in 𝕉₊. In practice, Q is unknown and the sample size is limited, and one can fit a gamma mixture by the method of maximum likelihood.

The second property is the nesting property of gamma mixtures.

Lemma 2. Let h(λ ; θ, μ₀) be the reparameterized gamma density where θ = α or β and μ₀ is the mean parameter. Then for any given δ > 0, there exists a unique probability measure G for μ such that for all λ > 0, ∫h(λ ; θ + δ, μ)dG(μ) = h(λ; θ, μ₀).

Proof. See the Appendix.

The existence part of Lemma 2 for θ = α is a result from Lindsay (1995, p. 54), given in the new parameterization. The existence result for θ = β case and the uniqueness for both α and β cases are new. The uniqueness is a consequence of the identifiability presented in Theorem 2 below. Subsequently, we shall use h{θ, G(μ)} to denote a gamma mixture defined as ∫h(λ; θ, μ)dG(μ). The following nesting property is an immediate consequence of Lemma 2.

Theorem 1. Let δ > 0, θ > 0 and h(θ, μ) be the reparameterized gamma density, where θ = α or β. Let 𝒢 = {G(μ) : G is a probability measure on 𝕉₊}. Define the mixture sets 𝒣_θ = [h{θ, G(μ)} : G ∈ 𝒢] and 𝒣_θ+δ = [h{θ + δ, G(μ)} : G ∈ 𝒢]. Then 𝒣_θ ⊂ 𝒣_θ+δ.

Lemma 2 implies that an arbitrary gamma distribution h(θ, μ₀) can be uniquely expressed as a gamma mixture h{θ + δ, G(μ)} for any given δ > 0. Hence, a gamma mixture model, h{θ, G(μ)} ∈ 𝒣_θ, can be rewritten as a model in 𝒣_θ+δ, but with a different G(μ). This implies that G(μ) is not identifiable in the entire distribution space 𝒢 if θ is unknown. Nevertheless, if θ is given, G is identifiable in 𝒢.

Theorem 2. Let h{θ, G(μ)} be a gamma mixture model as defined earlier with a given θ > 0 and an unknown G(μ) defined in 𝕉₊. Then G(μ) is identifiable in 𝒢.

Proof. See the Appendix.

The identifiability of the gamma mixture model under a given unified θ implies the uniqueness of G(μ), as stated in Lemma 2. The gamma mixture with different θs is not identifiable in the entire distribution space 𝒢. If we assume that the mixture is finite, i.e. the mixing distribution G is discrete with a finite number of support points, then the mixture becomes identifiable (Teicher, 1963). In practice, as we fit a finite mixture of gammas, the identifiability among the finite mixture set is adequate to justify the estimation.

2.3. Poisson-compound gamma model

The denseness property suggests that there exists a sequence of finite gamma mixtures that can approximate any Q in 𝕉₊. As Q is unknown, we need to estimate Q based on the data. If Q̃ is such an estimator in a finite gamma mixture form, then by the nesting property Q̃ can be rewritten as a new gamma mixture, say Q̃_θ, with a unified θ parameter. However, the likelihood for such Q̃_θ is always dominated by that of the smooth nonparametric maximum likelihood estimator Q̂_θ obtained with the given θ. In other words, the smooth nonparametric maximum likelihood estimator with an appropriate choice of θ can also approximate Q arbitrarily well regardless of the form of true Q. Hence, robustness of this method is anticipated.

We consider a Poisson-compound gamma model for the species problem. If the compound gamma has a fixed θ and a mixing distribution G for μ, then f_θ (x; G) = ∬ e^−λλ^x(x!)⁻¹ h(λ; θ, μ)dλdG(μ). We can define $f_{\theta }^{*}(x;\mu )=\int e^{-\lambda }{\lambda }^x{\left(x!\right)}^{-1}h(\lambda ;\theta ,\mu )d\lambda$. For example, under the (α, μ) parameterization, $f_{\alpha }^{*}(x;\mu )=\mathrm{\Gamma }(x+\alpha ){\left(\alpha /\mu \right)}^{\alpha }/\left\{x!\mathrm{\Gamma }(\alpha ){\left(1+\alpha /\mu \right)}^{x+\alpha }\right\}$. Then $f_{\alpha }(x;G)=\int f_{\alpha }^{*}(x;\mu )dG(\mu )$. The likelihood function is

$$L_{\theta }(N,G;n)\propto \left(\begin{array}{c}N\\ D\end{array}\right)f_{\theta }{\left(0;G\right)}^{N-D}{\{1-f_{\theta }(0;G)\}}^D\times \prod _{j>0}{\left\{\frac{f_{\theta }(j;G)}{1-f_{\theta }(0;G)}\right\}}^{n_j},$$ (1)

where the expressions before and after the multiplication sign are referred to as the marginal and conditional likelihood, respectively, and are denoted by $L_{\theta }^{\text{m}}$ and $L_{\theta }^{\text{c}}$ henceforth.

Based on the likelihood in (1), there are two possible likelihood approaches, conditional or unconditional (Sanathanan, 1972, 1977). In the conditional approach, the mixing distribution G, say, Ĝ_c, is first estimated from $L_{\theta }^{\text{c}}$ alone, and then the estimator of N by N̂_c = ⌈D/{1 − f_θ(0, Ĝ_c)}⌉, where ⌈x⌉ stands for the integer no greater than x. The unconditional approach seeks the pair (N̂_u, Ĝ_u) to maximize the full likelihood, which must also satisfy N̂_u = ⌈D/{1 − f_θ(0, Ĝ_u)}⌉ (Wang & Lindsay, 2005).

In this paper, we consider the conditional approach for its computational simplicity. Below we shall use Ĝ to denote the conditional nonparametric maximum likelihood estimator of G. A frequently occurring issue in the conditional discrete nonparametric maximum likelihood approach is the boundary problem, where a component λ ≈ 0⁺ is fitted. The same problem arises in the unconditional discrete (Wang & Lindsay, 2005) and the finite mixture approaches. Unfortunately, the continuity brought into Q in the Poisson-compound gamma model does not help overcome this issue. As the gamma distribution has μ = α/β, σ² = μ/β, with the given α, fitting a μ ≈ 0⁺ implies β → ∞, and hence, σ² → 0. Therefore, the fitted gamma component corresponds to a spike of λ at 0⁺. Such a tiny component is problematic because it blows up the resulting point estimate of N. In Wang & Lindsay (2008), it was further shown that this boundary problem is structural in the sense that the fitted probability at X = 1 for the zero-truncated sample must be no less than the empirical probability n₁/∑n_i. It can be shown that this is also true for smooth nonparametric maximum likelihood estimation.

An effective solution to this problem is to penalize the likelihood. Define the odds parameter ψ as ψ(G) = f_θ (0; G)/{1 − f_θ (0; G)}. Then N̂_c can be written as N̂_c = ⌈D{1 + ψ(Ĝ)}⌉. Thus, penalizing large ψ values can stabilize the conditional estimator. Such a penalty function of ψ in the likelihood has a natural interpretation as a Bayesian prior for ψ, in which, however, the hyperparameter needs to be subjectively prespecified (Wang & Lindsay, 2005). To overcome this weakness, here we use a partial prior approach similar to that proposed in Wang & Lindsay (2008) for the binomial mixture. We impose an exponential prior for the odds parameter in a form of p(ψ; η) = ηe^−ηψ (ψ, η > 0). As we are estimating the entire mixing distribution in ${ℝ}_{+}^{\infty }$ while the prior is specified for a functional of G in 𝕉₊, such a prior is termed a partial prior. We use an empirical Bayes method to find (η̂, Ĝ) that maximizes the joint likelihood

$${\ell }_{\theta }^{*}(G,\eta ;n)=\text{log}\left(L_{\theta }^{\text{c}}\right)+\text{log}(\eta )-\eta \psi ,$$ (2)

where $L_{\theta }^{\text{c}}$ is the conditional likelihood defined in (1). It was shown in Wang & Lindsay (2008) that, ψ(Ĝ) = 1/η̂, and Ĝ maximizes ${\ell }_{\theta }^{*}$ if and only if it maximizes the alternative likelihood

$${\ell }_{\theta }^{**}(G;n)=\text{log}\left(L_{\theta }^{\text{c}}\right)-\text{log}\left\{\psi (G)\right\}.$$ (3)

Clearly, as ψ → ∞, we have ${\ell }_{\theta }^{**}\to -\infty$. Thus, the boundary problem is prevented. Compared to the quadratic penalty in Wang & Lindsay (2005), the exponential partial prior approach is simpler and less ad hoc. The proposed estimator of N in this paper is based on the likelihood ${\ell }_{\theta }^{*}$ or its equivalent ${\ell }_{\theta }^{**}$. The penalized nonparametric maximum likelihood estimator of G from ${\ell }_{\theta }^{**}$ can be reliably found using an em algorithm (Wang, 2007).

3. Choosing θ by least-squares crossvalidation

The nesting property of the gamma mixture directly results in the following monotonicity property of the likelihood function defined in (1), (2) and (3), whose proof is omitted.

Proposition 1. Let ℓ̂_θ be the maximized loglikelihood value from the smooth nonparametric maximum likelihood estimation based on conditional or unconditional likelihood from (1), or based on the likelihood function with the partial prior as defined in (2) or (3). Then ℓ̂_θ is a monotone nondecreasing function of θ.

Proposition 1 implies that the global maximum likelihood is achieved at the discrete nonparametric maximum likelihood estimator, namely θ = ∞. When θ = α, as σ² = μ₂/α, a larger α for given μ implies a more spiky gamma density. As α → ∞, each gamma component will converge to a degenerate distribution. The same conclusion holds when θ = β. By design, the tuning parameter θ brings better smoothness to Q(λ), and should reduce the underestimation. The likelihood value under smooth nonparametric maximum likelihood estimation can be regarded as the profile likelihood for θ. Unfortunately, the corresponding likelihood ratio test does not have a known distribution (Lindsay, 1995). Hence we propose a crossvalidation procedure for model selection.

In our estimation procedure, the parameter θ plays the same role as a bandwidth parameter in kernel density estimation; it tunes the smoothness of the resulting gamma mixture density. However, our problem differs from the latter in two senses: the species abundance value from the gamma mixture is not directly observed, and the density estimator is defined as a gamma mixture rather than a convolution of Gaussian densities. We investigate a crossvalidation procedure similar to that in density estimation (Silverman, 1992, p. 48) to determine the value of the tuning parameter. Our goal is to find a θ that minimizes the summed square error

$$\text{SSE}=\sum _{j=1}^{\infty }{\left\{f(j;\widehat{G})-f(j;\widehat{G})\right\}}^2=\sum _{j=1}^{\infty }{\left\{f(j;\widehat{G})\right\}}^2-2\sum _{j=1}^{\infty }\left\{f(j;\widehat{G})f(j;\widehat{G})\right\}+\sum _{j=1}^{\infty }{\left\{f(j;G)\right\}}^2,$$ (4)

where f(j; Ĝ) is the estimated zero-truncated Poisson mixture probability at j, and f (j; G) is its true value. The last term in (4), ${\sum }_{j=1}^{\infty }{\left\{f(j;G)\right\}}^2$, does not involve Ĝ, so a good choice of Ĝ is one that minimizes ${\sum }_{j=1}^{\infty }{\left\{f(j;\widehat{G})\right\}}^2-2{\sum }_{j=1}^{\infty }\left\{f(j;\widehat{G})f(j;G)\right\}$. Because f (j; G) is unknown, we replace it with an empirical version, n_j/D. In addition we define f̂_−j as the leave-one-out crossvalidated probability density after excluding one species with count j from the sample. Further replacing f(j; Ĝ) with f̂_−j (j) in the second term leads us to minimize the function,

$$M_0(\theta )=\sum _{j=1}^{\infty }{\left\{f(j;\widehat{G})\right\}}^2-2\sum _{j\in \{j:n_j>0\}}\frac{n_j}{D}{\widehat{f}}_{-j}(j),$$ (5)

with respect to θ. As f̂_−j (j) ≈ f (j; Ĝ),

$$M_0(\theta )\approx \sum _{j=1}^{\infty }{\{f(j;\widehat{G})-n_j/D\}}^2-\sum _j{(n_j/D)}^2.$$

Given the data, minimizing M₀(θ) tends to minimize ${\sum }_{j=1}^{\infty }{\{f(j;\widehat{G})-n_j/D\}}^2$, to provide good fit to the data. Incorporating the leave-one-out crossvalidated probability f_−j(j) in M₀(θ) in (5) introduces a variance component into the objective function M₀(θ), owing to a small perturbation of the data. In the Poisson-compound gamma model, an over-small θ results in a large summed square error due to lack of fit or bias, whereas an over-large θ, though always beating any smaller θ in terms of the likelihood by the nesting property, may yield a relatively larger variance in f(j; Ĝ). Minimizing M₀(θ) over θ balances the two competing components in this model.

The summed square error defined earlier is analogous to the integrated square error used in the kernel density estimation. Although we are unable to establish similar asymptotics because of the complexity of nonparametric maximum likelihood estimation of mixtures, this procedure works surprisingly well in our application. We illustrate this using simulated species data from a Poisson-compound gamma model, where N = 1000, and the species abundance is a two-component gamma mixture 0.5Ga(α = 2, μ = 1) + 0.5Ga(α = 2, μ = 2). The data are: n = (n₁, . . ., n₁₄) = (269, 183, 88, 56, 34, 8, 9, 6, 0, 1, 1, 1, 1, 1). We consider the θ = α setting by treating α as a tuning parameter. In Fig. 1, we plot the maximized log conditional likelihood under the exponential partial prior on a grid of α ∈ 𝒜 = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 40, 60, 100} together with M₀(α) and the resulting N̂.

Fig. 1.

Plots of (a) maximized loglikelihood relative to that achieved at discrete nonparametric maximum likelihood, (b) M₀, and (c) N̂ versus log(α) at α = 1, . . ., 10, 20, 40, 60, 100 for the example data. The dashed line in (c) corresponds to the discrete nonparametric maximum likelihood estimator, N̂ = 886.

The loglikelihood plot in Fig. 1(a) confirms the monotonicity. The crossvalidation procedure identified α = 2 on the grid that minimizes M₀(α), yielding N̂ = 981. The estimator N̂ decreases in α, until reaching 886 at α = ∞. This overall decreasing pattern of N̂ is typical, though strict monotonicity seems not to hold based on our experience. The discrepancy between N̂ at α = 2 and at α = ∞ suggests that an appropriate choice of α model could lead to a less biased estimator than the discrete nonparametric maximum likelihood estimator.

4. Simulation study

We now investigate whether this new approach could preserve the desired robustness as a nonparametric maximum likelihood estimator, and whether it could significantly reduce the underestimation. Table 1 lists 12 settings, where the species abundance distribution Q(λ) varied from gamma, gamma mixture, lognormal and lognormal mixture, to discrete distributions, with the expected fraction of the sampled species, defined as the sampling depth, d, ranging from 0.20 to 0.90. For each setting, we simulated 200 samples from a population containing N = 1000 species. Crossvalidation was applied to each sample on a grid α ∈ 𝒜 = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 40, 60, 100, ∞}. One could carry out the crossvalidation for α < 1, but the gamma density goes to ∞ at 0⁺ for α < 1. Consequently, the resulting estimator can be extremely variable or biased. Hence, in this simulation, we took only α ⩾ 1.

Table 1.

Species abundance distribution and sampling depth in 12 simulation settings

Setting	Distribution type (Q)	d
	gamma
1	Ga(4,3.125)	0.90
2	Ga(4,1)	0.59
3	Ga(1,0.25)	0.20
	gamma mixture
4	0.5Ga(2,1) + 0.5Ga(2,2)	0.65
5	0.5Ga(2,1) + 0.5Ga(4,1)	0.57
	Lognormal
6	LN (0.75,0.75)	0.82
7	LN (−.5,2)	0.50
8	LN (−1,1)	0.36
	Lognormal mixture
9	0.5LN(−0.5,1) + 0.5LN(0.5,1)	0.61
	Discrete
10	0.8Δ1.2 + 0.2Δ6.7	0.76
11	0.89Δ0.5 + 0.11Δ6.7	0.46
12	0.8Δ0.2 + 0.2Δ1.3	0.29

Ga, gamma (α, μ); LN, lognormal(μ, σ); Δ_x, a degenerate distribution at x ; d is defined as the sampling depth, which is the expected fraction of species to be observed in the sample, i.e. d = 1 − f (0; Q).

Overall, the crossvalidation procedure works satisfactorily in selecting appropriate models for different settings. For the single gamma models including settings 1–3, and the gamma mixture setting 4 where the components had the same α parameter, the medians of the selected α values, denoted as α̂, were 4, 4, 1 and 2, respectively, which were consistent with the true values. For setting 5, where the gamma mixture components had differing α values, the crossvalidation tended to select some intermediate α̂ values with a median of 3. For the three discrete Q models, settings 10–12, the medians of α̂ were 100, 80 and 1, respectively. The difference probably arises because in the first two, the sampling was deeper, and the two discrete components were farther apart than setting 12. As a result, the crossvalidation procedure can better identify the latent structure in the former two. For the four lognormal models including settings 6–9, the medians were 2, 1, 1 and 1, respectively, suggesting that the crossvalidation procedure tended to select gamma components with a larger coefficient of variation to approximate the lognormal distributions. In Fig. 2, we present the histograms of α̂ from settings 2, 4, 5, 8, 11 and 12.

Fig. 2.

Histogram of selected α values in simulation settings (a) 2, (b) 4, (c) 5, (d) 8, (e) 11, and (f) 12. Each bar in the histogram represents the frequency of selected α models for α ∈ 𝒜 = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 40, 60, 100, ∞}. The bar at α = 101 represents the frequency at α̂ = ∞.

Table 2 summarizes the Monte Carlo estimates from the proposed estimator, denoted by N̂_pcg, in comparison with other popular estimators, including the penalized discrete nonparametric maximum likelihood estimator N̂_wl of Wang & Lindsay (2005); two sample coverage-based estimators N̂_c1 and N̂_c2, corresponding to N̂₂ and N̂₃, respectively from Chao & Lee (1992), where N̂_c2 provides further bias correction based on N̂_c1 ; a coverage-duplication estimator N̂_c3 based on a Poisson-gamma model from Chao & Bunge (2002); and a jackknife estimator N̂_j from Burnham & Overton (1978, 1979). The jackknife order was selected up to five based on a procedure proposed in the original papers. A confidence interval for N̂_pcg was obtained using a standard bootstrap procedure as follows. Let g(j; Ĝ_α̂)(j = 1, 2 . . .) be the fitted zero-truncated Poisson mixture probabilities from a Monte Carlo sample with D observed distinct species. We generated 200 bootstrap samples, each containing D observations sampled from g(j; Ĝ_α̂). The crossvalidation procedure was applied to each bootstrap sample and N̂_pcg was calculated. The interval spanned by the 0.025 and 0.975 quantiles of the N estimates was regarded as the bootstrap confidence interval. We also computed three additional likelihood-based estimators without a partial prior, including N̂_fm, N̂_pcg2 and N̂_pcg3. Estimator N̂_fm was obtained by fitting a finite mixture of zero-truncated Poisson distributions to the data. Estimators N̂ _pcg2 and N̂_pcg3 were the conditional and unconditional maximum likelihood estimators, respectively, based on the Poisson-compound gamma model without fixing α or β. In each case, the number of components was selected using the Bayesian information criterion; and the confidence interval coverage was based on the same bootstrap procedure as for N̂_pcg.

Table 2.

Comparing nine different estimators with respect to median bias, mean absolute error and 95% confidence interval coverage in 12 simulation settings listed in Table 1

Setting	N̂	M̂	s	MAE	Cov.	Setting	M̂	s	MAE	Cov.	Setting	M̂	s	MAE	Cov.
1	N̂pcg	1003	25	19	96	2	1021	135	95	95	3	925	234	195	98
	N̂wl	1004	46	33	90		998	112	86	92		674	148	314	82
	N̂c1	982	14	20	75		925	44	76	70		615	101	376	18
	N̂c2	988	16	16	88		953	57	60	89		654	150	326	46
	N̂c3	989	16	16	90		993	89	67	96		–	–	–	–
	N̂j	1075	17	74	100		1134	108	140	43		590	85	406	1
	N̂fm	974	16	27	38		868	87	131	14		541	177	448	22
	N̂pcg2	1001	17	13	92		991	90	68	95		–	–	–	–
	N̂pcg3	999	17	13	87		981	85	66	95		–	–	–	–
4	N̂pcg	1097	127	118	99	5	1012	140	104	96	6	1022	65	50	98
	N̂wl	955	98	84	99		965	110	88	99		1008	73	58	95
	N̂c1	900	34	101	23		895	44	102	45		965	21	34	69
	N̂c2	951	45	55	87		934	59	72	83		989	25	22	93
	N̂c3	1014	71	55	96		999	105	78	95		1001	28	22	95
	N̂j	1036	80	60	84		1094	106	107	33		1064	65	74	33
	N̂fm	874	170	145	27		886	189	148	22		885	31	114	14
	N̂pcg2	1038	83	72	92		1002	107	79	96		1100	207	120	26
	N̂pcg3	1028	81	66	95		991	101	76	94		1094	62	99	100
7	N̂pcg	847	130	156	78	8	1033	191	147	100	9	1041	139	108	100
	N̂wl	755	91	229	57		794	115	199	97		920	96	99	100
	N̂c1	674	29	325	0		769	72	226	23		856	35	142	7
	N̂c2	739	43	257	2		940	140	124	87		932	50	71	76
	N̂c3	846	98	153	73		–	–	–	–		1036	98	82	95
	N̂j	802	112	193	19		892	126	131	55		984	112	76	75
	N̂fm	628	50	363	80		699	106	285	32		821	66	176	28
	N̂pcg2	–	–	–	–		–	–	–	–		–	–	–	–
	N̂pcg3	581	97	396	25		–	–	–	–		–	–	–	–
10	N̂pcg	1013	36	31	83	11	1035	133	99	92	12	915	168	148	94
	N̂wl	1026	60	49	67		1048	142	116	66		742	129	247	89
	N̂c1	1060	36	65	52		1003	71	55	96		625	64	373	2
	N̂c2	1171	54	180	2		1487	172	490	2		676	93	314	21
	N̂c3	1343	115	365	0		–	–	–	–		–	–	–	–
	N̂j	1160	45	164	0		1183	144	215	46		797	104	205	32
	N̂fm	1002	31	25	86		1006	461	386	51		534	244	459	7
	N̂pcg2	1018	39	34	80		–	–	–	–		–	–	–	–
	N̂pcg3	1025	59	41	87		–	–	–	–		–	–	–	–

M̂, s and MAE are the sample median, standard deviation and mean of absolute error calculated based on 200 Monte Carlo estimates for each setting. Cov. is the average of 95% confidence interval coverage evaluated from 200 samples. N̂_c1, N̂_c2 and N̂_c3 were estimated based on less abundant subset of species defined as n_j ⩽ 10. For N̂_pcg, N̂_wl, N̂_fm, N̂_pcg2 and N̂_pcg3, the coverage of confidence interval was estimated based on a similar bootstrap procedure. ‘–’ stands for situations where extreme bias/variance occurred.

For each setting, the summary statistics based on 200 Monte Carlo estimates presented in Table 2 include the median, standard deviation, mean absolute error and the coverage of a nominal 95% confidence interval. For N̂_c1, N̂_c2 and N̂_c3, we used a log-transformation procedure from Chao (1987) for the 95% confidence interval. For computational reasons the number of Monte Carlo replicates is too small for precise estimation of coverages, but despite this major differences between the procedures are clear.

Overall, N̂_pcg exhibited the desirable robustness similar to N̂_wl, N̂_c1, N̂_j and N̂_fm, having no extreme behaviour in any setting. In contrast, N̂_pcg2, N̂_pcg3 and N̂_c3, which make gamma or gamma mixture assumptions for Q, are susceptible to model misspecification. For example, they all had large bias in multiple settings. In particular, for N̂_pcg2 and N̂_pcg3, the Bayesian information criterion tended to eliminate the boundary component if its weight was small, but not otherwise. Consequently, large bias was still occasionally observed. In most other cases, the positive bias appeared to be systematic, which occurred when using gamma models to approximate lognormal and discrete distributions in settings 7, 8, 9, 11 and 12. A large bias was frequently observed when a fitted gamma component had α̂ < 1. Such results were excluded from Table 2. Although N̂_pcg was built on the same model, its estimation strategy, including using the partial prior, crossvalidation and controlling α ⩾ 1, appeared to be effective in alleviating such issues regardless of the true form of Q. For N̂_fm, we observed a few extremely large estimates owing to the boundary problem in several settings. In the summary in Table 2, such extreme estimates, if greater than 2000, were replaced with 2000.

We further calculated the median relative bias, defined as bias/N for each setting. The average of the absolute median relative bias across different settings for N̂_pcg, N̂_wl, N̂_c1, N̂_c2, N̂_c3, N̂_j and N̂_fm was 4.6%, 10%, 14.6%, 14.8%, 6.3%, 13.9% and 19.2%, respectively. The average bias of 6.3% for N̂_c3 was obtained after excluding four settings where it had large bias. In particular, when the sampling was shallow and underestimation was a severe issue, N̂_pcg significantly improved over other estimators regarding bias.

The proposed estimator N̂_pcg pays a price of relatively larger variance for the achieved robustness, improved bias and confidence interval coverage when the true Q is continuous. For example, when compared to the coverage-based estimator N̂_c2, N̂_pcg had relatively larger mean absolute error in seven settings, though the coverage of confidence interval of N̂_pcg was uniformly better in such settings.

5. Data analysis

In the motivating genomic applications, each dataset often presents a shallow survey of the entire population of genes or microbial species. Hence, underestimation is probably the primary concern. In Table 3, we present three datasets including the microbial species data from Acinas et al. (2004), the Arabidopsis thaliana expressed sequence tag data from Wang & Lindsay (2005) and the traffic data from Simar (1976) and Böhning & Schön (2005).

Table 3.

Microbial species, Arabidopsis thaliana root expressed sequence tag and Traffic data

j	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17+
Microbial (nj)	381	65	23	18	4	5	3	0	1	0	4	0	3	2	0	1	3
Root (nj)	2187	490	133	121	37	51	22	19	7	8	6	7	6	4	5	5	18
Traffic (nj)	1317	239	42	14	4	4	1	0	0	0	0	0	0	0	0	0	0

In total, 513 species were observed in the microbial sample. The point estimate from N̂_pcg was 3005 at α̂ = 3 with 95% confidence interval (1691, 4530). The point estimate and confidence bounds from other estimators including N̂_wl, N̂_c1, N̂_c2, N̂_j, N̂_pcg2 and N̂_pcg3 are also presented in Table 4 for comparison. The estimator N̂_c3 gave all negative estimates, and was therefore excluded. Estimates from other estimators varied substantially. The point estimates from N̂_c1 and the order 5 jackknife estimator N̂_j were 1674 and 1913, respectively, and both are likely to be underestimates of N based on our experience gained from the simulation. It is possible that there are actually more than 3000 species, as indicated by the confidence limits from N̂_pcg.

Table 4.

Results for data in Table 3

Estimators	N̂pcg	N̂wl	N̂c1	N̂c2	N̂j	N̂pcg2	N̂pcg3
Microbial	3005 (1691, 4530)	2053 (1519, 2585)	1674 (1388, 2052)	2510 (1878, 3434)	1913 (1634, 2191)	– (521, 2272)	1098 (855, 2712)
Root	8980 (8383, 18 771)	8919 (8211, 10 918)	9090 (8470, 9783)	13 948 (12 459, 15 674)	9891 (9192, 10 589)	9483 –	9478 (6530, 18 033)
Traffic	6935 (5198, 12 409)	5496 (4991, 6918)	5684 (5031, 6461)	6788 (5665, 8223)	6588 (5892, 7284)	– –	– –

The point estimate and 95% confidence interval (parentheses) are presented for each dataset. N_wl was computed by using ESTstatCF program from http://bioinfo.stats.northwestern.edu/~jzwang. ‘–’ indicates that the estimates were extremely large due to boundary problems or possible bias.

For the expressed sequence tag data and traffic data, the estimates from N̂_pcg were 8980 (8383, 18 771) at α̂ = ∞ and 6935 (5198, 12 409) at α̂ = 3, respectively. In both the cases, the bootstrap estimates were highly skewed to the right. The large skewness should not be simply understood as the negative side of variability, but instead it may reflect an appealing feature in coverage. For example, the true N in the traffic data was known to be 9461, and N̂_pcg is the only estimator that provides a confidence interval that covers the true N. Similarly, for the root data, 8980 could be a conservative estimate of the total number of expressed genes in the root tissue. As the Arabidopsis thaliana genome contains approximately 27 000 protein coding genes, the N̂_pcg confidence limits suggest that there can be up to 2/3 of the entire transcriptome expressed in the root tissue.

6. Discussion

This estimation strategy can be potentially applied to many applications that involve mixture models. An immediate application is to the capture–recapture problem. Instead of using a finite mixture of binomials (Pledger, 2000), one could use a binomial-compound beta model (Morgan & Ridout, 2008). The crossvalidation procedure can be investigated further to select the number of components in finite mixture models. Detailed discussion can be found in the online Supplementary Material. The R package (R Development Core Team, 2010) species, which automates different estimators for species richness estimation, can be found at http://bioinfo.stats.northwestern.edu/~jzwang.

Supplementary material

Supplementary material is available at Biometrika online.

Appendix

Proof of Lemma 2. We show the existence of G for θ = α or β. The uniqueness is guaranteed by the identifiability of gamma mixtures under a unified α or β parameter, stated in Theorem 2.

When θ = α, given any δ > 0 and μ₀ > 0, the following G satisfies ∫h(λ ; α + δ, μ)dG(μ) = h(λ; α, μ₀), for all λ > 0:

$$dG(\mu )=\frac{\alpha +\delta }{{\mu }^2}\frac{\mathrm{\Gamma }(\alpha +\delta ){\left(\alpha /{\mu }_0\right)}^{\alpha }}{\mathrm{\Gamma }(\alpha )\mathrm{\Gamma }(\delta )}{\left(\frac{\alpha +\delta }{\mu }\right)}^{-\alpha -\delta }{\left(\frac{\alpha +\delta }{\mu }-\frac{\alpha }{{\mu }_0}\right)}^{\delta -1}I\left(\frac{\alpha +\delta }{\mu }>\frac{\alpha }{{\mu }_0}\right)d\mu .$$

It is straightforward to verify that G(μ) is a well-defined probability distribution function on {0, (α + δ)μ₀/α}.

When θ = β, if such a G(μ) exists, it has to satisfy that for all λ > 0,

$$\int \frac{{\left(\beta +\delta \right)}^{(\beta +\delta )\mu }}{\mathrm{\Gamma }\left\{(\beta +\delta )\mu \right\}}{\lambda }^{\left(\beta +\delta \right)\mu -\beta {\mu }_0}dG(\mu )=\frac{{\beta }^{\beta {\mu }_0}}{\mathrm{\Gamma }(\beta {\mu }_0)}{e}^{\delta \lambda }.$$ (A1)

We can define G(μ) as a discrete probability measure as such that

$$P_G\{(\beta +\delta )\mu =\beta {\mu }_0+i\}=\frac{{\beta }^{\beta {\mu }_0}}{\mathrm{\Gamma }(\beta {\mu }_0)}\frac{\mathrm{\Gamma }(\beta {\mu }_0+i)}{{\left(\beta +\delta \right)}^{\beta {\mu }_0+i}}\frac{{\delta }^i}{i!}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\left(i=0,1,\dots \right).$$ (A2)

To verify whether (A2) is a probability measure, consider a Po(x; δλ), where λ has a prior distribution of Ga(βμ₀, β). Then

$$\begin{array}{lll}\text{pr}(X=i)\hfill & =\hfill & \int \frac{e^{-\delta \lambda }{\left(\delta \lambda \right)}^i}{i!}\frac{{\beta }^{\beta {\mu }_0}}{\mathrm{\Gamma }(\beta {\mu }_0)}{\lambda }^{\beta {\mu }_0-1}{e}^{-\beta \lambda }d\lambda \hfill \\ \hfill & =\hfill & \frac{{\beta }^{\beta {\mu }_0}}{\mathrm{\Gamma }(\beta {\mu }_0)}\frac{\mathrm{\Gamma }(\beta {\mu }_0+i)}{{\left(\beta +\delta \right)}^{\beta {\mu }_0+i}}\frac{{\delta }^i}{i!}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\left(i=0,1,\dots \right).\hfill \end{array}$$

This is exactly in the same form as P_G. Plugging G into the left-hand side of the equality (A1), we have

$$\frac{{\beta }^{\beta {\mu }_0}}{\mathrm{\Gamma }(\beta {\mu }_0)}\sum _{i\ge 0}\frac{{\left(\delta \lambda \right)}^i}{i!}=\frac{{\beta }^{\beta {\mu }_0}}{\mathrm{\Gamma }(\beta {\mu }_0)}{e}^{\delta \lambda }.$$

Proof of Theorem 2. Suppose that under the reparameterization (α, μ) setting, the mixing distribution in μ is G(μ). As α is fixed, G(μ) corresponds to a unique mixing distribution in β, e.g. G^*(β), in the original (α, β) parameterization and G^*(β) = 1 − G{(α/β)⁻}, where G{(α/β)⁻} = P(μ < α/β). Conversely, we have G(μ) = 1 − G^*{(α/β)⁻}. As G(μ) and G^*(β) are uniquely determined by each other, it suffices to show that under the original parameterization (α, β), G^*(β) is identifiable. Suppose there is another G^** such that for all λ > 0,

$$\int \frac{{\beta }^{\alpha }}{\mathrm{\Gamma }(\alpha )}{\lambda }^{\alpha -1}{e}^{-\beta \lambda }d{G}^{*}(\beta )\equiv \int \frac{{\beta }^{\alpha }}{\mathrm{\Gamma }(\alpha )}{\lambda }^{\alpha -1}{e}^{-\beta \lambda }d{G}^{**}(\beta ).$$ (A3)

This equality holds at λ = 1, and hence

$$\int \frac{{\beta }^{\alpha }}{\mathrm{\Gamma }(\alpha )}{e}^{-\beta }d{G}^{*}(\beta )\equiv \int \frac{{\beta }^{\alpha }}{\mathrm{\Gamma }(\alpha )}{e}^{-\beta }d{G}^{**}(\beta ).$$ (A4)

The left- and right-hand sides of equation (A3) divided by those of (A4) in parallel gives

$$\int e^{-\beta (\lambda -1)}d{P}^{*}(\beta )\equiv \int e^{-\beta (\lambda -1)}d{P}^{**}(\beta ),$$ (A5)

where

$$d{P}^{*}(\beta )=\frac{{\beta }^{\alpha }{e}^{-\beta }d{G}^{*}(\beta )}{\int {\beta }^{\alpha }{e}^{-\beta }d{G}^{*}(\beta )},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}d{P}^{**}(\beta )=\frac{{\beta }^{\alpha }{e}^{-\beta }d{G}^{**}(\beta )}{\int {\beta }^{\alpha }{e}^{-\beta }d{G}^{**}(\beta )}.$$

If we let t = 1 − λ, then the left- and right-hand sides of (A5) are the moment generating functions for P^* and P^**, respectively, which are finite for t in a neighbourhood around zero, and hence P^*=P^**. As dG^*(β) = β^−αe^β dP^*(β)/{∫ β^−αe^β dP^*(β)}, dG^**(β) = β^−αe^β dP^**(β)/{∫ β^−αe^β dP^**(β)}, and therefore G^* = G^**. Hence G(μ) is identifiable in 𝒢.

Now consider the (β, μ) setting. Suppose the mixing distribution has a cumulative distribution function G(μ), and there exists another G^* such that for all λ > 0

$$\int \frac{{\beta }^{\beta \mu }}{\mathrm{\Gamma }(\beta \mu )}{\lambda }^{\beta \mu -1}{e}^{-\beta \lambda }dG(\mu )\equiv \int \frac{{\beta }^{\beta \mu }}{\mathrm{\Gamma }(\beta \mu )}{\lambda }^{\beta \mu -1}{e}^{-\beta \lambda }d{G}^{*}(\mu ).$$ (A6)

This equality holds at λ = 1, and hence

$$\int \frac{{\beta }^{\beta \mu }}{\mathrm{\Gamma }(\beta \mu )}dG(\mu )\equiv \int \frac{{\beta }^{\beta \mu }}{\mathrm{\Gamma }(\beta \mu )}d{G}^{*}(\mu ).$$ (A7)

The left- and right-hand sides of equation (A6) divided by those of (A7) in parallel gives

$$\int {\lambda }^{\beta \mu }dP(\mu )\equiv \int {\lambda }^{\beta \mu }d{P}^{*}(\mu ),$$ (A8)

where

$$dP(\mu )=\frac{{\beta }^{\beta \mu }{\left\{\mathrm{\Gamma }(\beta \mu )\right\}}^{-1}dG(\mu )}{\int {\beta }^{\beta \mu }{\left\{\mathrm{\Gamma }(\beta \mu )\right\}}^{-1}dG(\mu )}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{and}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}d{P}^{*}(\mu )=\frac{{\beta }^{\beta \mu }{\left\{\mathrm{\Gamma }(\beta \mu )\right\}}^{-1}d{G}^{*}(\beta )}{\int {\beta }^{\beta \mu }{\left\{\mathrm{\Gamma }(\beta \mu )\right\}}^{-1}d{G}^{*}(\mu )}.$$

Let t = β log(λ). The two sides of equation (A8) become the moment generating functions of P and P^*, respectively, and is bounded when λ is around 1, or t around 0. Thus, based on by the same argument as above, G(μ) is identifiable in 𝒢.