Population Size Estimation using Zero-truncated Poisson Regression with Measurement Error

SUMMARY

Population size estimation is an important research field in biological sciences. In practice, covariates are often measured upon capture on individuals sampled from the population. However, some biological measurements, such as body weight may vary over time within a subject’s capture history. This can be treated as a population size estimation problem in the presence of covariate measurement error. We show that if the unobserved true covariate and measurement error are both normally distributed, then a naïve estimator without taking into account measurement error will under-estimate the population size. We then develop new methods to correct for the effect of measurement errors. In particular, we present a conditional score and a nonparametric corrected score approach that are both consistent for population size estimation. Importantly, the proposed approaches do not require the distribution assumption on the true covariates, furthermore the latter does not require normality assumptions on the measurement errors. This is highly relevant in biological applications, as the distribution of covariates is often non-normal or unknown. We investigate finite sample performance of the new estimators via extensive simulated studies. The methods are applied to real data from a capture–recapture study.

1. Introduction

Population size estimation is an important problem in applied research such as ecology, epidemiology and bioinformatics. To estimate the unknown size of a population, capture–recapture experiments conducted on individuals in the population are often carried out. On capture, previously uncaptured individuals are marked and the marks of previously captured individuals are noted. The resulting data, often known as capture history data, are then used for fitting capture–recapture models. Individual covariates can also be collected upon capture and used in the analysis. We are motivated by a capture–recapture study conducted at the Mai Po Nature Reserve in Hong Kong on Yellow-bellied prinia (Prinia flaviventris), which is a resident bird species in Hong Kong, and also found in Pakistan, the southern Himalayan foothills, the northeastern Indian subcontinent, and Southeast Asia. In this study, 165 distinct birds were captured out of 206 captures during the non-breeding season in 1993. In addition to the capture histories, the data set includes two covariates, wing length, and fat index, that may be related to the capture probabilities. Fat index was recorded to classify the condition of total body fat which has four levels coded from 1 to 4. However, measurements of wing length may be error-prone. Estimation of the population size without taking into account covariate measurement error may be biased.

A number of different capture–recapture methods that incorporate information in the covariates have been developed in the literature; see Huggins (1989), Chao (2001); McCrea and Morgan (2015) and Liu, Li and Qin (2017). It is well-known in the capture–recapture literature that incorporating covariates in a capture–recapture analysis can enhance inference in population size estimation (Pollock, 2002). Because the data are usually collected over a fixed number of capture occasions, traditional capture–capture methods are primarily based on zero-truncated binomial regression models. For each occasion, some individuals are captured, marked (for those first captured), recorded and released back into the population. The resulting capture history of an observed individual is a vector of binary outcomes, but cannot be a zero vector. Therefore, a zero-truncated binomial type model is naturally one of the best models to fit these data.

In this study, we consider zero-truncated Poisson regression models (Van der Heijden, Cruyff and Van Houwelinge, 2003) to deal with a different type of capture–recapture data set as follows. For some animal species, such as sperm whales and grizzly bears (Wilson and Anderson, 1995), individual capture histories consist of capture (sighting) times during an observation period. The collection process for these types of data is a realization of a counting process, and is often called a continuous-time capture–recapture model. In fact, the Poisson process is the most popular counting process used for continuous-time capture–recapture models. Under a Poisson process framework, the number of captures for each individual contains all information about the inference on the population size (Schofield et al., 2018; Liu et al., 2018; Zhang and Bonner, 2020), while event times are only useful for inference on capture intensities. It follows that the number of captures is Poisson distributed and hence, a zero-truncated Poisson regression model is more suitable in continuous-time capture–recapture models.

It is well-known that measurement error in the covariates can lead to inconsistent estimates in regression analysis (Carroll et al., 2006). For example, in simple linear regression, inconsistent estimates arise due to attenuation effects caused by measurement errors. In capture–recapture studies, population size estimates will generally yield a negative bias when measurement errors are ignored. To the best of our knowledge, while effects of measurement error in population size estimation have been observed in the capture–recapture literature, there does not exist a sensible explanation of this finding. Nevertheless, various capture–recapture methods have been developed to correct for measurement error in the covariates. These include (approximate) regression calibration (Hwang and Huang, 2003), conditional score estimation (Hwang et al., 2007), parametric likelihood (Yip et al., 2005; Xi et al., 2009; Stoklosa et al., 2011), estimating equations (Huggins and Hwang, 2010), and a semi-efficient method with multiple measurements (Xu and Ma, 2014). These methods all assume the underlying model is binomial for each subject’s capture frequency. Note that Yip et al. (2005) and Xi et al. (2009) require a parametric distributional assumption on the true underlying covariate, and Xi et al. (2009) specify the parametric distributions of the unobserved covariates and measurement errors where an expectation maximization algorithm is used to find the maximum likelihood estimates. However, because the unobserved covariates are latent variables, the expectation step involves the evaluation of many numerical integrals which incurs significant computational cost, and it may be sensitive to distributional assumptions. Interestingly, very limited literature has been developed for Poisson regression, with the exception of Hwang and Huang (2007) who considered a regression calibration approach and conditional score estimation for Poisson capture–recapture modeling.

Over the last few decades, the majority of newly developed measurement error methods require the assumption of a “surrogate condition” for correcting the effects of measurement errors. Specifically, this assumption holds if measurement errors are independent of the response and other variables. However, in regression models for zero-truncated count data, such as capture–recapture data, the number of covariate measurements for a given subject may be equal to the actual count response. This implies that the measurement error variance is related to the response variable and hence the surrogate condition is not satisfied. This is seen in the Prinia flaviventris capture–recapture data where the total number of observed wing length measurements per individual is equal to the number of times the individual is captured, thus the surrogate condition is invalid when fitting measurement error models.

To address the problem of multiple measurements within a subject, Huang et al. (2011) proposed a modified conditional score approach to estimate model parameters in zero-truncated binomial and Poisson regression models. Their estimation methods were shown to be consistent, but they were computationally demanding and they did not consider population size estimation. Furthermore, these studies require a normality assumption on the measurement error, which may be inappropriate in some circumstances. In biological applications, the distribution of the observed covariates is often non-normal or unknown, so relaxing this assumption is beneficial.

In this paper, we begin by investigating the effects of measurement errors when estimating the population size. For multiple measurements within a subject, we first develop an average conditional score estimating function (Stefanski and Carroll, 1987) to estimate regression parameters and propose a corresponding population size estimator. We then develop a corrected score approach (Nakamura, 1990). The corrected score can be viewed as an unbiased estimate of the original score function where the true covariate is known. Unfortunately, in standard zero-truncated models, such as zero-truncated Poisson and binomial models, an unbiased estimate for the original score function does not exist (Stefanski, 1989). For other truncated models, the applicability of the corrected score approach remains an open research question. Here, we propose an estimating function that is correctable in the presence of measurement error. We also derive a new corrected score type estimating function with or without a parametric distributional assumption on the measurement errors. This allows us to develop consistent population size estimators. To our knowledge, this is the first time that the corrected score method has been developed for zero-truncated regression and population size estimation.

In Section 2 we introduce some notation and explain why an unadjusted population size estimate would typically yield a downward bias in the presence of measurement error. In Section 3, we develop a conditional score estimator that does not need a distributional assumption on the covariates. However, the conditional score estimator needs a normality assumption on the measurement error. To further relax this assumption, we propose a corrected score method in Section 4. A simulation study is carried out in Section 5 in which all methods are compared under various scenarios. In Section 6 we apply the methods to the Prinia flaviventris data mentioned above. Discussion and concluding remarks are given in Section 7.

2. Models and effects of measurement error

Let i = 1, …, N denote the index of an animal within a closed population where N is the unknown population size. For i = 1, …, N, let Y_i be the number of captures for the ith animal from a capture–recapture experiment. Without loss of generality, Y_i is positive for i ≤ n, in which n is the number of distinct observed/captured animals in the experiment. Further, let X_i and Z_i represent covariates for the ith subject where X_i is a univariate variable and Z_i is a vector. Suppose that Z_i is a vector of 1 and any covariates that are recorded precisely (such as sex or age group). We assume that all the covariates do not vary over time; this assumption is reasonable under a closed population setting. Instead of observing X_i, we observe the error-prone measurements W_ij for some j = 1, …, m_i where m_i is the number of measurements for the ith subject. We consider an additive measurement error model:

$$W_{ij}=X_i+U_{ij},\text{for}i=1,\dots ,N,\text{and}j=1,\dots ,m_i,$$ (1)

where U_ij is the measurement error which is independent of all other variables. We also assume that U_ij are symmetric identically distributed with zero mean for all i and j. Let ${\overline{W}}_i={\sum }_{j=1}^{m_i}{W}_{ij}/m_i$ be the surrogate of X_i, and write ${\overline{W}}_i=X_i+{\overline{U}}_i$ where ${\overline{U}}_i={\sum }_{j=1}^{m_i}{U}_{ij}/m_i$. In practice, the number of measurements m_i may be related to Y_i, such that the usual surrogate condition where the measurement error is independent of the response variable (Carroll et al., 2006) does not hold (i.e., ${\overline{U}}_i$ is not independent of Y_i). Generally speaking, m_i is equal to the number of captures Y_i.

Denote $\theta ={({\alpha }^{\mathrm{\top }},\beta )}^{\mathrm{\top }}$, which is a vector of parameters. We consider Y_i as a Pois(λ_i) distributed random variable with mean ${\lambda }_i(\theta )=\exp ({\alpha }^{\mathrm{\top }}{Z}_i+\beta X_i)$ for all i = 1, …, N. Since we can only observe individual covariates on captured animals, or equivalently, given the condition Y_i > 0, these Y_i are zero-truncated variables and have the following probability distribution:

$${\mathbb{P}}_{\theta }(Y_i=k\mid Z_i,X_i)=\frac{1}{{\pi }_{i}k!}\exp \{k({\alpha }^{\mathrm{\top }}{Z}_i+\beta X_i)-{\lambda }_i\},k=1,2,\dots$$

where ${\pi }_i=1-\exp (-{\lambda }_i)$ which is the probability of the ith individual being captured at least once. If the X_i’s are available, the log-likelihood function of θ is $\ell (\theta )={\sum }_{i=1}^n\{Y_i({\alpha }^{\mathrm{\top }}{Z}_i+\beta X_i)-{\lambda }_i-\ln ({\pi }_i)\}$. Hence, the score function of θ is

$$G_{\mathrm{x}}(\theta )=\sum _{i=1}^n\left(\begin{array}{l}\hfill Z_i\hfill \\ X_i\hfill \end{array}\right)\left(Y_i-\frac{{\text{λ}}_i}{{\pi }_i}\right),$$ (2)

and the maximum likelihood estimator $\widehat{\theta }$ solves $G_{\text{x}}(\theta )=0$. Let ${\widehat{\mathrm{\lambda }}}_i$ and ${\widehat{\pi }}_i$ be the corresponding estimators of λ_i(Θ) and π_i(θ) with $\widehat{\theta }$ substituted for θ. To estimate the population size N, the following two weighted estimators can be used:

$${\widehat{N}}_1=\sum _{i=1}^n\frac{1}{{\widehat{\pi }}_i}\text{and}{\widehat{N}}_2=\sum _{i=1}^n\frac{Y_i}{{\widehat{\mathrm{\lambda }}}_i}.$$

In the context of capture-recapture modeling, ${\widehat{N}}_1$ is generally referred to as the Horvitz-Thompson estimator because the denominator is the probability that the unit is ever selected in the sampling process (Alho et al., 1993). Whereas ${\widehat{N}}_2$ is a Hansen-Hurwitz estimator since the denominator is the expected selection on one occasion, and it is summed over all selections (Hansen and Hurwitz, 1943). Both population size estimators are consistent and asymptotically normally distributed, and they have similar finite sample performance.

We define a naïve score function G_w(θ) by replacing X_i with ${\overline{W}}_i$ in (2). A naïve estimator of θ, $\tilde{\theta }$, is then established via solving G_w(θ) = 0, and we can also similarly obtain the corresponding population size estimators. These types of naïve population size estimators generally exhibit a downward bias, as seen in the simulation studies of Hwang and Huang (2007). This kind of downward bias is also usually seen in the other measurement error capture–recapture models (Hwang and Huang, 2003; Xi et al., 2009), an explanation for this observation is that the contamination, due to measurement error, reduces the degree of heterogeneity in capture-recapture models. For example, suppose that ${\mathrm{\lambda }}_i(\theta )=\exp (\alpha +\beta X_i)$ and ${\sigma }_u^2$ is very large such that W_i cannot provide information for modeling heterogeneity in capture probabilities. As a consequence, the effect of the covariate would be excluded from the model, thus the data would be analyzed via a simple homogeneous capture-recapture model where λ_i is a constant. It can be shown that this kind of model misspecification yields a downward bias in estimating the population size (Hwang and Huggins, 2005).

In Web Appendix A, we provide a theoretical justification for this phenomenon when both X_i and U_i are normally distributed under the Poisson regression framework. Interestingly, if we consider ordinary Poisson regression with normality assumptions on X_i and U_i, then the naïve estimator for N is consistent even though the naïve estimator for θ is not. Specifically, it estimates the covariate effect of W_i through $\mathbb{E}(Y_i\mid W_i)$, which is not the true β in $\mathbb{E}(Y_i\mid X_i)$; see Web Appendix A. In particular, the resulting estimator of $\mathbb{E}(Y_i\mid W_i)$ is consistent and can be used as a surrogate for λ_i when being used in the population size estimators stated earlier. However, this argument is incorrect in our problem because under this situation, Y_i is drawn from a zero-truncated Poisson model. Moreover, we can show that the naïve method generally overestimates the average of λ_i which implies that the naïve population size estimators would generally result in a downward bias. See Proposition 1 in Web Appendix A.

3. Conditional score estimation

In this section, we develop a conditional score (CDS) estimator that does not need a distributional assumption on the latent variables X_i. Define ${\text{Δ}}_{ij}=Y_i\beta {\sigma }_u^2+W_{ij}$ for all i = 1, …, N and j = 1, …, m_i. Following Stefanski and Carroll (1987) and Hwang et al. (2007), it can be showed that Δ_ij is a sufficient and complete statistic for X_i if β is known. By sufficiency, the conditional distribution of Y_i given Δ_ij, Z_i is independent of X_i and can be shown to be

$$\mathbb{P}(Y_i=k\mid Z_i,{\text{Δ}}_{ij})=\frac{1}{k!{\mathcal{T}}_{ij}}\exp \left\{k({\alpha }^{\mathrm{\top }}{Z}_i+{\beta }^{\prime }{\text{Δ}}_{ij})-\frac{1}{2}{k}^2{\beta }^2{\sigma }_u^2\right\},$$ (3)

for k = 0, 1, 2, …, where ${\mathcal{T}}_{ij}=\sum _{k=0}^{\infty }\exp \{k({\alpha }^{\mathrm{\top }}{Z}_i+{\beta }^{\prime }{\text{Δ}}_{ij})-k^2{\beta }^2{\sigma }_u^2/2\}/k!$.

For capture–recapture data, we observe the data up to the individuals i ≤ n. Hence, for all i ≤ n and k = 1, 2, 3, …, we have

$$\mathbb{P}(Y_i=k\mid {\text{Δ}}_{ij},Z_i,Y_i\ge 1)=\frac{1}{k!({\mathcal{T}}_{ij}-1)}\exp \left\{k({\alpha }^{\mathrm{\top }}{Z}_i+{\beta }^{\prime }{\text{Δ}}_{ij})-\frac{1}{2}{k}^2{\beta }^2{\sigma }_u^2\right\}.$$

By plugging ${\widehat{\sigma }}_u^2$ into Δ_ij, the conditional score estimator for θ, denoted as ${\widehat{\theta }}_c$, solves

$${\text{Φ}}_{\text{c}}(\theta )=\sum _{i=1}^n{\varphi }_{\text{c}}(\theta ;Y_i,Z_i,{\text{Δ}}_{i.})=0,$$ (4)

where ϕ_c(θ; Y_i, Z_i, Δ_i) is the average of ${(Z_i^{\mathrm{\top }},{\text{Δ}}_{ij})}^{\mathrm{\top }}\{Y_i-\mathbb{E}(Y_i\mid {\text{Δ}}_{ij},Z_i,Y_i\ge 1)\}$ for j = 1, …, m_i. In order to implement the above CDS approach, ${\mathcal{T}}_{\mathit{ij}}$ and $\mathbb{E}(Y_i\mid {\text{Δ}}_{ij},Z_i,Y_i\ge 1)$ can be calculated numerically. Since both terms involve an infinite summation and they do not have closed forms, we need to approximate these terms using finite sums with upper-bound restrictions. Due to the variance of a zero-truncated Poisson random variable being smaller than its expectation, we adopt a simple strategy that uses the upper bound of $Y_i+5\sqrt{Y_i}$ (Hwang et al., 2007).

Let ϕ_c,i denote ϕ_c(θ; Y_i, Z_i, Δ_i.) and notice that ϕ_c,i are independent over all i. This implies that (4) is a classical unbiased estimating equation, thus a sandwich variance estimate of ${\widehat{\theta }}_{\text{c}}$ can be applied. Specifically, the variance estimator is

$${\left\{\frac{\partial {\text{Φ}}_{\text{c}}}{\partial \theta }\right\}}^{-1}\left(\sum _{i=1}^n{\varphi }_{\text{c},i}{\varphi }_{\text{c},i}^{\mathrm{\top }}\right){\left\{\frac{\partial {\text{Φ}}_{\text{c}}(\theta )}{\partial \theta }\right\}}^{-\mathrm{\top }},$$ (5)

where θ is evaluated at ${\widehat{\theta }}_{\text{c}}$. Note that the above variance estimator does not account for the variability from estimating ${\sigma }_u^2$. Nevertheless, from our simulation study, we found that variability due to estimating ${\sigma }_u^2$ is generally negligible, and it has almost no effect on the variance estimation.

For population size estimation, let ${\pi }_{\text{c},i}(\theta )=\sum _{j=1}^{m_i}\mathbb{P}(Y_i\ge 1\mid {\text{Δ}}_{ij},Z_i)/m_i$ where $\mathbb{P}(Y_i\ge 1\mid {\text{Δ}}_{ij},Z_i)=1-1/{\mathcal{T}}_{\mathit{ij}}$. We then mimic the from of ${\widehat{N}}_1$ and suggest to estimate N by

$${\widehat{N}}_{\text{c}}=\sum _{i=1}^n\frac{1}{{\pi }_{\text{c},i}({\widehat{\theta }}_{\text{c}})}.$$

To estimate the variance of ${\widehat{N}}_c$, by the delta method, we have

$$\mathrm{Var}({\widehat{N}}_{\text{c}})\approx \sum _{i=1}^n\frac{1-{\pi }_{\text{c},i}(\theta )}{{\pi }_{\text{c},i}^2(\theta )}+\frac{\partial {\widehat{N}}_{\text{c}}^{\mathrm{\top }}}{\partial \theta }\mathrm{Var}({\widehat{\theta }}_{\text{c}})\frac{\partial {\widehat{N}}_{\text{c}}}{\partial \theta }$$ (6)

and this yields a variance estimator for ${\widehat{N}}_c$ when all parameters are evaluated at ${\widehat{\theta }}_{\mathrm{c}}$.

Remark 1: An alternative approach is to construct an estimating function based on ${\overline{\mathrm{\Delta }}}_i={\sum }_{j=1}^{m_i}{\mathrm{\Delta }}_{ij}/m_i$. In fact, the use of ${\overline{\mathrm{\Delta }}}_i$ could be more efficient, but it is valid only if m_i are independent with other variables. Huang et al. (2011) called this method a naïve conditional score when m_i are correlated with Y_i (e.g., m_i = Y_i in the motivated example). They also proposed a Monte Carlo error augmentation conditional score method to solve this problem, but it is computational time consuming. Although not reported here, we found in simulation studies that the current proposed method performed equally well, and in some cases was more robust than the error augmentation conditional score.

4. Corrected score estimation

In generalized linear models with covariate measurement error, the corrected score approach of Nakamura (1990) also does not require specifying the underlying distribution of X_i. As shown in Stefanski (1989), a corrected score function with respect to the score function of a zero-truncated Poisson measurement error model does not exist. In the following, we consider an estimating function which is correctable when X_i are contaminated with measurement errors.

4.1. Parametric corrected score

Let ${\tilde{Y}}_i=Y_i-1$ and ${\tilde{T}}_i=Y_i/(Y_i+1)$, then we have the identity

$$\mathbb{E}({\tilde{Y}}_i\mid Z_i,X_i,Y_i>0)=\frac{{\mathrm{\lambda }}_i-{\pi }_i}{{\pi }_i}=\mathbb{E}({\tilde{T}}_i{\mathrm{\lambda }}_i\mid Z_i,X_i,Y_i>0).$$ (7)

To show this, we note that the first identity is trivial since $\mathbb{E}(Y_i\mid Z_i,X_i,Y_i>0)={\mathrm{\lambda }}_i/{\mathrm{\pi }}_i$. For the second identity, we have

$$\mathbb{E}\left(\frac{1}{Y_i+1}\mid Z_i,X_i,Y_i>0\right)=\frac{e^{-{\mathrm{\lambda }}_i}}{{\pi }_i}\sum _{k=1}^{\infty }\frac{{\mathrm{\lambda }}_i^k}{(k+1)!}=\frac{{\pi }_i-{\mathrm{\lambda }}_i{e}^{-{\mathrm{\lambda }}_i}}{{\mathrm{\lambda }}_i{\pi }_i}.$$

Hence,

$$\mathbb{E}({\tilde{T}}_i{\mathrm{\lambda }}_i\mid Z_i,X_i,Y_i>0)={\mathrm{\lambda }}_i\left\{1-\mathbb{E}\left(\frac{1}{Y_i+1}\mid Z_i,X_i,Y_i>0\right)\right\}=\frac{{\mathrm{\lambda }}_i}{{\pi }_i}-1$$

which proves the identity above. According to (7), when the X_is are available, we may estimate θ through the estimating equation

$$\mathcal{S}(\theta )=\sum _{i=1}^{n}S(\theta ;Y_i,Z_i,X_i)=\sum _{i=1}^n\left(\begin{array}{c}\hfill Z_i\hfill \\ \hfill X_i\hfill \end{array}\right)\left({\tilde{Y}}_i-{\tilde{T}}_i{\mathrm{\lambda }}_i\right)=0.$$ (8)

Remark 2: The estimating function in (8) is also considered a “modified score function” since it is the score function of a weighted partial likelihood proposed by Hwang et al. (2019). Here, we provide a simple justification for this result. Hwang et al. (2019) showed that the resulting estimator of (8) performed well, as it generally retained high efficiency in comparison with the maximum likelihood estimator. As shown in Hwang et al. (2019), this loss in efficiency is quite small; the resulting estimator of (8) will attain efficiency of at least 98% in contrast to the maximum likelihood estimator.

The estimating function is biased if we solve ${\mathcal{S}}_{\mathrm{w}}(\theta )=0$ where ${\mathcal{S}}_{\text{w}}(\theta )=\sum _{i=1}^{n}S(\theta ;Y_i,Z_i,{\overline{W}}_i)$. Now, let K_i(s) be the cumulant generating function of ${\overline{U}}_i$, that is, $K_i(s)=\ln [\mathbb{E}\{\exp (s{\overline{U}}_i)\}]$ whenever the expectation exists. Let $K_i^{\prime }(s)$ be the first derivative of K_i(s). We follow Nakamura (1990) and propose a corrected estimating function of S(θ), that is,

$${\text{Φ}}_{\text{p}}(\theta )=\sum _{i=1}^n{\varphi }_{\text{p}}(\theta ;Y_i,Z_i,{\overline{W}}_i),$$ (9)

where

$${\varphi }_{\text{p}}(\theta ;Y_i,Z_i,{\overline{W}}_i)={\tilde{Y}}_i\left(\begin{array}{c}\hfill Z_i\hfill \\ \hfill {\overline{W}}_i\hfill \end{array}\right)-{\tilde{t}}_i^{\ast }\exp \{{\alpha }^{\mathrm{\top }}{Z}_i+\beta {\overline{W}}_i-K_i(\beta )\}\left(\begin{array}{c}\hfill Z_i\hfill \\ \hfill {\overline{W}}_i-K_i^{\prime }(\beta )\hfill \end{array}\right).$$

It can be seen that $\mathbb{E}\{{\varphi }_{\text{p}}(\theta ;Y,Z,\overline{W})\mid Y,Z,X\}=S(\theta ;Y,Z,X)$. Therefore Φ_p(θ) = 0 is an unbiased estimating equation. A parametric corrected score (PCS) estimator can be obtained by assuming a known distribution for the measurement error (but with an unknown variance). A normality assumption is perhaps the most commonly used in practice. If we assume that $U_{ij}\sim N(0,{\sigma }_u^2)$, then $K_i(s)=s^2{\sigma }_u^2/(2m_i)$, and $K_i^{\prime }(s)=s{\sigma }_u^2/m_i$, where ${\sigma }_u^2$ can be estimated from the sample pooled variance of W_ij, denoted here as ${\widehat{\sigma }}_u^2$. Alternatively, if for example we assume that U_ij ~ Unif(−δ, δ), then $K_i(s)=m_i\log [m_i\{\exp (\delta s/m_i)-\exp (-\delta s/m_i)\}/2\delta s]$ where δ can be estimated by the moment method, that is, $\widehat{\delta }=\sqrt{3}{\widehat{\sigma }}_u$.

Denote the resulting PCS estimator by ${\widehat{\theta }}_{\text{p}}$ that solves Φ_p(θ) = 0 where K_i(β) and $K_i^{\prime }(\beta )$ are substituted by their corresponding estimates. Like the CDS approach, Φ_p(θ) is a summation of unbiased estimating functions, so a sandwich variance estimator of ${\widehat{\theta }}_{\text{p}}$ can be obtained with the same form as (5) but we replace the estimating functions ϕ_c,i with ϕ_p,i where ${\varphi }_{\text{p},i}={\varphi }_{\text{p}}(\theta ;Y_i,Z_i,{\overline{W}}_i)$.

To estimate the population size, write ${\widehat{\theta }}_{\text{p}}={({\widehat{\alpha }}_{\text{p}}^{\text{T}},{\widehat{\beta }}_{\text{p}})}^{\mathrm{\top }}$ and ${\widehat{\mathrm{\lambda }}}_{\text{p},i}=\exp \left\{{\widehat{\alpha }}_{\text{p}}^{\mathrm{\top }}{Z}_i+{\widehat{\beta }}_{\text{p}}{\overline{W}}_i+K_i({\widehat{\beta }}_{\text{p}})\right\}$. We propose the following corrected population size estimator

$${\widehat{N}}_{\text{p}}=\sum _{i=1}^n\frac{Y_i}{{\widehat{\mathrm{\lambda }}}_{\text{p},i}}.$$ (10)

Clearly, ${\widehat{N}}_{\text{p}}$ is reduced to ${\widehat{N}}_2$ when there is no measurement error and $\mathbb{E}\{{\widehat{N}}_{\text{p}}\}=N$ if ${\widehat{\theta }}_{\text{p}}=\theta$.

For the population size estimator ${\widehat{N}}_{\text{p}}$, by the delta method, we have

$$\mathrm{Var}({\widehat{N}}_{\text{p}})\approx \sum _{i=1}^n\frac{Y_i}{{\mathrm{\lambda }}_i^2}+\frac{\partial {\widehat{N}}_{\text{p}}^{\mathrm{\top }}}{\partial \theta }\mathrm{Var}({\widehat{\theta }}_{\text{p}})\frac{\partial {\widehat{N}}_{\text{p}}}{\partial \theta }+2\frac{\partial {\widehat{N}}_{\text{p}}^{\mathrm{\top }}}{\partial \theta }{\left\{\frac{\partial {\text{Φ}}_{\text{p}}(\theta )}{\partial \theta }\right\}}^{-1}\sum _{i=1}^n\frac{Y_i{\varphi }_{\text{p},i}}{{\mathrm{\lambda }}_i},$$ (11)

and this yields a variance estimator for ${\widehat{N}}_{\text{p}}$ when θ and λ_i are evaluated at ${\widehat{\theta }}_{\text{p}}$ as well as ${\widehat{\mathrm{\lambda }}}_{\text{p},i}$ respectively.

4.2. Nonparametric corrected score

The PCS estimator is restricted by an assumption on the measurement error distribution. We now propose a method to estimate K_i(β) and $K_i^{\prime }(\beta )$ in (9) and (10) nonparametrically, and it is referred to as the NPC method (Huang and Wang 2000, 2001). Here, the measurement errors U_ij are assumed to be independent and symmetric identically distributed random variables with the existence of moment generating functions. For i ≤ n, let j, ℓ ≤ m_i and j ≠ l, then $\mathbb{E}[\text{exp}\{t(W_{ij}-W_{i\ell })\}]={\{\mathbb{E}(t{U}_{i1})\}}^2$. Let

$$\mathcal{A}{(t)}^2=\frac{\sum _{i,m_i\ge 2}\sum _{j\ne \ell }\exp \{t(W_{ij}-W_{i\ell })\}}{\sum _{i,m_i\ge 2}{m}_i(m_i-1)}$$

We can then estimate $\exp \{K_i(\beta )\}$ by $\mathcal{A}{(\beta /m_i)}^{m_i}$ consistently. Similarly, the $K_i^{\prime }(\beta )$ can be estimated by ${\mathcal{A}}^{\prime }(\beta /m_i)/\mathcal{A}(\beta /m_i)=\mathcal{B}(\beta /m_i)/{\mathcal{A}}^2(\beta /m_i)$ where

$$\mathcal{B}(t)=\frac{1}{2}\frac{\sum _{i,m_i\ge 2}\sum _{j\ne \ell }(W_{ij}-W_{i\ell })\exp \{t(W_{ij}-W_{i\ell })\}}{\sum _{i,m_i\ge 2}{m}_i(m_i-1)}.$$

The NPC estimators can be obtained by substituting the resulting estimates of K_i(β) and $K_i^{\prime }(\beta )$ into (9) and (10), respectively. The corresponding variance estimators are similar to those of PCS, as given at the end of Section 4.1.

5. Simulations

We conducted a simulation study to compare the naïve, conditional score (CDS), and the nonparametric corrected score (NPC) estimators. The simulation study initially included the parametric corrected score (PCS) approach where the measurement error was assumed to follow a normal distribution. However, we do not present the results of PCS here because it performed worse than NPC and CDS in essentially all cases. In particular, the PCS was not as good as CDS or NPC even when the measurement error U_ij was normally distributed in which the assumption of PCS holds. A reason for this was that the numerical solution of PCS was less stable compared to NPC and CDS, such that the PCS estimator had a greater chance of yielding extremely large estimates. Moreover, the observed Fisher information matrix of PCS is unstable, which leads to large standard error and interval estimates. The numerical challenges of PCS were similarly observed in Huang (2014).

In the simulations below, we examined both normally and non-normally distributed measurement error structures with different magnitudes of measurement error variances. We also considered different distributions for the true covariates, different effect sizes, capture rates and population sizes.

We examined two cases for the true population size with N = 300 (moderate size), and N = 1000 (larger size), and considered two distributions for the true covariate X_i: the standard normal, and a location shifted, scaled chi-square distribution, where the latter is defined as $\{3-{\chi }^2(3)\}/\sqrt{6}$, and χ²(3) is a chi-square distributed random variable with 3 degrees of freedom. The expected number of times an individual was captured was set to λ_i = exp(α + βX_i), where α = −0.5 or 0.5 which corresponds to an average capture percentages 100 × (n/N) of approximately 45% and 80%, respectively, and β = 0.5 ln(2) ≈ 0.35 or ln(2) ≈ 0.69 (corresponding to moderate and large effect sizes). Thus, Y_i ~ Poiss(λ_i) for all i = 1, …, N. Let m_i = Y_i be the number of measurements for the ith individual, and let W_ij = X_i + U_i for all j = 1, …, m_i be the error-contaminated covariate where the measurement error is U_i. We generated U_i from two distributions: N(0, ${\sigma }_u^2$), and a symmetric uniform distribution with variance equal to ${\sigma }_u^2$.

For each combination of simulation settings, we generated 1,000 simulated data sets. Based on the 1,000 estimates, we calculated the “bias-%” as the median of relative bias (%), that is, we use $100\times \{(\widehat{\theta }-\theta )/\theta \}$; the median absolute deviations, denoted by “MAD”; the median of the standard error estimates, denoted by “Med.SE”, and the root mean square error denoted by “RMSE”. We also included 95% Wald-type confidence interval coverage probabilities (CP) for each estimator. Note that, MAD and Med.SE are robust measures of the standard deviation and the average of standard error estimates, respectively. We used these robust measures because we encountered several extreme estimates for each estimator. In the simulation, an estimation procedure was considered as divergence if its numerical solution did not converge, or the method resulted in a value of $\widehat{N}>10\times n$. Whenever divergence of this type had occurred, we used the naïve estimate instead. Overall, the naïve method did not encounter divergence and the divergence rate was less than 3% for CDS and NPC in all scenarios. Divergence could occur when α = −0.5, ${\sigma }_u^2=1$ or N = 300 but this had rarely occurred in all other cases.

In Tables 1–2, we report numerical summaries for estimates of β and N when ${\sigma }_u^2=1$. Specifically, in Table 1 we present results where X ~ N(0, 1) and U ~ N(0, 1), and in Table 2 we present results where X is non-normally distributed and U is uniformly distributed. As in many regression measurement error models, the naïve method showed an attenuation effect in estimating β in all cases, the relative bias was as large as −46% (Table 1) and −73% (Table 2). The naïve population size estimator also yielded a negative bias, as seen in Table 1. This agrees with our findings for the effects of measurement errors when X and U are normally distributed (see Section 2). Moreover, the downward bias of the naïve population size estimator was still apparent in Table 2 even though neither X nor U were normally distributed. Overall, the bias of the naïve population size estimator was not as severe as β where the worst case occurred around −13.1%.

Table 1:

Simulation study results for the naïve, CDS, and NPC methods with the population size N = 1000, the unobserved covariate X ~ N(0, 1) and the measurement error U ~ N(0, 1) for each considered parameter combination. We denote “bias-%” as the median of relative bias (%) – that is, 100 × {($\widehat{\theta }$ − θ)/θ}, “MAD” as the median absolute deviations, “Med.SE” as the median of the standard error estimates, “RMSE” as the root mean square error, and “CP” as the 95% confidence interval coverage percentages.

	β					N
	bias-%	MAD	Med.SE	RMSE	CP	bias-%	MAD	Med.SE	RMSE	CP
α = −0.5, β = 0.5ln(2)
Naïve	−45.1	0.050	0.049	0.167	10.8	−3.3	64.8	63.8	72.4	87.8
CDS	0.8	0.103	0.092	0.107	91.5	0.2	79.0	75.5	85.2	94.5
NPC	0.2	0.104	0.091	0.110	92.0	1.3	81.0	79.7	91.5	95.8
α = −0.5, β = 0.5ln(2)
Naïve	−38.0	0.027	0.025	0.136	0.1	−2.0	21.2	20.3	28.8	79.9
CDS	−0.4	0.041	0.040	0.041	95.0	−0.4	23.6	23.0	23.6	94.2
NPC	−0.0	0.042	0.040	0.042	93.7	0.1	24.4	23.8	24.2	94.7
α = −0.5, β = ln(2)
Naïve	−45.2	0.049	0.048	0.315	0.0	−12.8	57.0	54.6	136.1	39.7
CDS	−0.2	0.092	0.089	0.096	94.0	−1.3	92.4	90.4	104.2	93.6
NPC	0.2	0.103	0.091	0.117	92.3	0.5	106.3	98.4	129.0	95.6
α = −0.5, β = ln(2)
Naïve	−34.5	0.030	0.031	0.240	0.0	−5.3	25.0	23.9	57.7	42.1
CDS	−0.1	0.041	0.041	0.041	95.4	−0.4	31.0	31.2	32.6	93.5
NPC	0.1	0.044	0.041	0.046	91.7	−0.1	34.4	32.3	36.7	93.1

Table 2:

Simulation study results for the naïve, CDS, and NPC methods with the population size N = 1000, the unobserved covariate X is a location-shifted, scaled chi-square distribution and the measurement error U is uniformly distributed with variance ${\sigma }_u^2=1$ for each considered parameter combination. We denote “bias-%” as the median of relative bias (%) – that is, 100 × {($\widehat{\theta }$ − θ)/θ}, “MAD” as the median absolute deviations, “Med.SE” as the median of the standard error estimates, “RMSE” as the root mean square error, and “CP” as the 95% confidence interval coverage percentages.

	β					N
	bias-%	MAD	Med.SE	RMSE	CP	bias-%	MAD	Med.SE	RMSE	CP
α = −0.5, β = 0.5ln(2)
Naïve	−63.9	0.048	0.051	0.227	1.6	−3.8	63.0	60.4	73.4	86.1
CDS	5.7	0.208	0.185	0.249	92.7	1.4	99.2	88.3	199.9	95.1
NPC	−1.2	0.174	0.162	0.201	92.6	1.2	91.3	87.2	166.8	95.8
α = 0.5, β = 0.5ln(2)
Naïve	−53.9	0.026	0.026	0.190	0.0	−2.7	19.6	18.9	32.7	69.8
CDS	1.9	0.070	0.067	0.073	94.8	−0.1	26.7	25.1	27.6	93.9
NPC	−0.0	0.069	0.064	0.069	94.2	0.2	27.1	26.2	29.0	93.6
α = −0.5, β = ln(2)
Naïve	−72.0	0.047	0.047	0.500	0.0	−13.1	51.0	46.5	136.0	27.6
CDS	7.5	0.271	0.234	0.511	93.0	1.3	146.0	136.9	298.2	89.9
NPC	−0.1	0.255	0.222	0.294	91.2	0.2	142.6	131.6	353.1	91.8
α = 0.5, β = ln(2)
Naïve	−60.9	0.029	0.027	0.422	0.0	−8.7	18.8	15.9	87.9	0.7
CDS	1.0	0.090	0.090	0.096	94.0	−0.4	41.3	39.1	42.7	91.2
NPC	1.8	0.106	0.094	0.117	93.3	0.5	51.3	45.9	68.0	94.4

CDS performed very well for cases when U was normally distributed (see Table 1). However, when U was not normally distributed (Table 2), CDS may be biased for β whenever the capture probabilities were small (α = −0.5). Nevertheless, in most cases the CDS estimators for N were nearly unbiased. Moreover, CDS showed a certain degree of robustness in Table 2, this was most noticeable for α = 0.5 when the data was rich (larger n and $\overline{Y}$). Under many cases, the CDS estimator was better than the corrected score estimators in terms of RMSE. However, when the measurement error was uniformly distributed, it was slightly worse than NPC in terms of CP.

The nonparametric corrected score NPC is robust to distributional assumptions on U. Therefore, only NPC showed unbiasedness in all cases of Tables 1–2. Though not reported, NPC performed better than PCS, in particular, whenever U_ij was normally distributed. In Table 2 with α = −0.5, the MAD of NPC was smaller than CDS but this phenomena was generally reversed otherwise. We note that since NPC was constructed under a modified estimating function, standard errors (and MADs) were, in general, slightly larger compared to the maximum likelihood estimator when ${\sigma }_u^2=0$ (i.e., the reduced form of CDS), see Remark 2 in Section 4.

The proposed variance estimators of CDS and NPC worked reasonably well, as the resulting Med.SE were very close to MAD in most cases. We found that variability due to estimating ${\sigma }_u^2$ (or K(β)) is generally negligible, and it has almost no effect on the variance estimation.

In Figure 1, we plotted the bias-% against the measurement error variance ${\sigma }_u^2$ for the population size N where N = 1000. We have also included the results for the PCS method in these plots. For all cases, the naïve estimator for N exhibited a downward bias as ${\sigma }_u^2>0$, and the bias-% had significantly increased with increasing ${\sigma }_u^2$. The bias-% of CDS and NPC were generally quite low, with the exception of the case when α = 0.5, β = 0.69, and when X was non-normally distributed.

Figure 1:

Relative bias for N against increasing measurement error variances (${\sigma }_u^2$) when X ~ N(0, 1) (top) and X follows is a location-shifted, scaled chi-square distribution (bottom) for N = 1000. Sub-figures (a)–(d) correspond to the four effect size scenarios when U ~ N(0, ${\sigma }_u^2$), and sub-figures (e)–(h) correspond to the four effect size scenarios when U follows a symmetric uniform distribution. We denote “Relative Bias” as the median of relative bias (%) – that is, 100 × {($\widehat{\theta }$ − θ)/θ}.

In Figure 2 we plotted the CPs of N against ${\sigma }_u^2$ for N = 1000. The CPs for the naïve estimator were very poor for increasing ${\sigma }_u^2$, in some cases this was reported to be as low as 0. The CPs for the other methods were reasonable even though some CPs slightly decreased as ${\sigma }_u^2$ increased. Additional simulations with various sets of parameters are given in Web Tables 1–8 and Web Figures 1–8 of the Supplementary Materials.

Figure 2:

Coverage probabilities for N against increasing measurement error variances (${\sigma }_u^2$) when X ~ N(0, 1) (top) and X follows is a location-shifted, scaled chi-square distribution (bottom) for N = 1000. Sub-figures (a)–(d) correspond to the four effect size scenarios when U ~ N(0, ${\sigma }_u^2$), and sub-figures (e)–(h) correspond to the four effect size scenarios when U follows a symmetric uniform distribution.

Overall, the proposed correction methods had reduced bias and yielded accurate interval estimates. However, we also observed that both corrected methods yielded large population size estimates, especially when capture probabilities were low, combined with high heterogeneity. Briefly, in contrast to the naïve method, the proposed estimators generally have smaller bias but larger variances. This is a bias-variance trade-off. In this simulation study of N = 300, the proposed population size estimators performed poorly (in terms of RMSE) compared to the naïve method, but when capture probabilities and/or the population size N had increased, the proposed methods showed superiority in RMSE, so long as the benefit of bias is greater than the loss of variance. For example, when the population size is set to 5000, the RMSEs of the correction methods are significantly smaller than that of the naïve method for all considered scenarios (these results are not reported here).

6. Analysis of yellow-bellied prinia data

We applied the methods to the real capture–recapture data collected on the yellow-bellied prinia (Prinia flaviventris) study described in Section 1. The data were collected at the Mai Po Bird Sanctuary in Hong Kong from January to April in 1993. The yellow-bellied prinia is a resident bird in that area, and the period of January to April is a non-breeding season so that the closure population assumption is reasonable. We excluded two birds from the data due a missing value in wing length and one in fat index. As a consequence, there were n = 163 uniquely observed individuals captured, and the total number of captures made was $\sum _{i=1}^n{Y}_i=204$ with an average capture frequency of $\overline{Y}=1.251$. This recapture rate was very low. During the experiment, bird wing length measurements were recorded upon capture, which were known to be error-prone. The average wing length was $\overline{W}=45.366$ with a sample variance of 1.576. The estimated measurement error variance for the wing length was ${\widehat{\sigma }}_u^2=0.375$. Hence, the proportion of measurement error variance was about 24% (i.e., ${\sigma }_u^2/{\sigma }_w^2$), which suggests that the measurement error in bird wing lengths should not be ignored in the data analysis. Note that bird wing lengths will increase over time in nature, however the sampling period for this study is very short, and all measurements were collected on adult birds. We therefore assume bird wing lengths are constant. For fat index, we combined levels 2–4 as a single “fat” class (recorded here as 1) and used level 1 as a “not fat” class (recorded here as 0). Of the 163 observed birds, 70 had “no fat” recorded and 93 with fat. Fat index is considered as a precisely recorded covariate in this analysis.

We assume that ${\mathrm{\lambda }}_i=\exp (\alpha +\beta X_i+\gamma Z_i)$ is the expected count of the i-th individual where X_i = true wing length and Z_i = fat index. We fitted the naïve, CDS, PCS, and NPC as mentioned in Section 5. Model parameters and population size estimates (standard errors in parenthesis) are given in Table 3.

Table 3:

Regression coefficients and population size estimates (standard errors in parenthesis) for the Naïve, CDS, and NPC methods fitted to the Yellow-bellied Prinia capture–recapture data. The number of uniquely observed individuals was n = 163. Bird wing length measurements were used as a covariate where the estimated measurement error was ${\widehat{\sigma }}_u^2=0.375$ which yields a measurement reliability ratio of 76.2%. Notice the distinct difference in parameter and population estimates between the naïve and measurement error models.

	$\widehat{\alpha }$	$\widehat{\beta }$	$\widehat{\gamma }$	$\widehat{N}$
Naïve	−18.713 (3.640)	0.377 (0.080)	1.071 (0.400)	668.377 (157.907)
CDS	−24.690 (5.908)	0.508 (0.130)	1.063 (0.401)	721.761 (182.982)
PCS	−23.311 (4.831)	0.478 (0.106)	1.069 (0.401)	691.563 (173.311)
NPC	−22.109 (4.479)	0.452 (0.098)	1.074 (0.401)	687.119 (171.708)

From the result in Table 3, the naïve estimates for the intercept and slope were small compared with other methods which accounted for measurement errors. Specifically, the slope estimate of β was reduced by at least 16% compared with other methods. Hence the attenuation effect on the naïve method was quite remarkable. For population size estimation, the naïve method also yielded the smallest value, while the reduction was approximately 3% to 7%. This phenomena was similar to what we observed in the simulation study in Section 5. We found that PCS, and NPC produced comparable slope estimates and standard errors while CDS made a much larger correction for the measurement error. The estimate for the fat index effect was similar across all models (estimated at approximately 1.07), and was deemed significant for all models.

7. Discussion

In this study, we proposed three new zero-truncated Poisson regression methods that correct for uncertainty in the covariates that are collected in capture–recapture experiments. In particular, we introduced a corrected score method which has yet to be developed for zero-truncated regression and population size estimation. Through simulation studies and a real-life example, we observed that bias in the slope and population size estimates had significantly reduced when compared with naïve capture–recapture models. We also showed analytically, why the unadjusted population size estimate results in downward bias in the presence of measurement error.

Each of the proposed methods has its own advantages and disadvantages. For example, whilst both the conditional score and parametric corrected score methods are functional (that is, the distribution of the true covariate need not be specified), both methods still rely on normality or parametric assumptions on the measurement error. Furthermore, we found that the parametric corrected score method can be numerically unstable under some parameter settings.

Overall, we do not recommend the PCS method based on a normality measurement error assumption because it does not have a clear advantage in any aspect. However, the performance of PCS based on other parametric measurement error assumptions has not been fully understood. On a related research, the estimating function of PCS may be further modified (Huang, 2014) to improve the estimation efficiency. The naïve, CDS and NPC methods have their own advantages, we therefore provide some recommendations and guideline for their use when the aim is to estimate the population size:

Naïve method:

This estimator is typically biased, but can be used when the measurement error is small, the population size is small, or the capture-probabilities are low.

Conditional score:

This estimator usually has small bias when the measurement error is moderate or large. For moderate sample sizes, it works well even if the measurement error is not normally distributed. However, it may be biased with large sample sizes if there is violation of the normal error assumption.

Non-parametric corrected score:

This is the most robust estimator in terms of distributional assumptions. Compared with CDS, the NPC estimator is better when the measurement error distribution is not normally distributed. It usually works well with moderate sample sizes, and it performs the best when the population size is large.

We also assessed the performance of population size estimation on models that used “binned” error-contaminated covariates. Specifically, we evenly categorized the observed (error prone) continuous covariate into a fixed number of bins which were then used as dummy variables for the naïve model. We conducted several simulation studies (see Web Tables 9 and 10) where we considered 4 and 8 bins. We also fitted the binned covariate models to the yellow-bellied prinia bird capture–recapture data where we categorized the wing length covariate into 2, 3, 4 and 5 bins of equal frequency (see Web Table 11). In almost all the considered cases, we observed that the bias for the binned models was larger than our proposed estimators, and the coverage probabilities were below the nominal level. In summary, our simulation results showed that the method by binned error-contaminated covariates was not as good as our proposed estimators in general.

Several methodological problems remain to be addressed for future research. For the yellow-bellied prinia bird example, based on our analysis, the corrected score methods performed similarly in terms of parameter estimation but were substantially different compared to the fitted conditional score model. Thus, an unsolved and interesting problem remains in choosing the best capture–recapture model (or method) amongst a set of candidate models (that is, model selection in the presence of measurement error). Also, in many studies involving capture–recapture experiments, covariate values collected on captured individuals are missing at random; see Xi et al. (2009) and Stoklosa, Lee and Hwang (2019). A potential extension to our methods is to simultaneously account for missingness, in addition to measurement error in the covariates. We leave these research topics as future work.