Summary
The most common way to treat item nonresponse in surveys is to replace a missing value by a plausible value constructed on the basis of fully observed variables. Treating the imputed values as if they were observed may lead to invalid inferences. Bootstrap variance estimators for various finite population parameters are obtained using two pseudo-population bootstrap schemes. We establish the asymptotic properties of the resulting bootstrap variance estimators for population totals and population quantiles. A simulation study suggests that the methods perform well in terms of relative bias and coverage probability.
Keywords: Bootstrap, Doubly robust estimation, Imputation, Variance estimation
1. Introduction
Item nonresponse in surveys is usually dealt with through single imputation: a missing value is replaced by a single artificial value constructed using auxiliary information recorded for both respondents and nonrespondents. Treating imputed values as if they were observed may lead to underestimation of the variance of point estimators, leading to confidence intervals that are too narrow. Variance estimation in the presence of single imputation has been widely discussed; see, e.g., Särndal (1992), Rao & Shao (1992), Rao (1996), Shao & Sitter (1996), Shao & Steel (1999), Haziza (2009), and Kim & Rao (2009).
In the absence of nonresponse, bootstrap variance estimation procedures can be classified into three main groups. In the first, bootstrap samples are selected from the original sample (Rao & Wu, 1988; Sitter, 1992). Rao & Wu (1988) applied a scale adjustment directly to the data values to recover the usual variance formulae. The second group of procedures creates a pseudo-population from which bootstrap samples are then selected using the sampling design used to select the original samples; see Gross (1980), Bickel & Freedman (1984), Booth et al. (1994), a 2007 Université de Rennes 2 PhD thesis by G. Chauvet subsequently referred to as Chauvet (2007), Holmberg (1998), and Wang & Thompson (2012). In the third group of procedures, the adjustment is made to the survey weights, leading to bootstrap weights methods. For example, Rao et al. (1992) presented a modification of the method of Rao & Wu (1988), where the scale adjustment is applied to the survey weights rather than to data. In fact, many of the methods in the first two groups can be cast as bootstrap weights methods, whereby bootstrap weights are randomly generated so that the first two design moments of the sampling error are tracked by the corresponding bootstrap moments; see Antal & Tillé (2011) and Beaumont & Patak (2012). Mashreghi et al. (2016) survey bootstrap procedures in survey sampling.
To deal with imputed data, Shao & Sitter (1996) introduced a bootstrap method that consists of selecting bootstrap samples according to a complete-data bootstrap procedure and reimputing the missing values within each bootstrap sample using the imputation method used on the original data; see also Davison & Sardy (2007). The Shao–Sitter method carries the original response status for each sample unit, so the Shao–Sitter bootstrap variance estimator is consistent for the true variance only if the sampling fraction is negligible (Haziza, 2009; Mashreghi et al., 2014). For large sampling fractions, the Shao–Sitter bootstrap variance estimator tends to underestimate the true variance. Mashreghi et al. (2014) considered bootstrap procedures that work in the case of a stratified simple random sample without replacement and uniform nonresponse within strata, even for nonnegligible sampling fractions.
Assuming that the data are missing at random (Rubin, 1976), we develop novel pseudo-population bootstrap procedures for estimating the variance of imputed estimators that can be used when faced with large sampling fractions. Variance estimators in the presence of imputed data may be derived with respect to either the nonresponse model or the imputation model inferential approaches. In the first, inferences are conducted with respect to the joint distribution induced by the sampling design and the assumed nonresponse model, which is a set of assumptions about the unknown nonresponse mechanism. In the second, inferences are conducted with respect to the joint distribution induced by the sampling design, the nonresponse mechanism, and the assumed imputation model, a set of assumptions about the distribution of the variable requiring imputation. Most often, inferences are conducted with respect to the imputation model approach, but in some instances survey statisticians postulate a nonresponse model rather than an imputation model. This situation occurs, for example, when random hot-deck imputation is performed within classes, where the latter are assumed to be homogeneous with respect to the response propensity.
For each inferential framework, we propose a bootstrap procedure. Our procedures can be applied to high-entropy sampling designs (Berger, 1998, 2011), which include Poisson sampling, conditional Poisson sampling (Hájek, 1964), the Rao–Sampford procedure (Rao, 1965; Sampford, 1967) and the procedure of Chao (1982) as special cases. The extension to stratified sampling designs is straightforward, as sampling is performed independently within each stratum. Unlike those of Mashreghi et al. (2014), the proposed procedures may be applied to arbitrary nonresponse models.
2. Preliminaries
Let 
be a finite population consisting of 
distinct units. Let 
be a study variable and let 
be the 
-value attached to the 
th unit 
. We are interested in estimating the population total of the 
-values, 
, and a finite population quantile, 
where
 |
(1) |
with 
denoting the 
-values arranged in increasing order of size and 
the finite population distribution function (Särndal et al., 1992, Ch. 5),
 |
A sample 
is selected from 
according to a sampling design 
with first-order inclusion probabilities 
A complete-data estimator of 
is the Horvitz–Thompson estimator, 
, which is design-unbiased for 
; that is, 
where the subscript 
indicates that expectations and variances are evaluated with respect to the sampling design. A complete-data estimator of the population quantile 
is obtained from (1) by replacing 
with the estimated finite population distribution function
 |
(2) |
Under mild regularity conditions, the estimated quantile 
is design-consistent for 
(Wang & Opsomer, 2011).
When the 
-variable may be missing, it is not possible to compute 
or 
. Let 
be a response indicator associated with unit 
such that 
if unit 
is a respondent to item 
and 
otherwise. The 
-variable after imputation is denoted by 
, where 
if 
and 
if 
, with 
denoting the imputed value used for the missing 
An imputed estimator of 
, 
is obtained from the complete-data estimator 
by replacing 
with 
. For example, an imputed estimator of the population total 
is
 |
(3) |
Similarly, an imputed estimator of the quantile 
is obtained from (1) by replacing 
with 
and 
with 
, the imputed estimator of 
, obtained from (2) by replacing 
with 
We consider linear regression imputation, which uses the imputation model
 |
(4) |
where 
is a vector of fully observed variables attached to unit 
, 
is a vector of unknown parameters, and the errors 
satisfy 
with 
denoting an unknown parameter and 
being a known coefficient associated with unit 
. The subscript 
indicates that expectations and variances are evaluated with respect to the imputation model. We assume that 
, where 
is a vector of known constants. For example, a homoscedastic linear regression model that includes an intercept satisfies this condition.
Deterministic linear regression imputation consists of replacing the missing value 
by
 |
(5) |
where 
is a solution of the estimating equation
 |
(6) |
and 
is a coefficient attached to unit 
. Regardless of the choice of 
, the imputed estimator (3) is model-unbiased for 
if (4) holds (Haziza, 2009). The choice 
leads to survey-weighted deterministic linear regression imputation, whereas the choice 
leads to unweighted deterministic linear regression imputation. However, with these choices, the imputed estimator (3) is generally biased if the imputation model is misspecified. Protection against misspecification of the imputation model can be achieved through the use of
 |
(7) |
where 
denotes the estimated response probability attached to unit 
obtained by fitting the nonresponse model
 |
(8) |
with 
a vector of unknown coefficients. That is, 
where 
is a suitable estimator of 
.
Throughout the paper, we assume that the 
are missing at random, i.e.,
 |
and that units respond independently of one another. Under these assumptions, the choice 
ensures that 
is consistent for 
if the nonresponse model (8) is correctly specified, regardless of whether the imputation model (4) is correctly specified. With this choice of 
is often called doubly robust, as it remains consistent for the true parameter if either the nonresponse model or the imputation model is correctly specified. Doubly robust procedures have been studied in Robins et al. (1994), Scharfstein et al. (1999), Bang & Robins (2005), Tan (2006), Kang & Schafer (2007), Cao et al. (2009), Haziza & Rao (2006) and Kim & Haziza (2014).
For population quantiles, we consider random hot-deck imputation within classes, as deterministic regression imputation tends to distort the distribution of the variable being imputed, leading to biased estimators of quantiles. The random hot-deck procedure can be described as follows: first, preliminary predicted values 
are obtained for all the sample units, where 
is a solution of (6) with 
:
 |
The sample is then partitioned into 
imputation classes using an equal-quantile method that consists of ordering the sample from the lowest 
to the largest and then forming 
equal-size classes. A missing value in class 
is replaced by the value of a respondent, called a donor, belonging to the same class and chosen at random with probability proportional to 
; that is,
 |
(9) |
where 
is the set of sample units belonging to class 
. The choices 
and 
lead to unweighted and survey-weighted random hot-deck imputation, respectively. We consider the choice 
given by (7). Boistard et al. (2016) showed that the resulting estimator of 
is doubly robust under the cell mean model, which is a special case of (4), where 
is a 
-vector consisting of 
entries equal to zero and a single entry equal to 1, which identifies the class to which the unit 
belongs.
The commonly used unweighted and survey-weighted versions of deterministic linear regression imputation can be viewed as special cases of (5) based on a nonresponse model (8) that contains only the intercept, leading to 
the overall response rate, for all 
. Therefore, the bootstrap procedures proposed in § 4 and § 5 can be readily applied to the case of unweighted and survey-weighted deterministic linear regression imputation. The same is true for the commonly used random hot-deck imputation within classes.
To express the variance of 
under either the imputation model or the nonresponse model approach, we express the total error as
 |
(10) |
where 
denotes the complete-data estimator of 
. The first term on the right-hand side of (10) is the sampling error, whereas the second term represents the nonresponse error.
3. Complete-data pseudo-population bootstrap methods
In the case of complete data, Booth et al. (1994) proposed an algorithm for simple random sampling without replacement. The extension to fixed-size inclusion probability proportional to size sampling was considered in Holmberg (1998) and Chauvet (2007), while Poisson sampling was examined in Chauvet (2007). A general pseudo-population bootstrap algorithm can be described as in Algorithm 1.
Algorithm 1.
General pseudo-population bootstrap algorithm.
Step 1. Repeat the pair
a total of
times for all
in
to create
, the fixed part of the pseudo-population.
Step 2. To complete the pseudo-population
, draw
from
using the original sampling design that led to
, leading to
.
Step 3. Take a bootstrap sample
from
using the same sampling design that led to
.
Step 4. Compute the bootstrap statistic,
, on the bootstrap sample
.
Step 5. Repeat Steps 3 and 4 a large number of times,
, to get
. Define
with
.
Step 6. Repeat Steps 2 to 5
times to get
.
Step 7. Estimate
by
where the subscripts
and
refer to the bootstrap randomness induced by the completion of the pseudo-population in Step 2 and the bootstrap sampling in Step 3, respectively. In practice, we use the Monte Carlo approximation
In Step 1, if the original sample was selected according to simple random sampling without replacement, we first make 
copies of the vectors in 
to build the fixed part of the pseudo-population 
, where 
is the original sample size. In Step 2, a simple random sample without replacement of vectors, 
, of size 
is drawn from 
. For a fixed-size inclusion probability proportional to size sampling design, the pseudo-population is created by duplicating unit 
in 
a total of 
times to build 
and then taking a sample 
of size 
from 
in the same way that 
was selected from 
. Alternatively, Chauvet (2007) suggests that the completion of the pseudo-population may also be done by applying Poisson sampling with inclusion probability 
for unit 
, which simplifies the completion process.
In the case of Poisson sampling and the population total, 
, Chauvet (2007) showed that
 |
the textbook variance estimator under Poisson sampling (Särndal et al., 1992, Ch. 3).
In the case of the maximum-entropy sampling design (Hájek, 1964), also referred to as conditional Poisson sampling, Chauvet (2007) showed that
 |
(11) |
where 
The right-hand side of (11) is an estimator of the variance of the Horvitz–Thompson estimator based on the Hájek (1964) approximation of the second-order inclusion probabilities. Hájek (1964) showed that (11) is consistent under conditional Poisson sampling. Berger (2011) showed that if an estimator is consistent under the conditional Poisson sampling design, then it is consistent under any sampling design close to the maximum-entropy design. A number of empirical investigations have shown that (11) performs well in terms of bias and stability when the population and the sample sizes are large enough (Brewer & Donadio, 2003; Matei & Tillé, 2005; Haziza et al., 2008).
In the Supplementary Material, we show that 
is a consistent estimator of 
, where 
stands for either a population total, a smooth function of means or a population quantile. We establish consistency with respect to either Poisson sampling or a high-entropy fixed-size sampling design, which implies that normal-based confidence intervals with bootstrap estimators of variance are valid for such designs. We also establish the validity of bootstrap intervals for Poisson sampling.
4. Pseudo-population bootstrap method under the imputation model approach
Using (10), the variance of 
with respect to the imputation model approach can be expressed as
 |
(12) |
say, where 
denotes the set of respondents and the subscript 
indicates the expectations and variances with respect to the nonresponse model (Särndal, 1992). From (12), the variance of 
is the sum of three terms: the anticipated sampling variance, 
of the complete-data estimator 
; the nonresponse variance, 
; and a mixed component, 
In this section, the validity of the proposed bootstrap procedure requires the correct specification of the imputation model, whereas the nonresponse model need not be correctly specified.
Consider the imputed values 
given by (5). Define the standardized centred residual for 
by
 |
where 
and 
denotes the number of respondents. A pseudo-population 
of size 
of auxiliary variables, of inclusion probabilities, and of response indicators is first created from 
. In the case of a fixed-size sampling design, we have 
For Poisson sampling, we have 
where the subscript 
denotes the random mechanism used for creating 
The pseudo-population 
is constructed by applying a complete-data pseudo-population method on the sample 
. Then, an independent and identically distributed sample of size 
, 
, is selected from the set of standardized residuals 
. The bootstrap characteristic of interest 
is computed using the auxiliary variables 
in 
, the estimators of the model parameters based on the original set of respondents and the selected bootstrap errors 
; see Algorithm 2 and the discussion that follows. This leads to the bootstrap pseudo-population 
of vectors 
. The bootstrap sample 
is drawn from 
according to the original sampling design. The bootstrap set of respondents, 
is immediately identified through the response indicators, 
, obtained from the original response indicators when constructing 
. The bootstrap missing data are imputed using the original imputation method. A bootstrap variance estimator of 
is
 |
(13) |
where 
and 
are respectively the bootstrap parameter computed on 
and the bootstrap imputed estimator computed after imputing the bootstrap missing values. The subscripts 
and 
denote the bootstrap imputation model and the sampling mechanism used for selecting 
, respectively. The imputation model scheme can be described as in Algorithm 2.
Algorithm 2.
Imputation model scheme.
Step 1. Using the imputation model, obtain
by solving (6) and compute the set of standardized centred residuals
.
Step 2. Depending on the original sampling design, apply a complete-data pseudo-population bootstrap method on
to build the pseudo-population
of size
.
Step 3. Draw an independent and identically distributed sample of size
,
, from the sample of centred standardized residuals
computed in Step 1. Combining
and
and using the estimated imputation model computed on the original sample of respondents, define the bootstrap values
and then the bootstrap pseudo-population
. Compute
, the bootstrap analogue of the parameter
on the resulting pseudo-population
.
Step 4. The bootstrap sample
is drawn from
using the original sampling design. The set of respondents
is defined as those units in
for which the response indicator
is equal to 1.
Step 5. Impute the bootstrap missing values in
by applying the same imputation method used for the original missing data. The vector of imputed values is
if
and
if
. Compute
, the bootstrap estimator of
based on
, where the probabilities
are recomputed on the bootstrap sample
the same way the
s were computed on the original sample.
Step 6. Repeat Steps 3 to 5 a large number of times,
to get
and
. Define
Step 7. Repeat Steps 2 to 6
times to get
.
Step 8. A bootstrap variance estimator of
is (13). In practice, we use its Monte Carlo approximation
To build a pseudo-population in Step 2, first make 
copies of unit 
in 
, for all 
, to build a partial pseudo-population 
. Then draw a random sample, 
, from 
according to the original sampling design with inclusion probability 
for unit 
.
The bootstrap characteristic of interest 
in Step 3 is defined by 
where 
, and the imputed value 
in Step 5 is computed as 
, where
 |
with 
. For random hot-deck imputation within classes, we first estimate the imputation model based on the bootstrap sample of respondents. Using that model and the auxiliary variables available for all units in the bootstrap sample, we compute the bootstrap predicted values 
to form 
imputation classes, so that 
. We then apply random hot-deck imputation within each class, leading to
 |
In the Supplementary Material, we show that 
in (13) is a consistent estimator of the true variance 
in the cases of population totals, smooth functions of means and population quantiles when the original sample is selected according to Poisson sampling or a high-entropy fixed-size sampling design, leading to the validity of normal-based confidence intervals. We also establish the validity of bootstrap intervals for Poisson sampling.
5. Pseudo-population bootstrap method under the nonresponse model approach
Using the decomposition (10), the variance of 
with respect to the nonresponse model approach can be expressed as
 |
(14) |
The term 
in (14) is the sampling variance of the complete-data estimator 
whereas the term 
represents the nonresponse variance. In this section, correct specification of the nonresponse model is required for the validity of the proposed bootstrap procedure, though the imputation model need not be correctly specified.
To construct the pseudo-population, we use the fact that the set of respondents to item 
can be viewed as a sample that would have been selected by a Poisson sampling design with unknown inclusion probabilities 
. The 
s being unknown, the estimated response probabilities 
are used in the bootstrap procedures. The pseudo-population is created in two distinct steps. First, the set of respondents 
is augmented into a full sample 
of size 
, which we call a pseudo-sample, by applying the method of Chauvet (2007) for Poisson sampling. Then, depending on the original sampling design, the pseudo-population 
is constructed from the pseudo-sample 
using a pseudo-population bootstrap method. The pseudo-population 
is made up of vectors 
. Then, the bootstrap version of the parameter of interest, 
, is computed on 
and bootstrap samples 
are selected from 
according to the original sampling design.
Nonresponse is generated in 
according to Poisson sampling with the estimated original response probabilities 
as the inclusion probabilities. The resulting bootstrap set of respondents is denoted by 
. Missing values, which are those belonging to 
are filled in using the same imputation method that was utilized in the original sample, which involves re-estimating the response probabilities. Finally, the bootstrap imputed estimator 
is computed from the imputed bootstrap dataset. A bootstrap variance estimator of 
is
 |
(15) |
where the subscripts 
, 
and 
indicate, respectively, the sampling mechanisms for generating 
and 
and for selecting 
, while the subscript 
indicates the mechanism used to generate 
. The nonresponse model scheme can be described as in Algorithm 3.
Algorithm 3.
Nonresponse model scheme.
Step 1. For
, make
copies of
to construct
, the fixed part of the pseudo-sample. Use Poisson sampling with inclusion probabilities
for
to draw a further sample
of vectors
from
in order to complete the pseudo-sample
so that
of size
.
Step 2. Depending on the original sampling design, apply an appropriate complete-data pseudo-population bootstrap method on the resulting
to create the fixed part,
, and the random part,
, of the pseudo-population. The pseudo-population,
, of size
is obtained by combining
and
.
Step 3. The bootstrap sample
is drawn from
using the original sampling design. Generate the bootstrap sample of response indicators,
, using the original estimated response probabilities, i.e.,
for all
.
Step 4. Impute the bootstrap missing values in
by applying the same imputation method used for the original missing data. The vector of imputed values is
if
and
if
. Compute
, the imputed bootstrap estimator of
based on
, where
denotes the estimated response probability recomputed from the bootstrap values.
Step 5. Repeat Steps 3 and 4 a large number of times,
, to get
. Define
with
.
Step 6. Repeat Steps 1 and 5
times to get
.
Step 7. A bootstrap variance estimator of
is (15). In practice, we use its Monte Carlo approximation
For instance, if the original sample was selected according to simple random sampling, in Step 2 we would first make 
copies of the vectors in 
of size 
to build the fixed part of the pseudo-population 
. We would then draw a simple random sample of vectors, 
, of size 
from 
. The resulting pseudo-population is of size 
. In the case of Poisson sampling, the pseudo-population is created by duplicating unit 
in the pseudo-sample, 
, 
times to build 
and then taking a sample, 
, from 
according to Poisson sampling with inclusion probability 
for unit 
. In this case, 
.
In the Supplementary Material, we show that 
in (15) is a consistent estimator of the true variance 
in the cases of population totals, smooth functions of means and population quantiles when the original sample is selected according to Poisson sampling or a high-entropy fixed-size sampling design, establishing the validity of normal-based confidence intervals. We also show the validity of bootstrap intervals for Poisson sampling.
Remark 1.
The nonresponse model scheme can be readily applied for estimating the variance of adjusted estimators in the context of unit nonresponse. The latter is usually handled through some form of weight adjustment procedure in order to reduce the nonresponse bias. Except for Step 4, where the imputed estimator
would be replaced by a reweighted estimator such as a propensity score adjusted estimator, all the other steps involved in the nonresponse model scheme would be identical.
6. Simulation
We performed a simulation study to assess the performance of the proposed methods in terms of relative bias and coverage probability. We generated a population of size 
2000 with four variables: a survey variable 
and three auxiliary variables 
, 
and 
The 
-values and 
-values were generated from a gamma distribution with shape and scale parameters set to 1 and 5, respectively. The 
-values were generated from a lognormal distribution with mean 0 and standard deviation 0.5 on the log scale. Given 
and 
, the 
-values were generated according to the linear regression model
 |
(16) |
where the errors 
were drawn independently.
In the case of the nonresponse model inferential approach, a single fixed population was generated and samples were repeatedly selected from this population. In the case of the imputation model inferential approach, a new population was generated using (16) each time. We were interested in estimating the population total 
and the population median 
. More simulation results, including some results for the first and third population quartiles 
and 
, are presented in the Supplementary Material.
Separately for each approach, we drew 
3000 samples, 
of size 
according to simple random sampling without replacement and to conditional Poisson sampling with inclusion probabilities proportional to 
. The sample size 
was set to 
180 and 
900, which corresponds to sampling fractions of 
9% and 
45%, respectively.
In each sample, nonresponse indicators of the 
-variable were generated from a Bernoulli distribution with parameter 
where 
. The parameters were chosen so that the overall response rate was approximately 70%.
To replace the missing values in the case of the population total, 
, we used deterministic regression imputation, whereby the imputed values are given by (5). To handle nonresponse in the case of the population median, 
, we used random hot-deck imputation within five imputation classes, whereby the imputed values are given by (9). The imputed estimators of 
and 
are denoted by 
and 
, respectively.
We considered three distinct scenarios. In Scenario 1 both the imputation and the nonresponse working models were correctly specified, In Scenario 2 only the nonresponse working model was correctly specified, and Scenario 3 is where only the imputation working model was correctly specified. Under random hot-deck imputation within classes, when the imputation working model is correctly specified, it means that the preliminary predicted values 
were obtained using the correctly specified model.
The different scenarios and corresponding working models are shown in Table 1.
Table 1.
Working models used for imputation
| Scenario | Nonresponse working model | Outcome regression working model |
|---|---|---|
| 1 |
|
|
| 2 |
|
|
| 3 |
|
|
As a measure of bias of a point estimator 
we computed the Monte Carlo percentage relative bias. Results not shown here suggest that the point estimators were nearly unbiased in all scenarios.
To estimate the standard error of each point estimator, we also computed the bootstrap standard error estimators 
and 
assuming 
and 
under the imputation model scheme presented in § 4 and the nonresponse model scheme presented in § 5, respectively. While the nonresponse model and imputation model bootstrap schemes have been designed to estimate the variability of the imputed estimators under the nonresponse model and imputation model inferential approaches, 
and 
were evaluated as estimates of standard error under both inferential paradigms. We computed standard error estimates rather than variance estimates as they are the quantities that enter into the computation of coefficients of variation or confidence intervals.
In addition, we estimated the standard error using the Shao & Sitter (1996) method. Let 
be the resulting estimated bootstrap standard error. We obtained 
through the following algorithm: first, Steps 1 to 3 of Algorithm 1 were applied to 
as the dataset carries the original response status for each sampled unit in the bootstrap procedure. In Step 4 of Algorithm 1, nonrespondents in the bootstrap samples, identified using the original response indicators, are reimputed using the same imputation method that was used in the original sample. The bootstrap statistic, 
, was then computed based on the reimputed values. Steps 5 to 7 of Algorithm 1 were then performed with 
replaced by 
, which led to the bootstrap variance estimator 
. Finally, the bootstrap standard error was obtained as 
. Additional remarks about the Shao–Sitter method can be found in the Supplementary Material.
As a measure of bias of a standard error estimator 
we computed the Monte Carlo percentage relative bias
 |
where 
and 
with 
denoting the estimator 
in the 
th sample and 
The Monte Carlo variance 
approximates either 
or 
depending on whether the population was fixed or random.
In Table 2, the columns 
and 
correspond to the standard error of each point estimator under the nonresponse and imputation model approaches, respectively. The Monte Carlo standard error for 
was obtained by fixing the population and simulating the effect of sampling and nonresponse, whereas 
was obtained by generating a new population at each iteration and then simulating the effect of sampling and nonresponse.
Table 2.
Monte Carlo percentage relative bias of bootstrap standard error estimators based on 
samples of size 
and 
| SRSWOR | CPS | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
 9% |
 45% |
 9% |
 45% |
|||||||
|
Scenario | Scheme | NRM | IM | NRM | IM | NRM | IM | NRM | IM |
|
1 |
|
–1.8 | 0.9 | 0.2 | 1.0 | –1.6 | –0.1 | –1.8 | –1.1 |
|
0.8 | 3.2 | 1.0 | 1.4 | 0.7 | 0.9 | –0.6 | –0.6 | ||
|
–3.0 | –0.4 | –10.0 | –9.3 | –1.5 | –0.1 | –4.1 | –3.3 | ||
| 2 |
|
2.0 | –1.1 | 0.1 | 1.1 | –0.9 | –1.4 | –1.7 | –1.2 | |
|
6.5 | 3.8 | 4.5 | 6.1 | –1.7 | –2.9 | –2.6 | –2.5 | ||
|
0.8 | –2.3 | –10.0 | –9.3 | –0.9 | –1.5 | –4.0 | –3.4 | ||
| 3 |
|
–2.8 | –0.1 | –1.5 | –0.6 | –1.0 | 3.2 | 2.7 | 2.2 | |
|
–1.5 | 0.9 | –0.0 | 0.7 | –0.6 | 2.3 | 2.6 | 1.1 | ||
|
–3.1 | –0.6 | –10.2 | –9.4 | –2.5 | 1.7 | –0.9 | –1.4 | ||
|
1 |
|
0.3 | 1.5 | –2.0 | 0.4 | –0.0 | 3.1 | –3.4 | –0.1 |
|
–5.0 | 0.5 | –11.5 | 0.1 | –4.3 | 1.4 | –11.6 | –0.3 | ||
|
–0.7 | 0.6 | –9.6 | –8.0 | –0.7 | 2.2 | –9.0 | –6.3 | ||
| 2 |
|
1.5 | 0.9 | –1.2 | 1.2 | 2.1 | 1.0 | –1.5 | –1.9 | |
|
1.8 | 3.0 | –4.0 | 3.1 | 5.5 | 3.0 | 1.1 | –0.2 | ||
|
0.7 | –0.1 | –9.0 | –7.3 | 0.7 | –0.1 | –7.3 | –8.0 | ||
| 3 |
|
–1.5 | 1.8 | –6.9 | –1.8 | –2.7 | 0.7 | –6.0 | –3.1 | |
|
–4.1 | 1.6 | –10.3 | –0.6 | –5.4 | –0.1 | –9.1 | –2.1 | ||
|
0.4 | 1.7 | –7.8 | –8.4 | –1.8 | 0.0 | –6.0 | –8.5 | ||
SRSWOR, simple random sampling without replacement; CPS, conditional Poisson sampling; IM, imputation model approach; NRM, nonresponse model approach.
With 
9%, some bootstrap imputation classes sometimes ended up with one or no bootstrap respondent, which made it impossible to impute the missing data within these classes. In such cases, we have ignored that bootstrap sample and computed the bootstrap variance estimator on the remaining bootstrap samples. This happened in less than 
of 
3 000 000 bootstrap samples.
We also computed 95% normal-based confidence intervals using the bootstrap estimate of standard error.
Table 2 shows the relative bias of the bootstrap estimators of the standard error for both simple random sampling without replacement and conditional Poisson sampling. The two designs led to similar results.
In Scenarios 1 and 2, where the nonresponse model was correctly specified, the estimator 
performed well with an absolute relative bias less than 2.0% and 3.4% for the population total and the population median, respectively. In Scenarios 1 and 3, where the imputation model was correctly satisfied, 
performed well with an absolute relative bias less than 3.2% and 2.1% for the population total and the population median, respectively. The Shao–Sitter method performed well for 
9% but led to some underestimation for 
45%. This behaviour was expected (Mashreghi et al., 2014). The underestimation was larger in the case of simple random sampling without replacement.
The coverage probabilities of the 95% normal-based confidence intervals are shown in Table 3. Since the simulation was based on 3000 samples, a coverage between 94.2% and 95.8% is not statistically different from 95%. The normal-based confidence intervals based on the nonresponse model bootstrap scheme were excellent in Scenarios 1 and 2, where the nonresponse model was adequately specified. The bootstrap intervals based on the imputation model scheme did well to cover the parameter value computed based on the imputation model approach in Scenarios 1 and 3, where the imputation model was well specified. The bootstrap intervals based on the Shao–Sitter method did not do well in terms of coverage probability when 
45%. For conditional Poisson sampling, the normal confidence intervals were relatively good in the case of the population total even for 
45%, whereas the coverage probability varied between 90.6% and 93.3% for the population median.
Table 3.
Coverage probability of 
normal confidence intervals based on a standard error computed from the corresponding bootstrap method using 
samples of size 
and 
| SRSWOR | CPS | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
 9% |
 45% |
 9% |
 45% |
|||||||
|
Scenario | Scheme | NRM | IM | NRM | IM | NRM | IM | NRM | IM |
|
1 | NRM* | 94.4 | 95.5 | 95.1 | 95.1 | 94.6 | 94.4 | 94.2 | 94.8 |
| IM* | 95.0 | 95.9 | 95.4 | 95.1 | 95.0 | 94.8 | 94.6 | 95.0 | ||
| SHS* | 94.1 | 95.3 | 92.0 | 92.8 | 94.5 | 94.4 | 93.8 | 94.2 | ||
| 2 | NRM* | 94.9 | 94.4 | 95.1 | 95.0 | 94.3 | 94.2 | 94.2 | 94.8 | |
| IM* | 95.6 | 95.4 | 96.1 | 96.0 | 95.5 | 95.3 | 94.1 | 94.6 | ||
| SHS* | 94.6 | 94.0 | 92.0 | 92.7 | 93.9 | 94.0 | 93.8 | 94.3 | ||
| 3 | NRM* | 93.2 | 94.9 | 94.7 | 95.0 | 94.0 | 95.1 | 95.3 | 95.3 | |
| IM* | 93.6 | 95.2 | 94.9 | 95.1 | 94.4 | 95.5 | 95.8 | 95.2 | ||
| SHS* | 93.0 | 94.7 | 92.4 | 92.2 | 93.9 | 95.3 | 94.8 | 94.3 | ||
|
1 | NRM* | 94.9 | 94.3 | 93.6 | 94.6 | 93.9 | 95.0 | 92.6 | 94.2 |
| IM* | 94.9 | 94.4 | 91.8 | 95.5 | 93.9 | 95.3 | 90.7 | 94.6 | ||
| SHS* | 94.9 | 94.4 | 91.6 | 92.2 | 93.9 | 94.7 | 90.6 | 92.9 | ||
| 2 | NRM* | 94.6 | 94.4 | 93.2 | 94.7 | 94.3 | 94.2 | 93.1 | 94.8 | |
| IM* | 95.3 | 94.7 | 93.1 | 95.1 | 95.3 | 95.3 | 95.0 | 95.3 | ||
| SHS* | 94.1 | 94.0 | 91.3 | 92.6 | 93.9 | 94.0 | 91.1 | 93.3 | ||
| 3 | NRM* | 94.2 | 94.2 | 92.3 | 93.4 | 93.3 | 94.5 | 92.0 | 92.8 | |
| IM* | 94.5 | 95.3 | 91.8 | 94.4 | 93.3 | 95.1 | 91.9 | 94.0 | ||
| SHS* | 94.3 | 94.4 | 91.3 | 91.9 | 93.8 | 94.4 | 91.4 | 91.7 | ||
SRSWOR, simple random sampling without replacement; CPS, conditional Poisson sampling; IM, imputation model approach; NRM, nonresponse model approach; IM*, imputation model scheme; NRM*, nonresponse model scheme; SHS*, Shao–Sitter bootstrap procedure.
Supplementary Material
Acknowledgement
The authors thank the editor, an associate editor and two referees for their constructive comments. This work was supported by the Natural Sciences and Engineering Research Council of Canada and the Canadian Statistical Sciences Institute. Chen was partially supported by the National Institutes of Health.
Supplementary material
Supplementary material available at Biometrika online includes all theoretical developments related to the bootstrap methods presented in § 3, 4 and 5 as well as additional simulation results.
References
- Antal, E. & Tillé, Y. (2011). A direct bootstrap method for complex sampling designs from a finite population. J. Am. Statist. Assoc. 106, 534–43. [Google Scholar]
- Bang, H. & Robins, J. M. (2005). Doubly robust estimation in missing data and causal inference models. Biometrics 61, 962–72. [DOI] [PubMed] [Google Scholar]
- Beaumont, J.-F. & Patak, Z. (2012). On the generalized bootstrap for sample surveys with special attention to Poisson sampling. Int. Statist. Rev. 80, 127–48. [Google Scholar]
- Berger, Y. G. (1998). Rate of convergence for asymptotic variance of the Horvitz–Thompson estimator. J. Statist. Plan. Infer. 74, 149–68. [Google Scholar]
- Berger, Y. G. (2011). Asymptotic consistency under large entropy sampling designs with unequal probabilities. Pak. J. Statist. 27, 407–26. [Google Scholar]
- Bickel, P. J. & Freedman, D. A. (1984). Asymptotic normality and the bootstrap in stratified sampling. Ann. Statist. 12, 470–82. [Google Scholar]
- Boistard, H., Chauvet, G. & Haziza, D. (2016). Doubly robust inference for the distribution function in the presence of missing survey data. Scand. J. Statist. 43, 683–99. [Google Scholar]
- Booth, J. G., Butler, R. W. & Hall, P. (1994). Bootstrap methods for finite populations. J. Am. Statist. Assoc. 89, 1282–9. [Google Scholar]
- Brewer, K. R. W. & Donadio, M. E. (2003). The high entropy variance of the Horvitz–Thompson estimator. Survey Methodol. 29, 189–96. [Google Scholar]
- Cao, W., Tsiatis, A. A. & Davidian, M. (2009). Improving efficiency and robustness of the doubly robust estimator for a population mean with incomplete data. Biometrika 96, 723–34. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Chao, M. T. (1982). A general purpose unequal probability sampling plan. Biometrika 69, 653–6. [Google Scholar]
- Davison, A. C. & Sardy, S. (2007). Resampling variance estimation in surveys with missing data. J. Offic. Statist. 23, 371–86. [Google Scholar]
- Gross, S. (1980). Median estimation in sample surveys In Proc. Survey Res. Meth. Sec., American Statistical Association, Houston, Texas. American Statistical Association. [Google Scholar]
- Hájek, J. (1964). Asymptotic theory of rejective sampling with varying probabilities from a finite population. Ann. Math. Statist. 35, 1491–523. [Google Scholar]
- Haziza, D. (2009). Imputation and inference in the presence of missing data. In Handbook of Statistics, Rao, C. R. ed., vol. 29A Oxford: Elsevier, pp. 215–46. [Google Scholar]
- Haziza, D. & Rao, J. N. K. (2006). A non-response model approach to inference under imputation for missing survey data. Survey Methodol. 32, 53–64. [Google Scholar]
- Haziza, D., Rao, J. N. K. & Mecatti, F. (2008). Evaluation of some approximate variance estimators under the Rao–Sampford unequal probability sampling design. Metron 66, 91–108. [Google Scholar]
- Holmberg, A. (1998). A bootstrap approach to probability proportional to size sampling In Proc. Survey Res. Meth. Sec., American Statistical Association, Dallas, Texas. American Statistical Association. [Google Scholar]
- Kang, J. D. & Schafer, J. L. (2007). Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data. Statist. Sci. 22, 523–39. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kim, J. K. & Haziza, D. (2014). Doubly robust inference with missing data in survey sampling. Statist. Sinica 24, 375–94. [Google Scholar]
- Kim, J. K. & Rao, J. N. K. (2009). A unified approach to linearization variance estimation from survey data after imputation for item nonresponse. Biometrika 96, 917–32. [Google Scholar]
- Mashreghi, Z., Haziza, D. & Léger, C. (2016). A survey of bootstrap methods in finite population sampling. Statist. Surveys 10, 1–52. [Google Scholar]
- Mashreghi, Z., Léger, C. & Haziza, D. (2014). Bootstrap methods for imputed data from regression, ratio and hot-deck imputation. Can. J. Statist. 42, 142–67. [Google Scholar]
- Matei, A. & Tillé, Y. (2005). Evaluation of variance approximations and estimators in maximum entropy sampling with unequal probability and fixed sample size. J. Offic. Statist. 21, 543–70. [Google Scholar]
- Rao, J. N. K. (1965). On two simple schemes of unequal probability sampling without replacement. J. Indian Statist. Assoc. 3, 173–80. [Google Scholar]
- Rao, J. N. K. (1996). On variance estimation with imputed survey data. J. Am. Statist. Assoc. 91, 499–506. [Google Scholar]
- Rao, J. N. K. & Shao, J. (1992). Jackknife variance estimation with survey data under hot deck imputation. Biometrika 79, 811–22. [Google Scholar]
- Rao, J. N. K. & Wu, C. F. J. (1988). Resampling inference with complex survey data. J. Am. Statist. Assoc. 83, 231–41. [Google Scholar]
- Rao, J. N. K., Wu, C. F. J. & Yue, K. (1992). Some recent work on resampling methods for complex surveys. Survey Methodol. 18, 209–17. [Google Scholar]
- Robins, J. M., Rotnitzky, A. & Zhao, L. P. (1994). Estimation of regression coefficients when some regressors are not always observed. J. Am. Statist. Assoc. 89, 846–66. [Google Scholar]
- Rubin, D. B. (1976). Inference and missing data. Biometrika 63, 581–92. [Google Scholar]
- Sampford, M. R. (1967). On sampling without replacement with unequal probabilities of selection. Biometrika 54, 499–513. [PubMed] [Google Scholar]
- Särndal, C.-E. (1992). Methods for estimating the precision of survey estimates when imputation has been used. Survey Methodol. 18, 241–52. [Google Scholar]
- Särndal, C.-E., Swensson, B. & Wretman, J. (1992). Model-Assisted Survey Sampling. New York: Springer. [Google Scholar]
- Scharfstein, D. O., Rotnitzky, A. & Robins, J. M. (1999). Adjusting for nonignorable drop-out using semiparametric nonresponse models. J. Am. Statist. Assoc. 94, 1096–120. [Google Scholar]
- Shao, J. & Sitter, R. R. (1996). Bootstrap for imputed survey data. J. Am. Statist. Assoc. 91, 1278–88. [Google Scholar]
- Shao, J. & Steel, P. (1999). Variance estimation for survey data with composite imputation and nonnegligible sampling fractions. J. Am. Statist. Assoc. 94, 254–65. [Google Scholar]
- Sitter, R. R. (1992). A resampling procedure for complex survey data. J. Am. Statist. Assoc. 87, 755–65. [Google Scholar]
- Tan, Z. (2006). A distributional approach for causal inference using propensity scores. J. Am. Statist. Assoc. 101, 1619–37. [Google Scholar]
- Wang, J. C. & Opsomer, J. D. (2011). On asymptotic normality and variance estimation for nondifferentiable survey estimators. Biometrika 98, 91–106. [Google Scholar]
- Wang, Z. & Thompson, M. E. (2012). A resampling approach to estimate variance components of multilevel models. Can. J. Statist. 40, 150–71. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.