An adaptive resampling test for detecting the presence of significant predictors

Abstract

This paper investigates marginal screening for detecting the presence of significant predictors in high-dimensional regression. Screening large numbers of predictors is a challenging problem due to the non-standard limiting behavior of post-model-selected estimators. There is a common misconception that the oracle property for such estimators is a panacea, but the oracle property only holds away from the null hypothesis of interest in marginal screening. To address this difficulty, we propose an adaptive resampling test (ART). Our approach provides an alternative to the popular (yet conservative) Bonferroni method of controlling familywise error rates. ART is adaptive in the sense that thresholding is used to decide whether the centered percentile bootstrap applies, and otherwise adapts to the non-standard asymptotics in the tightest way possible. The performance of the approach is evaluated using a simulation study and applied to gene expression data and HIV drug resistance data.

1 Introduction

The problem of selecting significant predictors is a central aspect of scientific discovery, and has become increasingly important in an era in which massive data sets are readily available (Fan and Li, 2006). Much of the modern statistical literature in this area focuses on consistency of variable selection in high-dimensional settings based on machine learning and data mining techniques (e.g., Fan and Li 2001; Zou and Hastie 2005; Huang et al. 2008; Fan and Lv 2008; Genovese et al. 2012). A major gap in this literature, however, has been the scarcity of formal hypothesis testing procedures that take variable selection into account; the oracle property enjoyed by many variable selection methods in the presence of high dimensionality can not be applied directly for testing whether a post-model-selected variable is significant. In bioinformatics, for example, variable selection techniques based on penalization (such as lasso, scad, etc) are routinely used to produce lists of differentially-expressed genes that are most related to disease risk, but few methods for obtaining valid p-values have been developed.

A more traditional approach to the selection of significant predictors is multiple testing to control either family-wise error rate (FWER), or false-discovery rate (Benjamini and Hochberg 1995; Dudoit et al. 2003; Efron 2006; Dudoit and van der Laan 2008; Efron 2010). Procedures that control FWER (e.g., Bonferroni, or Holm's procedure) are often criticized as being too conservative (in the sense of having low power). False-discovery rate methods, on the other hand, although having greater power, incur the cost of inflated FWER. Our aim in the present paper is to introduce a more powerful single test that can be used as an alternative screening procedure to detect the presence of some significant predictor while rigorously controlling FWER.

The proposed procedure uses marginal linear regression to select the predictor (from among covariates X₁, . . . , X_p) that has maximal sample correlation with a scalar outcome Y (as in marginal screening or correlation learning, Genovese et al. 2012). The test is based on ${\widehat{\theta }}_n$ the estimated marginal regression coefficient of the selected predictor. If there is a unique predictor, say X_k0, maximally correlated with the outcome, then the selection procedure consistently estimates k₀, and ${\widehat{\theta }}_n$ is asymptotically normal; if all predictors are uncorrelated with the outcome, then the selected predictor does not converge (in probability) and ${\widehat{\theta }}_n$ has a non-normal limiting distribution. In particular, the limiting distribution is discontinuous (at zero) as a function of the regression coefficient of X_k0 (where k₀ is not identifiable), and this “non-regularity” causes non-uniform convergence.

Breiman (1992) drew early attention to the issue of invalid post-model-selection inference, calling it the “quiet scandal” of Statistics; even earlier references are mentioned in Berk et al. (2013). Samworth (2003) gave a detailed account of the inaccuracy of bootstrap methods applied to super-efficient estimators. Leeb and Pötscher (2006) (and other papers by the same authors) established that non-uniform limiting behavior of post-model-selected estimators is at the root of the problem, and that estimates of asymptotic null distributions in such settings can give a misleading picture of finite-sample performance. In particular, calibrating a test based on ${\widehat{\theta }}_n$ in a way that does not adapt to the implicit post-model-selection will be extremely inaccurate. This type of non-regularity occurs in various other settings as well, e.g., when a nuisance parameter is only defined under an alternative hypothesis (Davies, 1977), and when the parameter of interest under the null hypothesis is on the boundary of the parameter space (Andrews, 2000). McCloskey (2012) surveyed non-standard testing problems in econometrics, and introduced some Bonferroni-based size-correction methods designed to improve power. As far as we know, however, there is not yet a resolution of these issues for marginal screening.

In this paper we introduce an adaptive resampling test (ART) for marginal screening that adapts to the small sample behavior of ${\widehat{\theta }}_n$ in terms of a local model. Under local alternatives, we find an explicit representation of the asymptotic distribution of ${\widehat{\theta }}_n$ and construct a suitable bootstrap estimator of this distribution that is consistent, thus circumventing the non-regularity mentioned above. Under non-local alternatives, we show that the critical values obtained in this way agree asymptotically with those used by the oracle (who is given knowledge of k₀), so ART can be expected to provide good power as well.

Several new approaches to post-model selection inference for linear regression have been proposed in recent years. Meinshausen et al. (2009) introduced a random sample splitting procedure in the high-dimensional setting to obtain (conservative) Bonferroni-adjusted p-values following variable selection. Chatterjee and Lahiri (2011) developed a modified bootstrap method that provides an asymptotically valid confidence region for the regression parameters based on the lasso estimator; this method depends on the presence of at least one active predictor, so it is not applicable to marginal screening (under the null hypothesis there is no active predictor).

More relevant to marginal screening, the covariance test recently introduced by Lockhart et al. (2014) uses a forward stepwise lasso procedure to test for active predictors entering a sparse linear model under the assumption of normal errors. Also in the sparse linear model setting with normal errors, but further assuming that the predictors are nearly uncorrelated, Ingster et al. (2010) and Arias-Castro et al. (2011) have studied the detection boundary and optimality properties of general classes of multiple testing procedures (including Bonferroni and Higher Criticism). Berk et al. (2013) developed a valid method of post-model selection inference that is feasible for up to about p = 20 predictors, also assuming normal errors. In various sparse high-dimensional settings, Belloni et al. (2013), Bühlmann (2013), Zhang and Zhang (2014) and Ning and Liu (2015) have established asymptotically valid confidence intervals for a preconceived regression parameter after variable selection on the remaining predictors, but this does not apply to marginal screening (where no regression parameter is singled-out a priori).

This paper is organized as follows. We formulate the problem and discuss the issue of non-regularity in Section 2. In Section 3, we develop the ART procedure and establish the consistency of the underlying bootstrap. Simulation studies and applications to gene expression data and HIV drug resistance data are presented in Section 4. Concluding discussion appears in Section 5, and proofs are collected in the Appendix.

2 Marginal regression and non-regularity

Consider a scalar outcome Y and a p-dimensional vector of covariates X = (X₁, . . . , X_p)^T such that the marginal variance of each covariate is finite and non-zero. Marginal regression consists in using separate linear models to predict Y from each X_k. Let k₀ be the label of a covariate that maximizes the absolute correlation with Y :

$$k_0\in \arg \underset{k=1,\dots ,p}{\max }\mid \mathrm{Corr}(X_k,Y)\mid ,$$

and let α₀ + θ₀X_k0 be the best linear predictor based on X_k0, i.e.,

$$({\alpha }_0,{\theta }_0)=\arg \underset{\alpha ,\theta \in \mathbb{R}}{\min }E{(Y-\alpha -\theta X_{k_0})}^2=\left(EY-{\theta }_{0}E{X}_{k_0},\frac{\mathrm{Cov}(X_{k_0},Y)}{\mathrm{Var}\left(X_{k_0}\right)}\right).$$ (1)

We are interested in testing whether at least one of the covariates is correlated with Y, for which it suffices to check whether X_k0 and Y are correlated. This is equivalent to testing

$$H_0:{\theta }_0=0\text{versus}{H}_a:{\theta }_0\ne 0.$$

Given an iid sample of size n, let ${\widehat{\alpha }}_n$, ${\widehat{\theta }}_n$, and k̂_n be the least squares estimates of α₀, θ₀, and k₀, respectively:

$${\widehat{\alpha }}_n={\mathbb{P}}_{n}Y-{\widehat{\theta }}_n{\mathbb{P}}_n{X}_{{\widehat{k}}_0},{\widehat{\theta }}_n=\frac{\widehat{\mathrm{Cov}}(X_{{\widehat{k}}_n},Y)}{\widehat{\mathrm{Var}}\left(X_{{\widehat{k}}_n}\right)},{\widehat{k}}_n\in \arg \underset{k=1,\dots ,p}{\max }\mid \widehat{\mathrm{Corr}}(X_k,Y)\mid ,$$

where ${\mathbb{P}}_n$ is the empirical distribution, and the hats indicate sample versions. It is natural to base the test on ${\widehat{\theta }}_n$ but calibration is problematic because the distribution of $\sqrt{n}({\widehat{\theta }}_n-{\theta }_0)$ does not converge uniformly with respect to θ₀, as mentioned in the Introduction. The non-uniformity occurs in the neighborhood of θ₀ = 0. Specifically, there exists a bounded continuous function $h:\mathbb{R}\to \mathbb{R}$ such that $f_n\left({\theta }_0\right)\equiv \mathrm{E}h\left(\sqrt{n}({\widehat{\theta }}_n-{\theta }_0)\right)$ does not converge uniformly in any neighborhood of θ₀ = 0, despite converging pointwise. To see this, first note that under mild conditions

$$\sqrt{n}({\widehat{\theta }}_n-{\theta }_0)\stackrel{d}{\to }U\equiv \{\begin{array}{cc}{Z}_{k_0}∕V_{k_0}\hfill & \text{if}{\theta }_0\ne 0,\hfill \\ Z_K∕V_K\hfill & \text{if}{\theta }_0=0,\hfill \end{array}\phantom{\}}$$

where V_k = Var(X_k), $K=\arg {\max }_{k=1,\dots ,p}{Z}_k^2∕V_k$, and (Z₁, . . . , Z_p)^T is a mean-zero normal random vector with covariance matrix depending on parameters of the full linear model (this is a special case of Theorem 1 below). From the form of the distribution of U, we can choose h so that f_∞(θ₀) ≡ Eh(U) is discontinuous at θ₀ = 0 (this is the non-regularity mentioned in the Introduction). If f_n were to converge uniformly to f_∞ on some compact neighborhood of zero, we would have a contradiction because each f_n is continuous, and the uniform limit of a sequence of continuous functions on a compact interval is continuous.

To address this problem, in the next section we develop a formal test procedure (ART) inspired by work of Cheng (2008, 2015) concerning robust confidence intervals for non-linear regression parameters in the presence of weak-identifiability. Other variations of this approach have been used by Laber and Murphy (2011) to construct a confidence interval for the classification error, by Laber et al. (2014) in a sequential decision making problem, and by Laber and Murphy (2013) to provide robust confidence intervals for adaptive lasso. As already noted, the distribution of $\sqrt{n}({\widehat{\theta }}_n-{\theta }_0)$ does not converge uniformly in the neighborhood of θ₀ = 0, so its small sample behavior can be very far from normal when the true parameter is close to zero. Therefore an understanding of the asymptotic behavior of ${\widehat{\theta }}_n$ under local alternatives plays a crucial role in devising a suitable test, or more generally in providing robust confidence intervals for θ₀.

3 Adaptive resampling test

In this section, we develop the proposed ART procedure for detecting the presence of a significant predictor. The idea is to adapt to the inherent non-regular behavior of the post-model-selected estimator ${\widehat{\theta }}_n$ in a way that accurately captures its asymptotic behavior in $\sqrt{n}$-neighborhoods of the null hypothesis.

We frame the problem in terms of the general local linear model

$$Y={\alpha }_0+{\mathbf{X}}^T{\beta }_n+ϵ,$$ (2)

where ${\alpha }_0\in \mathbb{R}$, ${\beta }_n\in {\mathbb{R}}^p$, the noise ε has mean 0, finite variance, and is uncorrelated with X, and β_n = β₀ + n^−1/2b₀, where ${\mathit{b}}_0\in {\mathbb{R}}^p$ is the local parameter. The distributions of ε and X are assumed to be fixed, so only the distribution of Y depends on n (although we suppress n in the notation for Y). The relevant hypotheses are now

$$H_0:{\theta }_n=0\text{versus}{H}_a:{\theta }_n\ne 0,$$

where θ_n = Cov(X_kn, Y)/Var(X_kn) and k_n is the label of a component of X that maximizes absolute correlation with Y.

Our first result gives the asymptotic distribution of ${\widehat{\theta }}_n$. To state the result, we need the notation

$$\stackrel{‒}{k}\left(\mathit{b}\right)\equiv \arg \underset{k=1,\dots ,p}{\max }\mid \mathrm{Corr}(X_k,{\mathbf{X}}^T\mathit{b})\mid$$

for any $\mathit{b}\in {\mathbb{R}}^p$. Note that k_n = k̄(β_n) under the local model. If k₀ ≡ k̄(β₀) is unique (so β₀ ≠ = 0), then k_n → k₀, and θ_n is asymptotically bounded away from zero (a non-local alternative). On the other hand, if β₀ = 0 and k̄(b₀) is unique, then k_n = k̄(b₀); also θ_n is in the neighborhood of zero and represents a local alternative. Finally, if β₀ = b₀ = 0, then k_n is not well-defined and the null hypothesis θ_n = 0 holds. We need the uniqueness of the most active predictor k₀ (away from the null hypothesis), but this seems to be a very mild condition because the likelihood that there would be two or more predictors having exactly the same maximal correlation with Y seems remote in practice. Even in practice, as we will see in the simulation study, non-uniqueness of the maximally correlated predictor does not adversely affect power.

Theorem 1

Suppose that k₀ = k̄(b₀) is unique when β₀ ≠ 0, and k̄(b₀) is unique when β₀ = 0 and b₀ ≠ 0. Then, under the local model (2),

$$\sqrt{n}({\widehat{\theta }}_n-{\theta }_n)\stackrel{d}{\to }\{\begin{array}{cc}{Z}_{k_0}\left({\beta }_0\right)∕V_{k_0}\hfill & if{\beta }_0\ne 0,\hfill \\ Z_K\left(0\right)∕V_K+{(C_K∕V_K-C_{\stackrel{‒}{k}\left({\mathit{b}}_0\right)}∕V_{\stackrel{‒}{k}\left({\mathit{b}}_0\right)})}^T{\mathit{b}}_0\hfill & if{\beta }_0=0,\hfill \end{array}\phantom{\}}$$

where $K=\arg {\max }_{k=1,\dots ,p}{[Z_k\left(0\right)+C_k^T{\mathit{b}}_0]}^2∕V_k,C_k=\mathrm{Cov}(X_k,\mathbf{X})$, and ${\left(Z_k\left(\beta \right)\right)}_{k=1}^p$ is a mean-zero normal random vector with covariance matrix Σ(β) given by that of the random vector with components

$$\left({(\mathbf{X}-E\mathbf{X})}^T\beta -(X_k-E{X}_k)C_k^T\beta ∕V_k+ϵ\right)(X_k-E{X}_k),$$

for k = 1, . . . , p, and Σ(β₀) is assumed to exist.

The non-regularity at β₀ = 0 is explained by the dependence of the limiting distribution on the (non-identifiable) local parameter b₀. The limiting distribution is nevertheless continuous as a function of ${\mathit{b}}_0\in {\mathbb{R}}^p$ into the space of distribution functions (this is a simple consequence of Lemma 3 in the Appendix), and the convergence is uniform over compact subsets of ${\mathbb{R}}^p$, unlike the limiting behavior discussed in the previous section, so finite-sample accuracy should be less of an issue when designing a screening test using this result. On the other hand, naive resampling methods that do not take into account the local asymptotic behavior will fail to provide consistent estimates of the distribution of $\sqrt{n}({\widehat{\theta }}_n-{\theta }_n)$, as discussed in the Introduction for the non-local case.

To get around this problem, we decompose $\sqrt{n}({\widehat{\theta }}_n-{\theta }_n)$ in a way that isolates the possibility that β₀ ≠ 0 by comparing |T_n| to some threshold λ_n (to be specified later), where $T_n={\widehat{\theta }}_n∕s_n$ is the post-model-selected t-statistic and s_n is the standard error of the slope estimator when regressing Y on X_k̂n. Specifically,

$$\begin{array}{cc}\hfill \sqrt{n}({\widehat{\theta }}_n-{\theta }_n)=& \sqrt{n}({\widehat{\theta }}_n-{\theta }_n)1_{\mid T_n\mid >{\lambda }_n\text{or}{\beta }_0\ne 0}+\sqrt{n}({\widehat{\theta }}_n-{\theta }_n)1_{\mid T_n\mid \le {\lambda }_n,{\beta }_0=0}\hfill \\ \hfill =& \sqrt{n}({\widehat{\theta }}_n-{\theta }_n)1_{\mid T_n\mid >{\lambda }_n\text{or}{\beta }_0\ne 0}+\left[\frac{{\mathbb{Z}}_{n,{\widehat{k}}_n}+\widehat{\mathrm{Cov}}(X_{{\widehat{k}}_n},{\mathbf{X}}^T{\mathit{b}}_0)}{\widehat{\mathrm{Var}}\left(X_{{\widehat{k}}_n}\right)}-\frac{\mathrm{Cov}(X_{k_n},{\mathbf{X}}^T{\mathit{b}}_0)}{\mathrm{Var}\left(X_{k_n}\right)}\right]1_{\mid T_n\mid \le {\lambda }_n,{\beta }_0=0},\hfill \end{array}$$ (3)

where ${\mathbb{Z}}_{n,k}={\mathbb{G}}_n\left[ϵ(X_k-{\mathbb{P}}_n{X}_k)\right]$, ${\mathbb{G}}_n=\sqrt{n}({\mathbb{P}}_n-P_n)$ is the empirical process, and P_n is the distribution of (X, Y). It is clear that the nonparametric bootstrap is consistent for the first term in (3) if ${\lambda }_n=o\left(\sqrt{n}\right)$ and λ_n → ∞, since it is easily shown that P(|T_n| > λ_n) → 1_β0≠0. The second term is more problematic though because k̂_n does not converge in probability to k₀ when β₀ = 0. Denote the term in the square brackets by ${\mathbb{V}}_n\left(\mathit{b}\right)$, indexed by $\mathit{b}={\mathit{b}}_0\in {\mathbb{R}}^p$. Note that when this term is active (under β₀ = 0), ${\widehat{k}}_n={\mathbb{K}}_n\left({\mathit{b}}_0\right)$ and k_n = K̄(b₀), where

$${\mathbb{K}}_n\left(\mathit{b}\right)=\arg \underset{k=1,\dots ,p}{\max }\frac{{[{\mathbb{Z}}_{n,k}+\widehat{\mathbf{Cov}}(X_k,{\mathbf{X}}^T\mathit{b})]}^2}{\widehat{\mathrm{Var}}\left(X_k\right)}$$

and

$$\stackrel{‒}{K}\left(\mathit{b}\right)=\arg \underset{k=1,\dots ,p}{\max }\frac{{\left[\mathrm{Cov}(X_k,{\mathbf{X}}^T\mathit{b})\right]}^2}{\mathrm{Var}\left(X_k\right)},$$

so

$${\mathbb{V}}_n\left(\mathit{b}\right)=\frac{{\mathbb{Z}}_{n,{\mathbb{K}}_n\left(\mathit{b}\right)}+\widehat{\mathrm{Cov}}(X_{{\mathbb{K}}_n\left(\mathit{b}\right)},{\mathbf{X}}^T\mathit{b})}{\widehat{\mathrm{Var}}\left(X_{{\mathbb{K}}_n\left(\mathit{b}\right)}\right)}-\frac{\mathrm{Cov}(X_{\stackrel{‒}{K}\left(\mathit{b}\right)},{\mathbf{X}}^T\mathit{b})}{\mathrm{Var}\left(X_{\stackrel{‒}{K}\left(\mathit{b}\right)}\right)}.$$ (4)

All parts of ${\mathbb{V}}_n\left(\mathit{b}\right)$ are now seen to be smooth functions of ${\mathbb{P}}_n$, so it is reasonable to expect that a consistent bootstrap can be constructed by replacing ${\mathbb{P}}_n$ by its nonparametric bootstrap ${\mathbb{P}}_n^{\ast }$, and replacing P_n by ${\mathbb{P}}_n$. In such a construction, the event indicated in the second term of (3) is naturally replaced by the event that $\mid T_n^{\ast }\mid \le {\lambda }_n$ and |T_n| ≤ λ_n.

Here and throughout the paper, a superscript * is used to indicate the nonparametric bootstrap (sometimes called “bootrapping in pairs” in regression settings, to distinguish it from the residual bootstrap). The above arguments lead to our main result showing that $\sqrt{n}({\widehat{\theta }}_n-{\theta }_n)$ can indeed be consistently bootstrapped under the general local model. The precise definition of ${\mathbb{V}}_n^{\ast }$ is given at the start of the proof.

Theorem 2

Suppose all assumptions in Theorem 1 hold, and the tuning parameter λ_n satisfies ${\lambda }_n=o\left(\sqrt{n}\right)$ and λ_n → ∞ as n → ∞. Then, under the local model (2),

$$\sqrt{n}({\widehat{\theta }}_n^{\ast }-{\widehat{\theta }}_n)1_{\mid T_n^{\ast }\mid >{\lambda }_n\text{or}\mid T_n\mid >{\lambda }_n}+{\mathbb{V}}_n^{\ast }\left({\mathit{b}}_0\right)1_{\mid T_n^{\ast }\mid \le {\lambda }_n,\mid T_n\mid \le {\lambda }_n}$$

converges to the limiting distribution of $\sqrt{n}({\widehat{\theta }}_n-{\theta }_0)$ conditionally (on the data) in probability.

ART procedure

ART provides a bootstrap calibration for the test statistic $\sqrt{n}{\widehat{\theta }}_n$ based on a special case of the above theorem. Under H₀ we have the simplification ${\mathbb{V}}_n^{\ast }\left({\mathit{b}}_0\right)={\mathbb{V}}_n^{\ast }\left(0\right)$. For some nominal level γ, let c_l and c_u be the lower and upper γ/2 quantiles, respectively, of

$$A_n^{\ast }=\sqrt{n}({\widehat{\theta }}_n^{\ast }-{\widehat{\theta }}_n)1_{\mid T_n^{\ast }\mid >{\lambda }_n\text{or}\mid T_n\mid >{\lambda }_n}+{\mathbb{V}}_n^{\ast }\left(0\right)1_{\mid T_n^{\ast }\mid \le {\lambda }_n,\mid T_n\mid \le {\lambda }_n}.$$

If $\sqrt{n}{\widehat{\theta }}_n$ falls outside the interval [c_l, c_u], then we reject H₀ and conclude that there is at least one significant predictor.

Before applying ART, it is advisable to standardize all the variables X_k and Y (by sample mean and standard deviation), which has the advantage of making the procedure scale invariant (${\widehat{\theta }}_n$ is then the maximal sample correlation); our results naturally extend, but we develop the theory only for the unstandardized variables to keep the presentation simple.

Robust confidence intervals

The above theorem also allows the construction of a robust confidence interval for θ_n by treating b₀ as unknown, then finding the widest bootstrap quantiles over all b₀. Here by “robust” we mean asymptotically valid uniformly over b₀. For testing purposes, however, this approach would be too conservative and also computationally intensive (grid search over ${\mathbb{R}}^p$ is needed); for this reason, in ART we set b₀ = 0 under the null, so the critical values can be readily computed from $A_n^{\ast }$. In contrast, Laber and Murphy (2013) propose using almost sure bounds over their local parameter b₀ to find robust confidence intervals for adaptive lasso; this involves less computation than distributional bounds, but is still computationally intensive, and it produces more conservative confidence intervals than the distributional approach.

Choice of the tuning parameter λ_n

The above theorem requires that ${\lambda }_n=o\left(\sqrt{n}\right)$ and λ_n → ∞ as n → ∞. Under this condition, the thresholding provides a consistent pre-test (for θ_n = 0) with asymptotically negligible type I error rate: lim_n→∞ P(|T_n| > λ_n|θ_n = 0) = 0. On the other hand, if λ_n increases too quickly, the pre-test will be conservative. One simple choice would be to set ${\lambda }_n=\sqrt{a\log n}$, for some constant a > 0, but it is also desirable that λ_n increase with p, see Section 5 for discussion about the null limiting behavior of T_n as both p and n → ∞. To that end, note that by Theorem 1 in the special case that ε and X are independent, under θ_n = 0 (or b₀ = 0 and β₀ = 0) we have $T_n\stackrel{d}{\to }{\tilde{Z}}_K$, where $K=\arg {\max }_{k=1,\dots ,p}{\tilde{Z}}_k^2$, and (Z̃₁, . . . ,Z̃_p)^T is a vector of standard normal random variables. Thus, for any fixed λ > 0,

$$\mathrm{P}\left(\mid T_n\mid >\lambda \mid {\theta }_n=0\right)\to \mathrm{P}\left(\underset{k=1,\dots ,p}{\max }\mid {\tilde{Z}}_k\mid >\lambda \right)\le \sum _{k=1}^p\mathrm{P}\left(\mid {\tilde{Z}}_k\mid >\lambda \right).$$

Hence the pre-test type I error rate can be asymptotically controlled below level γ, without sacrificing consistency, by choosing

$${\lambda }_n=\max \{\sqrt{a\log n},\text{upper}\gamma ∕\left(2p\right)\text{-quantile of}N(0,1)\}.$$ (5)

In the simulation study below we describe a way of specifying the constant a via the double bootstrap, and this is used whenever we refer to ART in the sequel.

Forward stepwise ART

If we find a significant predictor using ART, it would be reasonable to continue applying the procedure in a forward stepwise fashion until no more significant predictors are detected. That is, in successive stages the residual $Y-{\widehat{\alpha }}_n-{\widehat{\theta }}_n{X}_{{\widehat{k}}_n}$ is treated as a new outcome variable and marginal regression carried out on the remaining predictors. Although it would be challenging to extend our theoretical results to this procedure, we find that in real data applications it performs well, and in a similar way to the covariance test of Lockhart et al. (2014), as we discuss in the HIV drug resistance example considered in the next section.

4 Numerical studies

In this section, we study the performance of the proposed ART procedure using simulated data, and give illustrations of the approach in two real data examples.

4.1 Finite sample simulations

We compare the performance of ART with four procedures that are commonly used for detecting the presence of a significant predictor:

Likelihood ratio test (LRT)

This test is based on assuming a full linear model involving all of the covariates, and is applicable when n > p. Under the null hypothesis, all the regression coefficients are zero. The reduction in the residual sum of squares is compared to the residual sum of squares for the full model using an F-ratio [see, e.g. Section 7.4 of Johnson and Wichern (2007)]. When the full linear model holds, it can be seen that both null and alternative hypotheses are identical to those used in ART.

Multiple testing with Bonferroni correction

As in ART, marginal linear models are used to predict Y from each X_k. A t-test with Bonferroni correction is then carried out to detect whether each regression coefficient is non-zero. The intersection of the p null hypotheses coincides with the null used in ART.

Centered percentile bootstrap (CPB)

This procedure is similar to ART, except $\sqrt{n}({\widehat{\theta }}_n^{\ast }-{\widehat{\theta }}_n)$ is used to estimate the upper and lower quantiles of $\sqrt{n}({\widehat{\theta }}_n-{\theta }_0)$, providing critical values for the test statistic $\sqrt{n}{\widehat{\theta }}_n$, see Efron and Tibshirani (1993).

Higher Criticism (HC)

This is a test originally proposed by John Tukey for determining the overall significance of a collection of independent p-values. We apply the statistic ${\mathrm{HC}}_N^{+}$ developed by Donoho and Jin (2004, 2015), which is expected to perform well if the predictors are nearly uncorrelated.

We consider three examples for the data generating model: i) Y = ε, ii) Y = X₁/4+ε, and iii) $Y={\sum }_{k=1}^p{\beta }_k{X}_k+ϵ$, where β₁ = . . . = β₅ = 0.15, β₆ = . . . = β₁₀ = −0.1, and β_k = 0 for k = 11, . . . , p. In the first example, there is no active predictor, in the second there is a single active predictor, and in the third there are 10 active predictors and the maximally correlated predictor is not unique. The covariate vector X is distributed as p-dimensional normal with each component X_k ~ N(0, 1), an exchangeable correlation structure Corr(X_j, X_k) = ρ for j ≠ k, where ρ takes values 0, 0.5 and 0.8, and the noise ε ~ N(0, 1) is independent of X.

We consider two sample sizes (n = 100 and 200), and five values of the dimension (p = 10, 50, 100, 150 and 200). A nominal 5% significance level is used throughout. The bootstrap sample size is taken as 1,000. To specify the threshold λ_n in ART the double bootstrap is implemented by generating 1,000 bootstrap estimates ${\widehat{\theta }}_n^{\ast }$, then choosing λ_n so that 5% of the ARTs (based on 1,000 nested bootstrap samples) with test statistic $\sqrt{n}({\widehat{\theta }}_n^{\ast }-{\widehat{\theta }}_n)$ reject.

Empirical rejection rates based on 1,000 Monte Carlo replications are reported in Figures 1–3. For model i), the figures provide type I error rates, which should be compared with the 5% nominal rate; for models ii) and iii), the figures provide the power of each test. The ART procedure has good control of the type I error rate throughout (compared to all the other methods), while consistently maintaining relatively high power. Comparing the results of models ii) and iii), non-uniqueness of the maximally correlated predictor has no adverse effect on the power of ART.

Figure 1.

Empirical rejection rates based on 1,000 samples generated from models i), ii) and iii) as the dimension ranges from p = 10 to p = 200, for n = 100 (top row) and n = 200 (bottom row), and ρ = 0.8.

Figure 3.

Empirical rejection rates as in Figure 1 except for independent predictors.

Bonferroni is highly conservative when ρ = 0.5 and 0.8, see the left panels of Figures 1 and 2. The CPB method is highly anti-conservative, with empirical type I error rates exceeding 15% for both sample sizes (and thus out of range for most of the panels on the left). The LRT effectively controls the type I error rate at around the nominal 5% level when it is applicable, but it has very low power compared with all the other methods, except under model iii) in the “classical case” of small numbers of predictors that are not highly correlated, see the right panels of Figures 2 and 3. Higher Criticism fails to control type I error except when the predictors are independent (Figure 3), in which case it is slightly anti-conservative and has excellent power under model iii), but very poor under model ii). That is, HC performs well (under zero correlation) when there are multiple active predictors, but not in the sparse case of only one active predictor. Except in the case of independent predictors, when Bonferroni is slightly better, ART outperforms all the competing procedures when both type I error and power are taken into account, and the improvement increases with the correlation between predictors.

Figure 2.

Empirical rejection rates as in Figure 1 except with lower correlation between predictors: ρ = 0.5.

4.2 Asymptotic power

In this section, we carry out a simulation study to assess the asymptotic power of ART compared with that of the Bonferroni procedure. The computational expense of implementing ART is high because of the double bootstrap, so our full simulation study of the previous section is only feasible for small sample sizes. Nevertheless, we are able to assess asymptotic power by making use of our results on the local model in Section 3.

Consider the local model Y = (n^−1/2b₀)X₁ + ε, where $b_0\in \mathbb{R}$. Here X and ε are generated in the same way as Section 4.1, but now we only consider ρ = 0.5. The local parameter b₀ takes the special form (b₀, 0, . . . , 0)^T, and we allow b₀ to vary over a grid in [0, 5], in increments of 0.5. We set β₀ = 0, b₀ = (b₀, 0, . . . , 0)^T and make use of the given covariance structure of X and the explicit form of the limiting distribution in Theorem 1 to generate a draws from the asymptotic distribution of $\sqrt{n}{\widehat{\theta }}_n$. Specifically, we carry out the following steps:

1. For each value of b₀ on the grid, take 5,000 draws from the limiting distribution of $\sqrt{n}({\widehat{\theta }}_n-{\theta }_0)$ given in Theorem 1 (this distribution only depends on b₀ and the given distribution of (X, Y )), then add b₀ to obtain draws from the limiting distribution of $\sqrt{n}{\widehat{\theta }}_n$. Based on these draws, we can obtain the (approximate) rejection rate of the test statistic $\sqrt{n}{\widehat{\theta }}_n$ for any given rejection region. In particular, the asymptotic rejection rate of ART (for any given b₀ on the grid) can be calculated by referring to the rejection rate corresponding to the particular critical values c_l and c_u generated by ART.

2. To assess the asymptotic power of ART at each given b₀, we generate 10 independent large samples (with n = 5,000) from the local model, find c_l and c_u for each sample, and display in a boxplot the corresponding asymptotic rejection rates (using the results of step 1).

3. For comparison, we also plot the asymptotic power of the Bonferroni procedure, which is approximated using 1,000 samples each of size n = 5,000.

The results are presented in Figure 4 for p = 10 and 50. The main source of variation within each boxplot is due to randomness over the 10 independent samples drawn from the local model, rather than bootstrap randomness (in view of bootstrap consistency and the large sample size n = 5,000). The median of each boxplot provides a suitable reference point to compare with the asymptotic power of Bonferroni (indicated by the circle). Note that ART provides accurate control of asymptotic type I error, and, as expected, Bonferroni is slightly conservative. In terms of median power, ART always outperforms Bonferroni, and can provide an additional 25% power (e.g., at b₀ = 3 for p = 10, and at b₀ = 3.5 for p = 50).

Figure 4.

Asymptotic type I error and power of ART (box plots) compared with Bonferroni (circles) as a function of the local parameter b₀, for p = 10 and 50, ρ = 0.5, calculated using steps 1–3 in Section 4.2.

The cost of implementing the double bootstrap part of ART makes it prohibitive to extend the results in Figure 4 to larger p, but if we fix λ_n, then it becomes practical to run the simulations for p = 1000. Figure 5 shows how the asymptotic power of ART compares with Bonferroni as the constant a used to specify λ_n takes values 2, 4, 5 and 8 (the corresponding λ_n are 4.3, 6.1, 6.8 and 8.6). Note that as a increases (going from one panel to the next), ART becomes more stable and provides more accurate type I error control, but the overall power decreases. At small values of a, ART behaves like the CPB, which is anti-conservative (as we have already seen in the previous section), whereas at larger values the influence of CPB is diluted. For the CPB (which corresponds to setting λ_n = 0), the plot (not shown) appears very similar to that for a = 2; also, for a > 8 the plots appear very similar to a = 8. The best choice of a, therefore, is a trade-off between type I error control and power; comparing with Figure 4, ART with double bootstrapping appears to achieve a satisfactory balance in this regard. Also note that, even at the largest value a = 8, ART can provide an additional 20% power over Bonferroni, and thus outperform Bonferroni by a considerable margin in high-dimensional settings as well, at least when there is a high degree of correlation among the components of X.

Figure 5.

Asymptotic type I error and power of ART compared with Bonferroni for p = 1,000 and ρ = 0.5, where ART is implemented using a fixed threshold λ_n specified by a = 2, 4, 5, 8, and each box plot is based on 20 independent replications with n = 10,000.

4.3 Gene expression example

We consider gene expression profiles from the tumors of n = 156 patients diagnosed with a common type of adult brain cancer (glioblastoma), collected as part of the Cancer Genome Atlas pilot project (TCGA, 2008). Our analysis is based on log gene expression levels X at p = 181 loci along chromosome 1. We are interested in detecting the presence of a gene that is significantly related to log-survival time Y.

We compare the results from applying the Bonferroni, CPB and ART procedures; LRT is not applicable since p > n. The three methods yield very different p-values. The smallest Bonferroni adjusted p-value is 40.8%, suggesting that no gene is significantly related to Y. The CPB and ART p-values are 3.2% and 17.2%, respectively, from 1000 bootstrap samples. Figure 6 shows how these p-values are calculated. Thus the CPB method suggests the presence of a significant genetic effect, whereas ART does not.

Figure 6.

Gene expression example. Left panel: histogram of $\sqrt{n}({\widehat{\theta }}_n^{\ast }-{\widehat{\theta }}_n)$ showing that the two-sided CPB p-value is 3.2%. Right panel: histogram of $A_n^{\ast }$ showing that the two-sided ART p-value is 17.2%.

4.4 HIV drug resistance example

Our second example uses data from the HIV Drug Resistance Database (2014), an important public resource for understanding how HIV-1 mutation patterns cause resistance to antiretroviral drugs (Rhee et al., 2002). We will compare our results with those of Lockhart et al. (2014), who applied their covariance test to data on the susceptibility (a measure of drug resistance) of the nucleotide reverse transcriptase inhibitor lamivudine (3TC). We code susceptibility on a log-scale (Y), and each predictor X_j is taken as indicating the presence/absence of a mutation at a given sequence position. The viral sequence positions are indexed by j. Excluding missing data and rare mutations resulted in data on p = 103 positions and a total of 1266 isolates.

We randomly split the data 50 times into a training set of size n = 126 and a test set of size 1140. For each split, we carry out 20 steps of forward stepwise ART and standard forward stepwise regression using the training data, and calculate the corresponding prediction error (including all previously selected variables) using the test data. The left panel of Figure 7 shows the training data p-values (mean ± SD) for the newly entered predictor at each step, over the 50 random splits, and the right panel shows the corresponding prediction errors (mean ± SD). Forward stepwise ART detects one very highly significant mutation, but no more, as confirmed by the test set error plot, and this result is roughly consistent with the findings of Lockhart et al. (2014). Standard forward stepwise regression picks out at least 10 mutations, but there is no improvement in test set error after the first predictor enters the model; moreover, the test error almost exactly coincides with ART.

Figure 7.

HIV drug resistance example. Left panel: training set p-values (mean ± SD) over 50 random splits of the data for forward stepwise ART (solid line), standard forward stepwise regression (dash-dot line) and the 0.05 alpha level (dotted). Right panel: test set error for the corresponding models (including all previously selected variables); the two lines are almost indistinguishable.

5 Discussion

In this paper we have developed an adaptive resampling test (ART) for detecting the existence of a significant predictor, X_k0, from among predictors X₁, . . . , X_p. The procedure is designed to adjust to the non-regular limiting behavior of the estimated marginal regression coefficient ${\widehat{\theta }}_n$ of the selected predictor. This is done by using a thresholded version of the bootstrap that adapts to the non-regularity: if there is at least one significant predictor, it reduces to a centered percentile bootstrap, otherwise it mimics the local (non-uniform) asymptotic behavior of ${\widehat{\theta }}_n$. We have shown that in simulation studies, ART performs favorably compared with standard methods such as Bonferroni, but also compared with more sophisticated methods such as Higher Criticism. The advantage of ART may stem from it being designed to take into account correlations between predictors, while also avoiding distributional assumptions (the nonparametric bootstrap steps in ART are essentially distribution free). We have restricted attention to linear models, but our approach has much wider applicability (e.g., generalized linear models, quantile regression, and censored time-to-event outcomes), and these will be studied in future papers.

Although our simulation results suggest that ART is useful and remarkably stable in “large p, small n” settings, the asymptotic theory that we have used to calibrate ART relies on assuming a fixed p, with n tending to infinity. In view of the conservative nature of the Bonferroni procedure in high-dimensional settings, there is a pressing need for more powerful tests in this area. In future work it would be of interest to develop the asymptotic theory of ART for the case of p growing with n, although this would be very challenging. As far as we know, formal testing procedures that provably control FWER and adjust to non-regularity under diverging p are not yet available, except for Higher Criticism in the case that the predictors are nearly uncorrelated, as established by Ingster et al. (2010) and Arias-Castro et al. (2011). In the only other instance we know of, under the strong assumption that X₁, . . . , X_p, Y are iid N(0, 1), results of Cai and Jiang (2012) can be used to find the weak limit of ${\widehat{\rho }}_n={\max }_{k=1,\dots ,p}\mid \widehat{\mathrm{Corr}}(X_k,Y)\mid$ and thus devise an asymptotically correct calibration: if p = p_n → ∞ at sub-exponential rate, log(p)/n → 0, then ${\widehat{\rho }}_n{\to }_{p}0$ and $n{\widehat{\rho }}_n^2-2\log p+\log \log p\stackrel{d}{\to }F$ where $F\left(y\right)=e^{-e^{-y∕2}∕\left(2\sqrt{\pi }\right)}$. In the super-exponential case, log(p)/n → ∞, then ${\widehat{\rho }}_n{\to }_{p}1$ and there is a similar weak limit.

Another interesting direction for future work would be to study the forward stepwise version of ART discussed in Section 3. Modifications to ART when applied stepwise in this way would be needed to adjust for the implicit dependence among the new outcomes. By repeating such a procedure until no more significant predictors are detected, the aim would be to correctly identify all active predictors.

Appendix: Proofs

Proof of Theorem 1

For k = 1, . . . , p, let $({\widehat{\alpha }}_k,{\widehat{\theta }}_k)=\arg {\min }_{(\alpha ,\theta )}{\mathbb{P}}_n{(Y-\alpha -\theta X_k)}^2$. Then ${\widehat{k}}_n=\arg {\min }_{k=1,\dots ,p}{\mathbb{P}}_n{(Y-{\widehat{\alpha }}_k-{\widehat{\theta }}_k{X}_k)}^2$ and $({\widehat{\alpha }}_n,{\widehat{\theta }}_n)=({\widehat{\alpha }}_{{\widehat{k}}_n},{\widehat{\theta }}_{{\widehat{k}}_n})$. It is easy to verify that ${\widehat{\alpha }}_k={\mathbb{P}}_n(Y-{\widehat{\theta }}_k{X}_k)$,

$$\begin{array}{cc}\hfill \sqrt{n}{\widehat{\theta }}_k=& \frac{\sqrt{n}\widehat{\mathrm{Cov}}(X_k,Y)}{\widehat{\mathrm{Var}}\left(X_k\right)}\hfill \\ \hfill =& \frac{\sqrt{n}\widehat{\mathrm{Cov}}(X_k,{\mathbf{X}}^T){\beta }_n+{\mathbb{G}}_n\left[ϵ(X_k-{\mathbb{P}}_n{X}_k)\right]}{\widehat{\mathrm{Var}}\left(X_k\right)}\hfill \\ \hfill =& \frac{({\mathbb{G}}_n{X}_k{\mathbf{X}}^T-P_n{X}_k{\mathbb{G}}_n{\mathbf{X}}^T-{\mathbb{G}}_n{X}_k{\mathbb{P}}_n{\mathbf{X}}^T){\beta }_n}{\widehat{\mathrm{Var}}\left(X_k\right)}+\frac{{\mathbb{G}}_n\left[ϵ(X_k-P_n{X}_k)\right]-{\mathbb{P}}_nϵ{\mathbb{G}}_n{X}_k+\sqrt{n}\mathrm{Cov}(X_k,{\mathbf{X}}^T){\beta }_n}{\widehat{\mathrm{Var}}\left(X_k\right)},\hfill \end{array}$$ (6)

where P_n is the distribution of (Y, X), and the mean residual squared error

$${\widehat{R}}_k\equiv {\mathbb{P}}_n{[Y-{\widehat{\alpha }}_k-{\widehat{\theta }}_k{X}_k]}^2=\widehat{\mathrm{Var}}\left(Y\right)-\widehat{\mathrm{Var}}\left(X_k\right){\widehat{\theta }}_k^2.$$ (7)

The result then follows immediately from the following two lemmas. The first lemma verifies the oracle property for marginal regression under the assumption that there is at least one active predictor; the proof is included for completeness. The second lemma gives the (non-regular) asymptotic behavior of ${\widehat{\theta }}_n$ when there are no active predictors.

Lemma 1

If all conditions in Theorem 1 hold and β₀ ≠ 0, then ${\widehat{k}}_n\stackrel{a.s.}{\to }{k}_0$ and $\sqrt{n}({\widehat{\theta }}_n-{\theta }_n)\stackrel{d}{\to }{Z}_{k_0}\left({\beta }_0\right)∕V_{k_0}$, where Z_k0 is defined in Theorem 1.

Proof

Denote R̂ ≡ (R̂₁, . . . ,R̂_p)^T. When β₀ ≠ 0, Var(X^T β₀) > 0. By the SLLN

$$\frac{\widehat{\mathrm{Var}}\left(Y\right)-\widehat{\mathbf{R}}}{\mathrm{Var}\left({\mathbf{X}}^T{\beta }_0\right)}\stackrel{a.s.}{\to }{\left({\mathrm{Corr}}^2(X_1,{\mathbf{X}}^T{\beta }_0),\dots ,{\mathrm{Corr}}^2(X_p,{\mathbf{X}}^T{\beta }_0)\right)}^T.$$

Since ${\widehat{k}}_n=\arg {\max }_{k=1,\dots ,p}[\widehat{\mathrm{Var}}\left(Y\right)-{\widehat{R}}_k]∕\mathrm{Var}\left({\mathbf{X}}^T{\beta }_0\right)$ and Corr²(X_k, X^Tβ₀) is maximized at k = k₀, it follows immediately that ${\widehat{k}}_n\stackrel{a.s.}{\to }{k}_0$.

Next, denote X̂ = X_k̂n and X_n = X_kn. Since ${\mathbb{P}}_n[Y-{\mathbb{P}}_{n}Y-{\widehat{\theta }}_n(\widehat{X}-{\mathbb{P}}_n\widehat{X})]\widehat{X}=0$ and Y = α₀ + X^Tβ_n + ε, we have

$$\begin{array}{cc}\hfill \sqrt{n}({\widehat{\theta }}_n-{\theta }_n)\widehat{\mathrm{Var}}\left(\widehat{X}\right)=& \sqrt{n}\widehat{\mathrm{Cov}}(\widehat{X},{\mathbf{X}}^T){\beta }_n+\sqrt{b}{\mathbb{P}}_n\left(ϵ(\widehat{X}-{\mathbb{P}}_n\widehat{X})\right)-\sqrt{n}\widehat{\mathrm{Var}}\left(\widehat{X}\right)\frac{\mathrm{Cov}{(X_n,\mathbf{X})}^T{\beta }_n+\mathrm{Cov}(X_n,ϵ)}{\mathrm{Var}\left(X_n\right)}\hfill \\ \hfill =& \sqrt{n}\widehat{\mathrm{Cov}}(X_{k_0},{\mathbf{X}}^T){\beta }_n+\sqrt{n}{\mathbb{P}}_n\left(ϵ(X_{k_0}-{\mathbb{P}}_n{X}_{k_0})\right)-\sqrt{n}\widehat{\mathrm{Var}}\left(X_{k_0}\right)\frac{\mathrm{Cov}{(X_{k_0},\mathbf{X})}^T{\beta }_n+\mathrm{Cov}(X_{k_0},ϵ)}{\mathrm{Var}\left(X_{k_0}\right)}+o_{P_n}\left(1\right)\hfill \\ \hfill =& {\mathbb{G}}_n\left[\left(ϵ+{(\mathbf{X}-P_n\mathbf{X})}^T{\beta }_0-\frac{\mathrm{Cov}{(X_{k_0},\mathbf{X})}^T{\beta }_0}{\mathrm{Var}\left(X_{k_0}\right)}(X_{k_0}-P_n{X}_{k_0})\right)(X_{k_0}-P_n{X}_{k_0})\right]+o_{P_n}\left(1\right),\hfill \end{array}$$

where the second equality uses ${\widehat{k}}_n\stackrel{a.s}{\to }{k}_0$ and k_n → k₀ as n → ∞, and the third equality follows from the LLN and Cov(ε, X_k0) = 0. Similarly, $\widehat{\mathrm{Var}}\left(\widehat{X}\right)\stackrel{P_n}{\to }{V}_{k_0}\equiv \mathrm{Var}\left(X_{k_0}\right)$. The proof is completed using Slutsky's lemma and the CLT.

Lemma 2

If all conditions in Theorem 1 hold and β₀ = 0, then $\sqrt{n}({\widehat{\theta }}_n-{\theta }_n)\stackrel{d}{\to }{Z}_K\left(0\right)∕V_K+{(C_K∕V_K-C_{\stackrel{‒}{k}\left({\mathit{b}}_0\right)}∕V_{\stackrel{‒}{k}\left({\mathit{b}}_0\right)})}^T{\mathit{b}}_0$.

Proof

Since (Z₁(0), . . . , Z_p(0))^T is a normal random vector and |Corr(X_j, X_k)| < 1 for j ≠ k, it is easy to see that

$$\frac{{(Z_j\left(0\right)+C_j^T{\mathit{b}}_0)}^2}{V_j}\ne \frac{{(Z_k\left(0\right)+C_k^T{\mathit{b}}_0)}^2}{V_k}\text{for any}j\ne k\mathrm{a}.\mathrm{s}.$$ (8)

So K is unique a.s.

Denote $\widehat{\theta }={({\widehat{\theta }}_1,\dots ,{\widehat{\theta }}_p)}^T$. Note that when β₀ = 0, $\sqrt{n}{\beta }_n={\mathit{b}}_0$. By the CLT and Slutsky's lemma, we see from (6) that

$$\sqrt{n}\widehat{\theta }\stackrel{d}{\to }{\left(\frac{Z_1\left(0\right)+C_1^T{\mathit{b}}_0}{V_1},\dots ,\frac{Z_p\left(0\right)+C_p^T{\mathit{b}}_0}{V_p}\right)}^T.$$

From (7), we have

$$n[\widehat{\mathrm{Var}}\left(Y\right)-\widehat{\mathbf{R}}]=\left(\sqrt{n}\widehat{\theta }\right)\odot \left(\sqrt{n}\widehat{\theta }\right)\odot {\left(\widehat{\mathrm{Var}}\left(X_1\right),\dots ,\widehat{\mathrm{Var}}\left(X_p\right)\right)}^T,$$

where ☉ denotes the elementwise (Hadamard) product, so, by the continuous mapping theorem and Slutsky's lemma,

$$\left(\begin{array}{c}\hfill \sqrt{n}\widehat{\theta }\hfill \\ \hfill n\left[\widehat{\mathrm{Var}}\left(Y\right)-\widehat{\mathbf{R}}\right]\hfill \end{array}\right)\stackrel{d}{\to }\left(\begin{array}{c}\hfill {\left((Z_1\left(0\right)+C_1^T{\mathit{b}}_0)∕V_1,\dots ,(Z_p\left(0\right)+C_p^T{\mathit{b}}_0)∕V_p\right)}^T\hfill \\ \hfill {\left({(Z_1\left(0\right)+C_1^T{\mathit{b}}_0)}^2∕V_1,\dots ,{(Z_p\left(0\right)+C_p^T{\mathit{b}}_0)}^2∕V_p\right)}^T\hfill \end{array}\right).$$

Define h(t) = (1_{arg maxktk=1}, . . . , 1_{arg maxktk=p})^T, where $\mathbf{t}={(t_1,\dots ,t_p)}^T\in {\mathbb{R}}^p$. Note that h is continuous at t if arg max_k t_k is unique. Thus, using (8) and since $\sqrt{n}{\widehat{\theta }}_n=\sqrt{n}{\widehat{\theta }}^{T}h\left(n[\widehat{\mathrm{Var}}\left(Y\right)-\widehat{\mathbf{R}}]\right)$, the result follows by applying the continuous mapping theorem to the above display.

Lemma 3

Let Z be a p-dimensional random vector and $f:{\mathbb{R}}^{2p}\to {\mathbb{R}}^p$ a function such that f(z, ·) is continuous for every $\mathbf{z}\in {\mathbb{R}}^p$, and f(Z, b)_j ≠ f(Z, b)_k a.s. for all j ≠ k and $\mathit{b}\in {\mathbb{R}}^p$. Then K(b) ≡ arg max_k=1...,p f(Z, b)_k is unique a.s. Also, if b_l → b₀, then K(b_l) = K(b₀) for l sufficiently large a.s.

The proof is omitted. An immediate consequence of this lemma is the continuity of the limiting distribution in Theorem 1 as a function of b₀; this is seen by setting $f{(z_1,\dots ,z_p,\mathit{b})}_k={(z_k+C_k^T\mathit{b})}^2∕V_k$ for k = 1, . . . , p, and using (8).

Proof of Theorem 2

The notation ${\widehat{\theta }}_n^{\ast }$ and ${\widehat{k}}_n^{\ast }$ means that ${\widehat{\theta }}_n$ and k̂_n are based on n iid observations taken from ${\mathbb{P}}_n$. The bootstrapped process ${\mathbb{V}}_n^{\ast }\left(\mathbf{b}\right)$ in the statement of the theorem is defined by re-expressing (4), along with K̄(b) and ${\mathbb{K}}_n\left(\mathbf{b}\right)$, in terms of P_n and ${\mathbb{P}}_n$ operating on functions of (X, Y), then replacing P_n by ${\mathbb{P}}_n$ and ${\mathbb{P}}_n$ by ${\mathbb{P}}_n^{\ast }$ throughout. In the case of ${\mathbb{Z}}_{n,k}$ in which ε is not observed, we also replace ε by ${\widehat{ϵ}}_n={\widehat{ϵ}}_n(\mathbf{X},Y)\equiv Y-{\widehat{\alpha }}_n-{\widehat{\theta }}_n\widehat{X}$, resulting in

$${\mathbb{Z}}_{n,k}^{\ast }={\mathbb{G}}_n^{\ast }\left[{\widehat{ϵ}}_n(X_k-{\mathbb{P}}_n^{\ast }{X}_k)\right]={\mathbb{G}}_n^{\ast }\left[{\widehat{ϵ}}_n{X}_k\right]-\left[{\mathbb{G}}_n^{\ast }{\widehat{ϵ}}_n\right]\left[{\mathbb{P}}_n^{\ast }{X}_k\right]$$ (9)

where ${\mathbb{G}}_n^{\ast }=\sqrt{n}({\mathbb{P}}_n^{\ast }-{\mathbb{P}}_n)$ is the bootstrapped empirical process. As is conventional in empirical process theory, ${\mathbb{P}}_n^{\ast }$, ${\mathbb{P}}_n$ and P_n are assumed to operate only on functions that are defined on (X, Y), explaining why ${\mathbb{P}}_n^{\ast }{X}_k$ can be separated in the above display.

Let E^M denote expectation conditional on the data, and let P^M be the corresponding probability measure. We will show that $1_{\mid T_n^{\ast }\mid >{\lambda }_n\text{or}\mid T_n\mid >{\lambda }_n}\stackrel{P^M}{\to }{1}_{{\beta }_0\ne 0}$ and $1_{\mid T_n^{\ast }\mid \le {\lambda }_n}{1}_{\mid T_n\mid \le {\lambda }_n}\stackrel{P^M}{\to }{1}_{{\beta }_0=0}$ conditionally (on the data) in probability. This together with Lemmas 4 and 5 below implies the result.

For k = 1, . . . , p, the bootstrapped marginal regression coefficient ${\widehat{\theta }}_k^{\ast }$ satisfies

$$\begin{array}{cc}\hfill \sqrt{n}{\widehat{\theta }}_k^{\ast }=& \frac{\sqrt{n}[{\mathbb{P}}_n^{\ast }{X}_{k}Y-\left({\mathbb{P}}_n^{\ast }{X}_k\right)\left({\mathbb{P}}_n^{\ast }Y\right)]}{{\mathbb{P}}_n^{\ast }{X}_k^2-{\left({\mathbb{P}}_n^{\ast }{X}_k\right)}^2}\hfill \\ \hfill =& \frac{{\mathbb{G}}_n^{\ast }{X}_{k}Y-{\mathbb{G}}_n^{\ast }{X}_k{\mathbb{P}}_n^{\ast }Y-\left({\mathbb{P}}_n{X}_k\right)\left({\mathbb{G}}_n^{\ast }Y\right)+\sqrt{n}[{\mathbb{P}}_n{X}_{k}Y-\left({\mathbb{P}}_n{X}_k\right)\left({\mathbb{P}}_{n}Y\right)]}{{\mathbb{P}}_n^{\ast }{X}_k^2-{\left({\mathbb{P}}_n^{\ast }{X}_k\right)}^2}\hfill \\ \hfill =& \frac{{\mathbb{G}}_n^{\ast }{X}_{k}Y-{\mathbb{G}}_n^{\ast }{X}_k{\mathbb{P}}_n^{\ast }Y-\left({\mathbb{P}}_n{X}_k\right)\left({\mathbb{G}}_n^{\ast }Y\right)+\sqrt{n}{\widehat{\theta }}_k[{\mathbb{P}}_n{X}_k^2-{\left({\mathbb{P}}_n{X}_k\right)}^2]}{{\mathbb{P}}_n^{\ast }{X}_k^2-{\left({\mathbb{P}}_n^{\ast }{X}_k\right)}^2}.\hfill \end{array}$$ (10)

When β₀ = 0, by Lemma 2 and the condition that λ_n → ∞ as n → ∞, we have $T_n^{\ast }∕{\lambda }_n\stackrel{P^M}{\to }0$ in probability. When β₀ ≠ 0, it is easy to verify that $\mid {\theta }_n\mid \to \mid C_{k_0}^T{\beta }_0\mid ∕V_{k_0}$, which is positive under the condition that k₀ is unique. Thus

$$\begin{array}{cc}\hfill P^M(\mid T_n^{\ast }\mid \le {\lambda }_n)& =P^M(\mid ({\widehat{\theta }}_n^{\ast }-{\widehat{\theta }}_n)+({\widehat{\theta }}_n-{\theta }_n)+{\theta }_n\mid \le {\lambda }_n{s}_n^{\ast })\hfill \\ \hfill & \le P^M(\mid {\theta }_n\mid \le {\lambda }_n{s}_n^{\ast }+\mid {\widehat{\theta }}_n^{\ast }-{\widehat{\theta }}_n\mid +\mid {\widehat{\theta }}_n-{\theta }_n\mid )\hfill \end{array}$$

tends to zero in probability when β₀ ≠ 0, where the convergence follows from Lemma 1, Lemma 4 (below) and the condition that ${\lambda }_n=o\left(\sqrt{n}\right)$. Hence

$$\begin{array}{cc}\hfill E^M\mid 1_{\mid T_n^{\ast }\mid \le {\lambda }_m}-1_{{\beta }_0=0}\mid =& E^M\mid 1_{\mid T_n^{\ast }\mid >{\lambda }_n}-1_{{\beta }_0\ne 0}\mid \hfill \\ \hfill =& P^M(\mid T_n^{\ast }\mid >{\lambda }_n,{\beta }_0=0)+P^M(\mid T_n^{\ast }\mid \le {\lambda }_n,{\beta }_0\ne 0)\hfill \\ \hfill =& P^M(\mid T_n^{\ast }\mid >{\lambda }_n\mid {\beta }_0=0)1_{{\beta }_0=0}+P^M(\mid T_n^{\ast }\mid \le {\lambda }_n\mid \beta \ne 0)1_{{\beta }_0\ne 0}\hfill \end{array}$$

tends to zero in probability. This implies that $1_{\mid T_n^{\ast }\mid >{\lambda }_n}\stackrel{P^M}{\to }{1}_{{\beta }_0\ne 0}$ and $1_{\mid T_n^{\ast }\mid \le {\lambda }_n}\stackrel{P^M}{\to }{1}_{{\beta }_0=0}$ conditionally in probability. Since 1_|Tn|≤λn converges to 1_β0=0 in probability, the result follows from Slutsky's lemma.

Lemma 4

If the conditions in Theorem 1 hold and β₀ ≠ 0, then ${\widehat{k}}_n^{\ast }\stackrel{P^M}{\to }{k}_0$ conditionally (on the data) a.s. and $\sqrt{n}({\widehat{\theta }}_n^{\ast }-{\widehat{\theta }}_n)\stackrel{d}{\to }{Z}_{k_0}\left({\beta }_0\right)∕V_{k_0}$ conditionally (on the data) in probability.

Proof

It follows from (10), the SLLN and Slutsky's lemma that, when β₀ ≠ 0,

$${\widehat{\mathrm{Var}}}^{\ast }\left(X_k\right){\widehat{\theta }}_k^{\ast }=n^{-1∕2}[{\mathbb{G}}_n^{\ast }{X}_{k}Y-{\mathbb{G}}_n^{\ast }{X}_k{\mathbb{P}}_n^{\ast }Y-\left({\mathbb{P}}_n{X}_k\right)\left({\mathbb{G}}_n^{\ast }Y\right)]+{\widehat{\theta }}_k[{\mathbb{P}}_n{X}_k^2-{\left({\mathbb{P}}_n{X}_k\right)}^2]\stackrel{P^M}{\to }{C}_k^T{\beta }_0$$

and ${\widehat{\theta }}_k^{\ast }\stackrel{P^M}{\to }{C}_k^T{\beta }_0∕V_k$ a.s. for k = 1, . . . , p. Denote the bootstrap mean squared error

$${\widehat{R}}_k^{\ast }\equiv {\mathbb{P}}_n^{\ast }{[Y-{\widehat{\alpha }}_k^{\ast }-{\widehat{\theta }}_k^{\ast }{X}_k]}^2={\widehat{\mathrm{Var}}}^{\ast }\left(Y\right)-{\left({\widehat{\theta }}_k^{\ast }\right)}^2{\widehat{\mathrm{Var}}}^{\ast }\left(X_k\right),$$

where ${\widehat{\mathrm{Var}}}^{\ast }\left(Y\right)={\mathbb{P}}_n^{\ast }{Y}^2-{\left({\mathbb{P}}_n^{\ast }Y\right)}^2$ and ${\widehat{\mathrm{Var}}}^{\ast }\left(X_k\right)={\mathbb{P}}_n^{\ast }{X}_k^2-{\left({\mathbb{P}}_n^{\ast }{X}_k\right)}^2$. Then we can write

$${\widehat{k}}_n^{\ast }=\arg \underset{k=1,\dots ,p}{\max }\frac{{\widehat{\mathrm{Var}}}^{\ast }\left(Y\right)-{\widehat{R}}_k^{\ast }}{\mathrm{Var}\left({\mathbf{X}}^T{\beta }_0\right)}=\arg \underset{k=1,\dots ,p}{\max }\frac{{\left({\widehat{\theta }}_k^{\ast }\right)}^2{\widehat{\mathrm{Var}}}^{\ast }\left(X_k\right)}{\mathrm{Var}\left({\mathbf{X}}^T{\beta }_0\right)}$$

since the denominator plays no role. By Slutsky's lemma

$$\frac{{\left({\widehat{\theta }}_k^{\ast }\right)}^2{\widehat{\mathrm{Var}}}^{\ast }\left(X_k\right)}{\mathrm{Var}\left({\mathbf{X}}^T{\beta }_0\right)}\stackrel{P^M}{\to }{\mathrm{Corr}}^2(X_k,{\mathbf{X}}^T{\beta }_0)$$

a.s. for k = 1, . . . , p, so we obtain

$$\begin{array}{cc}\hfill P^M({\widehat{k}}_n^{\ast }\ne k_0)& =P^M\left(\bigcup _{k:k\ne k_0}\left\{\frac{{\left({\widehat{\theta }}_{k_0}^{\ast }\right)}^2{\widehat{\mathrm{Var}}}^{\ast }\left(X_{k_0}\right)}{\mathrm{Var}\left({\mathbf{X}}^T{\beta }_0\right)}\le \frac{{\left({\widehat{\theta }}_k^{\ast }\right)}^2{\widehat{\mathrm{Var}}}^{\ast }\left(X_k\right)}{\mathrm{Var}\left({\mathbf{X}}^T{\beta }_0\right)}\right\}\right)\hfill \\ \hfill & \le \sum _{k:k\ne k_0}{P}^M\left(\frac{{\left({\widehat{\theta }}_{k_0}^{\ast }\right)}^2{\widehat{\mathrm{Var}}}^{\ast }\left(X_{k_0}\right)}{\mathrm{Var}\left({\mathbf{X}}^T{\beta }_0\right)}\le \frac{{\left({\widehat{\theta }}_k^{\ast }\right)}^2{\widehat{\mathrm{Var}}}^{\ast }\left(X_{k_0}\right)}{\mathrm{Var}\left({\mathbf{X}}^T{\beta }_0\right)}\right)\hfill \\ \hfill & \to 0\mathrm{a}.\mathrm{s}.,\hfill \end{array}$$

where the convergence follows from the condition that k₀ is unique when β₀ ≠ 0.

Recall that ${\widehat{ϵ}}_n\equiv Y-{\widehat{\alpha }}_n-{\widehat{\theta }}_n\widehat{X}$, where X̂ ≡ X_k̂n. Note that ${\mathbb{P}}_n{\widehat{ϵ}}_n=0$. By the definition of ${\widehat{\theta }}_n^{\ast }$, we have

$$\begin{array}{cc}\hfill \sqrt{n}({\widehat{\theta }}_n^{\ast }-{\widehat{\theta }}_n)[{\mathbb{P}}_n^{\ast }{X}_{{\widehat{k}}_n^{\ast }}^2-{\left({\mathbb{P}}_n^{\ast }{X}_{{\widehat{k}}_n^{\ast }}\right)}^2]& =\sqrt{n}[{\mathbb{P}}_n^{\ast }{X}_{{\widehat{k}}_n^{\ast }}Y-\left({\mathbb{P}}_n^{\ast }{X}_{{\widehat{k}}_n^{\ast }}\right)\left({\mathbb{P}}_n^{\ast }Y\right)-{\widehat{\theta }}_n({\mathbb{P}}_n^{\ast }{X}_{{\widehat{k}}_n^{\ast }}^2-{\left({\mathbb{P}}_n^{\ast }{X}_{{\widehat{k}}_n^{\ast }}\right)}^2)]\hfill \\ \hfill & =\sqrt{n}({\mathbb{P}}_n^{\ast }{X}_{{\widehat{k}}_n^{\ast }}{\widehat{ϵ}}_n-{\mathbb{P}}_n^{\ast }{X}_{{\widehat{k}}_n^{\ast }}{\mathbb{P}}_n^{\ast }{\widehat{ϵ}}_n)+\sqrt{n}{\widehat{\theta }}_n\left[{\left({\mathbb{P}}_n^{\ast }{X}_{{\widehat{k}}_n^{\ast }}\right)}^2-{\mathbb{P}}_n^{\ast }{X}_{{\widehat{k}}_n^{\ast }}^2+{\mathbb{P}}_n^{\ast }{X}_{{\widehat{k}}_n^{\ast }}\widehat{X}-\left({\mathbb{P}}_n^{\ast }{X}_{{\widehat{k}}_n^{\ast }}\right)\left({\mathbb{P}}_n^{\ast }\widehat{X}\right)\right]\hfill \\ \hfill & ={\mathbb{G}}_n^{\ast }{\widehat{ϵ}}_n(X_{{\widehat{k}}_n^{\ast }}-P_n{X}_{{\widehat{k}}_n^{\ast }})-{\mathbb{G}}_n^{\ast }{X}_{{\widehat{k}}_n^{\ast }}({\mathbb{P}}_n^{\ast }-{\mathbb{P}}_n){\widehat{ϵ}}_n-{\mathbb{G}}_n^{\ast }{\widehat{ϵ}}_n({\mathbb{P}}_n-P_n)X_{{\widehat{k}}_n^{\ast }}+\sqrt{n}{\widehat{\theta }}_n\left[{\left({\mathbb{P}}_n^{\ast }{X}_{{\widehat{k}}_n^{\ast }}\right)}^2-{\mathbb{P}}_n^{\ast }{X}_{{\widehat{k}}_n^{\ast }}^2+{\mathbb{P}}_n^{\ast }{X}_{{\widehat{k}}_n^{\ast }}\widehat{X}-\left({\mathbb{P}}_n^{\ast }{X}_{{\widehat{k}}_n^{\ast }}\right)\left({\mathbb{P}}_n^{\ast }\widehat{X}\right)\right].\hfill \end{array}$$ (11)

The last term in (11) is o_p^M (1) a.s. because the first and last terms within the square bracket cancel asymptotically, similarly for the second and third terms, due to ${\widehat{k}}_n^{\ast }\stackrel{P^M}{\to }{k}_0$ and k̂_n → k₀ a.s. We next show that the first term in (11) converges in distribution to Z_k0 (β₀) conditionally (on the data) in probability. By Lemma 1, it is easy to verify that ${\widehat{\theta }}_n\stackrel{P_n}{\to }{\theta }_0\triangleq C_{k_0}^T{\beta }_0∕V_{k_0}$ and ${\widehat{\alpha }}_n\stackrel{P_n}{\to }{\alpha }_0+E{\mathbf{X}}^T{\beta }_0-{\theta }_{0}E{X}_{k_0}$. Denote $\stackrel{‒}{ϵ}=ϵ+{(\mathbf{X}-E\mathbf{X})}^T{\beta }_0-{\theta }_0(X_{k_0}-E{X}_{k_0})$. Then the first term can be decomposed as

$${\mathbb{G}}_n^{\ast }{\widehat{ϵ}}_n[(X_{{\widehat{k}}_n^{\ast }}-P_n{X}_{{\widehat{k}}_n^{\ast }}-(X_{k_0}-P_n{X}_{k_0})]+{\mathbb{G}}_n^{\ast }\left[({\widehat{ϵ}}_n-\stackrel{‒}{ϵ})(X_{k_0}-P_n{X}_{k_0})\right]+{\mathbb{G}}_n^{\ast }\left[\stackrel{‒}{ϵ}(X_{k_0}-P_n{X}_{k_0})\right].$$ (12)

The first term in (12) is o_P^M (1) a.s. since ${\widehat{k}}_n^{\ast }\stackrel{P^M}{\to }{k}_0$. The second term in (12) can be written as

$$[({\alpha }_0+E{\mathbf{X}}^T{\beta }_0-{\theta }_{0}E{X}_{k_0})-{\widehat{\alpha }}_n]{\mathbb{G}}_n^{\ast }(X_{k_0}-P_n{X}_{k_0})+({\mathbb{P}}_n^{\ast }-{\mathbb{P}}_n)\left[(X_{k_0}-P_n{X}_{k_0}){\mathbf{X}}^T{\mathit{b}}_0\right]+({\theta }_0-{\widehat{\theta }}_n){\mathbb{G}}_n^{\ast }\left[X_{k_0}(X_{k_0}-P_n{X}_{k_0})\right]-{\widehat{\theta }}_n{\mathbb{G}}_n^{\ast }\left[(\widehat{X}-X_{k_0})(X_{k_0}-P_n{X}_{k_0})\right]$$

which is o_P^M (1) in probability by bootstrap consistency of the sample mean [see, e.g., Theorem 23.4 of van der Vaart (1998)], and the fact that X̂ = X_k0 for n sufficiently large a.s. Bootstrap consistency of the sample mean also gives that the third term in (12) converges in distribution to Z_k0(β₀) conditionally (on the data) in probability.

Similarly, the second and third terms in (11) and ${\mathbb{P}}_n^{\ast }{X}_{{\widehat{k}}_n^{\ast }}^2-{\left({\mathbb{P}}_n^{\ast }{X}_{{\widehat{k}}_n^{\ast }}\right)}^2-\mathrm{Var}\left(X_{k_0}\right)$ can be shown to be o_P^M (1) in probability. The result then follows from Slutsky's lemma.

Lemma 5

If all conditions in Theorem 1 hold and β₀ = 0, then ${\mathbb{V}}_n^{\ast }\left({\mathit{b}}_0\right)$ converges to the same limiting distribution as $\sqrt{n}({\widehat{\theta }}_n-{\theta }_n)$ conditionally (on the data) in probability.

Proof

Define ${\mathbb{Z}}_n$, ${\mathbb{M}}_n\left(\mathit{b}\right)$ and M′(b) to be p-vectors with kth components given by ${\mathbb{Z}}_{n,k}={\mathbb{G}}_n\left[ϵ(X_k-{\mathbb{P}}_n{X}_k)\right]$,

$$\frac{{[\widehat{\mathrm{Cov}}(X_k,{\mathbf{X}}^T\mathit{b})+{\mathbb{Z}}_{n,k}]}^2}{\widehat{\mathrm{Var}}\left(X_k\right)}\text{and}\frac{{\left[\mathrm{Cov}(X_k,{\mathbf{X}}^T\mathit{b})\right]}^2}{\mathrm{Var}\left(X_k\right)},$$

respectively. Let ${\mathbb{W}}_n\left(\mathit{b}\right)$ be a p × p matrix with the (j, k)-th component given by

$$\frac{\widehat{\mathrm{Cov}}(X_k,{\mathbf{X}}^T\mathit{b})+{\mathbb{Z}}_{n,k}}{\widehat{\mathrm{Var}}\left(X_k\right)}-\frac{\mathrm{Cov}(X_j,{\mathbf{X}}^T\mathit{b})}{\mathrm{Var}\left(X_j\right)}.$$

Also, let ${\mathbb{D}}_n\left(\mathit{b}\right)$ and D′(b) be p-vectors of zeros, apart from a 1 in the entry that maximizes ${\mathbb{M}}_n\left(\mathit{b}\right)$ and M′(b), respectively. Then

$${\mathbb{V}}_n\left(\mathit{b}\right)=D^{\prime }{\left(\mathit{b}\right)}^T{\mathbb{W}}_n\left(\mathit{b}\right){\mathbb{D}}_n\left(\mathit{b}\right).$$

Similarly, define $\mathbb{M}\left(\mathit{b}\right)$, $\mathbb{W}\left(\mathit{b}\right)$ and $\mathbb{D}\left(\mathit{b}\right)$ (without indexing by n) to be processes of the same form as ${\mathbb{M}}_n\left(\mathit{b}\right)$, ${\mathbb{W}}_n\left(\mathit{b}\right)$ and ${\mathbb{D}}_n\left(\mathit{b}\right)$, except with ${\mathbb{Z}}_{n,k}$ replaced by Z_k(0), and the sample variances/covariances replaced by their population versions.

Referring to the notation in (4), it is clear that when β₀ = 0,

$$\sqrt{n}({\widehat{\theta }}_n-{\theta }_n)={\mathbb{V}}_n\left({\mathit{b}}_0\right)=D^{\prime }{\left({\mathit{b}}_0\right)}^T{\mathbb{W}}_n\left({\mathit{b}}_0\right){\mathbb{D}}_n\left({\mathit{b}}_0\right)\stackrel{d}{\to }{D}^{\prime }{\left({\mathit{b}}_0\right)}^T\mathbb{W}\left({\mathit{b}}_0\right)\mathbb{D}\left({\mathit{b}}_0\right).$$

Moreover, the second equality in the above display also holds for the bootstrap version. Writing the bootstrapped version of ${\mathbb{Z}}_{n,k}$ in (9) as

$${\mathbb{Z}}_{n,k}^{\ast }={\mathbb{G}}_n^{\ast }\left[ϵ(X_k-P_n{X}_k)\right]+{\mathbb{G}}_n^{\ast }\left[({\widehat{ϵ}}_n-ϵ)(X_k-P_n{X}_k)\right]+\left[(P_n-{\mathbb{P}}_n^{\ast })X_k\right]{\mathbb{G}}_n^{\ast }{\widehat{ϵ}}_n],$$

and using arguments similar to those in the proof Lemma 4 for handling (12), we have ${\mathbb{Z}}_n^{\ast }\stackrel{d}{\to }{(Z_1\left(0\right),\dots ,Z_p\left(0\right))}^T$ conditionally (on the data) in probability. As a result, $({\widehat{D}}_n^{\prime }\left({\mathit{b}}_0\right),{\mathbb{W}}_n^{\ast }\left({\mathit{b}}_0\right),{\mathbb{M}}_n^{\ast }\left({\mathit{b}}_0\right))\stackrel{d}{\to }(D^{\prime }\left({\mathit{b}}_0\right),\mathbb{W}\left({\mathit{b}}_0\right),\mathbb{M}\left({\mathit{b}}_0\right))$ conditionally (on the data) in probability, where ${\widehat{D}}_n^{\prime }\left(\mathit{b}\right)$ is the sample version of D′(b), and ${\mathbb{W}}_n^{\ast }\left(\mathit{b}\right)$ and ${\mathbb{M}}_n^{\ast }\left(\mathit{b}\right)$ are the bootstrap versions of ${\mathbb{W}}_n\left(\mathit{b}\right)$ and ${\mathbb{M}}_n\left(\mathit{b}\right)$, respectively. Finally, using similar arguments to those at the end of the proof of Lemma 2, along with the continuous mapping theorem, we conclude that

$${\mathbb{V}}_n^{\ast }\left({\mathit{b}}_0\right)={\widehat{D}}_n^{\prime }{\left({\mathit{b}}_0\right)}^T{\mathbb{W}}_n^{\ast }\left({\mathit{b}}_0\right){\mathbb{D}}_n^{\ast }\left({\mathit{b}}_0\right)\stackrel{d}{\to }{D}^{\prime }{\left({\mathit{b}}_0\right)}^T\mathbb{W}\left({\mathit{b}}_0\right)\mathbb{D}\left({\mathit{b}}_0\right)$$

conditionally (on the data) in probability.