HIGH DIMENSIONAL VARIABLE SELECTION

Abstract

This paper explores the following question: what kind of statistical guarantees can be given when doing variable selection in high dimensional models? In particular, we look at the error rates and power of some multi-stage regression methods. In the first stage we fit a set of candidate models. In the second stage we select one model by cross-validation. In the third stage we use hypothesis testing to eliminate some variables. We refer to the first two stages as “screening” and the last stage as “cleaning.” We consider three screening methods: the lasso, marginal regression, and forward stepwise regression. Our method gives consistent variable selection under certain conditions.

1. Introduction

Several methods have been developed lately for high dimensional linear regression such as the lasso (Tibshirani 1996), Lars (Efron et al. 2004) and boosting (Bühlmann 2006). There are at least two different goals when using these methods. The first is to find models with good prediction error. The second is to estimate the true “sparsity pattern,” that is, the set of covariates with nonzero regression coefficients. These goals are quite different and this paper will deal with the second goal. (Some discussion of prediction is in the appendix.) Other papers on this topic include Meinshausen and Bühlmann (2006), Candes and Tao (2007), Wainwright (2006), Zhao and Yu (2006), Zou (2006), Fan and Lv (2008), Meinshausen and Yu (2008), Tropp (2004, 2006), Donoho (2006) and Zhang and Huang (2006). In particular, the current paper builds on ideas in Meinshausen and Yu (2008) and Meinshausen (2007).

Let (X₁, Y₁),…,(X_n, Y_n) be iid observations from the regression model

$$Y_i=X_i^T\beta +{\epsilon }_i$$ (1)

where ε ~ N(0, σ²), X_i = (X_i1,…, X_ip)^T ∈ ℝ^p and p = p_n > n. Let X be the n × p design matrix with j^th column X_•j = (X_1j,…, X_nj)^T and let Y = (Y₁,…, Y_n)^T. Let

$$D=\left\{j:{\beta }_j\ne 0\right\}$$

be the set of covariates with nonzero regression coefficients. Without loss of generality, assume that D = {1,…, s}for some s. A variable selection procedure D̂ _n maps the data into subsets of S = {1,…, p}.

The main goal of this paper is to derive a procedure D̂ _n such that

$$\underset{n\to \infty }{limsup}\mathbb{P}({\widehat{D}}_n\subset D)\ge 1-\alpha ,$$ (2)

that is, the asymptotic type I error is no more than α. Note that throughout the paper we use ⊂ to denote non-strict set-inclusion. Moreover, we want D̂ _n to have nontrivial power. Meinshausen and Bühlmann (2006) control a different error measure. Their method guarantees lim sup_n→∞ ℙ(D̂ _n ∩V ≠∅) ≤ α where V is the set of variables not connected to Y by any path in an undirected graph.

Our procedure involves three stages. In stage I we fit a suite of candidate models, each model depending on a tuning parameter λ,

$$\mathcal{S}=\{{\widehat{S}}_n(\lambda ):\lambda \in \mathrm{\Lambda }\}.$$

In stage II we select one of those models Ŝ _n using cross-validation to select λ̂ . In stage III we eliminate some variables by hypothesis testing. Schematically:

$$\underset{\text{screen}}{\underbrace{\begin{array}{llll}\text{data}\hfill & \stackrel{\text{stage}\text{I}}{\to }\hfill & \mathcal{S}\hfill & \stackrel{\text{stage}\text{II}}{\to }\hfill \end{array}}}{\widehat{S}}_n\underset{\text{clean}}{\underbrace{\begin{array}{ll}\stackrel{\text{stage}\text{III}}{\to }\hfill & {\widehat{D}}_n\hfill \end{array}}}$$

Genetic epidemiology provides a natural setting for applying screen and clean. Typically the number of subjects, n, is in the thousands, while p ranges from tens of thousands to hundereds of thousands of genetic features. The number of genes exhibiting a detectable association with a trait is extremely small. Indeed, for Type I diabetes only ten genes have exhibited a reproducible signal (Wellcome Trust 2007). Hence it is natural to assume that the true model is sparse. A common experimental design involves a 2-stage sampling of data, with stages 1 and 2 corresponding to the screening and cleaning processes, respectively.

In stage 1 of a genetic association study, n₁ subjects are sampled and one or more traits such as bone mineral density are recorded. Each subject is also measured at p locations on the chromosomes. These genetic covariates usually have two forms in the population due to variability at a single nucleotide and hence are called single nucleotide polymorphisms (SNPs). The distinct forms are called alleles. Each covariate takes on a value (0, 1 or 2) indicating the number of copies of the less common allele observed. For a well designed genetic study, individual SNPs are nearly uncorrelated unless they are physically located in very close proximity. This feature makes it much easier to draw causal inferences about the relationship between SNPs and quantitative traits. It is standard in the field to infer that an association discovered between a SNP and a quantitative trait implies a causal genetic variant is physically located near the one exhibiting association. In stage 2, n₂ subjects are sampled at a subset of the SNPs assessed in stage 1. SNPs measured in stage 2 are often those that achieved a test statistic that exceeded a predetermined threshold of significance in stage 1. In essence, the two stage design pairs naturally with a screen and clean procedure.

For the screen and clean procedure it is essential that Ŝ _n has two properties as n → ∞

$$\mathbb{P}(D\subset {\widehat{S}}_n)\to 1$$ (3)

and

$$\mid {\widehat{S}}_n\mid =o_P(n)$$ (4)

where |M| denotes the number of elements in a set M. Condition (3) ensures the validity of the test in stage III while condition (4) ensures that the power of the test is not too small. Without condition (3), the hypothesis test in stage III would be biased. We will see that the power goes to 1, so taking α= α_n → 0 implies consistency: ℙ(D̂ _n = D) → 1. For fixed α, the method also produces a confidence sandwich for D, namely,

$$\underset{n\to \infty }{liminf}\mathbb{P}({\widehat{D}}_n\subset D\subset {\widehat{S}}_n)\ge 1-\alpha .$$

To fit the suite of candidate models, we consider three methods. In Method 1,

$${\widehat{S}}_n(\lambda )=\{j:{\stackrel{\sim }{\beta }}_j(\lambda )\ne 0\}$$

where β̃_j(λ) is the lasso estimator, the value of β that minimizes

$$\sum _{i=1}^n{(Y_i-X_i^T\beta )}^2+\lambda \sum _{j=1}^p\mid {\beta }_j\mid .$$

In Method 2, take Ŝ _n(λ) to be the set of variables chosen by forward stepwise regression after λ steps. In Method 3, marginal regression, we take

$${\widehat{S}}_n=\{j:\mid {\widehat{\mu }}_j\mid >\lambda \}$$

where μ̂ _j is the marginal regression coefficient from regressing Y on X_j. (This is equivalent to ordering by the absolute t-statistics since we will assume that the covariates are standardized.) These three methods are very similar to basis pursuit, orthogonal matching pursuit and thresholding; see, for example, Tropp (2004, 2006) and Donoho (2006).

Notation

Let ψ = min_j∈D|β_j|. Define the loss of any estimator β̂ by

$$L(\widehat{\beta })=\frac{1}{n}{(\widehat{\beta }-\beta )}^T{X}^{T}X(\widehat{\beta }-\beta )={(\widehat{\beta }-\beta )}^T{\widehat{\mathrm{\sum }}}_n(\widehat{\beta }-\beta )$$ (5)

where Σ̂ _n = n⁻¹X^T X. For convenience, when β̂ ≡ β̂ (λ) depends on λ we write L(λ) instead of L(β̂ (λ)). If M ⊂ S, let X_M be the design matrix with columns (X_•j: j ∈ M) and let ${\widehat{\beta }}_M={(X_M^T{X}_M)}^{-1}{X}_M^{T}Y$ denote the least squares estimator, assuming it is well-defined. Note that our use of X_•j differs from standard ANOVA notation. Write X_λ instead of X_M when M = Ŝ _n(λ). When convenient, we extend β̂ _M to length p by setting β̂ _M (j) = 0 for j ∉ M. We use the norms:

$$\parallel v\parallel =\sqrt{\sum _j{v}_j^2},\parallel v\mid {\mid }_1=\sum _j\mid v_j\mid ,\parallel v\mid {\mid }_{\infty }=\underset{j}{max}\mid v_j\mid .$$

If C is any square matrix, let φ(C) and Φ(C) denote the smallest and largest eigenvalues of C. Also, if k is an integer define

$${\phi }_n(k)=\underset{M:\mid M\mid =k}{min}\phi \left(\frac{1}{n}{X}_M^T{X}_M\right),\text{and}{\mathrm{\Phi }}_n(k)=\underset{M:\mid M\mid =k}{max}\mathrm{\Phi }\left(\frac{1}{n}{X}_M^T{X}_M\right).$$

We will write z_u for the upper quantile of a standard Normal, so that ℙ(Z > z_u) = u where Z ~ N (0, 1).

Our method will involve splitting the data randomly into three groups , and . For ease of notation, assume the total sample size is 3n and that the sample size of each group is n.

Summary of Assumptions

We will use the following assumptions throughout except in Section 8.

• (A1) $Y_i=X_i^T\beta +{\epsilon }_i$ where ε_i ~ N (0, σ²), for i = 1, …, n.

  (A2) The dimension p_n of X satisfies p_n → ∞ and p_n ≤ c₁e^nc₂ for some c₁ > 0 and 0 ≤ c₂ < 1.

  (A3) s ≡ |{j: β_j ≠ 0}| = O(1) and ψ = min{|β_j|: β_j ≠ 0} > 0.

  (A4) There exist positive constants C₀, C₁ and κ such that ℙ (lim sup_{n→ ∞} Φ_n(n) ≤ C₀) = 1 and ℙ(lim inf_{n→ ∞} φ_n(C₁ log n) ≥ κ) = 1. Also, ℙ(φ_n(n) > 0) = 1 for all n.

• (A5) The covariates are standardized: (X_ij) = 0 and $\mathbb{E}(X_{ij}^2)=1$. Also, there exists 0 < B < ∞ such that ℙ(|X_jk| ≤ B) = 1.

For simplicity, we include no intercepts in the regressions. The assumptions can be weakened at the expense of more complicated proofs. In particular, we can let s increase with n and ψ decrease with n. Similarly, the Normality and constant variance assumptions can be relaxed.

2. Error Control

Define the type I error rate q(D̂ _n) = ℙ(D̂ _n ∩ D^c ≠ ∅) and the asymptotic error rate lim sup_{n→ ∞} q(D̂ _n). We define the power π(D̂ _n) = ℙ (D ⊂ D̂ _n) and the average power

$${\pi }_{\text{av}}=\frac{1}{s}\sum _{j\in D}\mathbb{P}(j\in {\widehat{D}}_n).$$

It is well known that controlling the error rate is difficult for at least three reasons: correlation of covariates, high dimensionality of the covariate and unfaithfulness (cancellations of correlations due to confounding). Let us briefly review these issues.

It is easy to construct examples where, q(D̂ _n) ≤ α implies that π(D̂ _n) ≈ α. Consider two models for random variables Z = (Y, X₁, X₂):

Model 1	Model 2
X1 ~ N (0, 1)	X2 ~ N (0, 1)
Y = ψX1 + N (0, 1)	Y= ψX2 + N (0, 1)
X2= ρX1 + N (0, τ2)	X1= ρX2 + N (0, τ2).

Under models 1 and 2, the marginal distribution of Z is P₁ = N (0, Σ₁) and P₂ = N (0, Σ₂) where

$${\mathrm{\sum }}_1=\left(\begin{array}{ccc}{\psi }^2+1& \psi & \rho \psi \\ \psi & 1& \rho \\ \rho \psi & \rho & {\rho }^2+{\tau }^2\end{array}\right),{\mathrm{\sum }}_2=\left(\begin{array}{ccc}{\psi }^2+1& \rho \psi & \psi \\ \rho \psi & {\rho }^2+{\tau }^2& \rho \\ \psi & \rho & 1\end{array}\right).$$

Given any ε > 0 we can choose ρ sufficiently close to 1 and τ sufficiently close to 0 such that Σ₁ and Σ₂ are as close as we like and hence $d(P_1^n,P_2^n)<\epsilon$ where d is total variation distance. It follows that

$${\mathbb{P}}_2(2\notin \widehat{D})\ge {\mathbb{P}}_1(2\notin \widehat{D})-\epsilon \ge 1-\alpha -\epsilon .$$

Thus, if q ≤ α then the power is less than α + ε.

Dimensionality is less of an issue thanks to recent methods. Most methods, including those in this paper, allow p_n to grow exponentially. But all the methods require some restrictions on the number s of nonzero β_j’s. In other words, some sparsity assumption is required. In this paper we take s fixed and allow p_n to grow.

False negatives can occur during screening due to cancellations of correlations. For example, the correlation between Y and X₁ can be 0 even when β₁ is huge. This problem is called unfaithfulness in the causality literature; see Spirtes, Glymour and Scheines (2001) and Robins, Spirtes, Scheines and Wasserman (2003). False negatives during screening can lead to false positives during the second stage.

Let μ̂ _j denote the regression coefficient from regressing Y on X_j. Fix j ≤ s and note that

$${\mu }_j\equiv \mathbb{E}({\widehat{\mu }}_j)={\beta }_j+\sum _{\stackrel{k\ne j}{1\le k\le s}}{\beta }_k{\rho }_{kj}$$

where ρ_kj = corr(X_k, X_j). If

$$\sum _{\stackrel{k\ne j}{1\le k\le s}}{\beta }_k{\rho }_{kj}\approx -{\beta }_j$$

then μ_j ≈ 0 no matter how large β_j is. This problem can occur even when n is large and p is small.

For example, suppose that β = (10, −10, 0, 0) and that ρ(X_i, X_j) = 0 except that ρ(X₁, X₂) =ρ(X₁, X₃) = ρ(X₂, X₄) = 1 − ε where ε > 0 is small. Then

$$\beta =(10,-10,0,0)\text{but}\mu \approx (0,0,10,-10).$$

Marginal regression is extremely susceptible to unfaithfulness. The lasso and forward stepwise, less so. However, unobserved covariates can induce unfaithfulness in all the methods.

3. Loss and Cross-validation

Let X_λ = (X_•j: j ∈ Ŝ _n(λ)) denote the design matrix corresponding to the covariates in Ŝ _n(λ) and let β̂ (λ) be the least squares estimator for the regression restricted to Ŝ _n(λ), assuming the estimator is well defined. Hence, $\widehat{\beta }(\lambda )={(X_{\lambda }^T{X}_{\lambda })}^{-1}{X}_{\lambda }^{T}Y$. More generally, β̂ _M is the least squares estimator for any subset of variables M. When convenient, we extend β̂ (λ) to length p by setting β̂ _j(λ) = 0 for j ∉ Ŝ _n(λ).

3.1. Loss

Now we record some properties of the loss function. The first part of the following lemma is essentially Lemma 3 of Meinshausen and Yu (2008).

Lemma 3.1

Let ${\mathcal{M}}_m^{+}=\{M\subset S:\mid M\mid \le m,D\subset M\}$. Then,

$$\mathbb{P}\left(\underset{M\in {\mathcal{M}}_m^{+}}{sup}L({\widehat{\beta }}_M)\le \frac{4\mathit{m}log\mathit{p}}{n{\phi }_n(m)}\right)\to 1.$$ (6)

Let ${\mathcal{M}}_m^{-}=\{M\subset S:\mid M\mid \le m,D\not\subset M\}$. Then,

$$\mathbb{P}\left(\underset{M\in {\mathcal{M}}_m^{-}}{inf}L({\widehat{\beta }}_M)\ge {\psi }^2{\phi }_n(m+s)\right)\to 1.$$ (7)

3.2. Cross-validation

Recall that the data have been split into groups , , and each of size n. Construct β̂ (λ) from and let

$$\widehat{L}(\lambda )=\frac{1}{n}\sum _{X_i\in {\mathcal{D}}_2}{(Y_i-X_i^T\widehat{\beta }(\lambda ))}^2.$$ (8)

We would like L̂ (λ) to order the models the same way as the true loss L(λ) (defined after equation (5)). This requires that, asymptotically, L̂ (λ) − L(λ) ≈ δ_n where δ_n does not involve λ. The following bounds will be useful. Note that L(λ) and L̂ (λ) are both step functions that only change value when a variable enters or leaves the model.

Theorem 3.2

Suppose that max_λ∈Λn |Ŝ _n(λ)| ≤ k_n. Then there exists a sequence of random variables δ_n = O_P (1) that do not depend on λ or X, such that, with probability tending to 1,

$$\underset{\lambda \in {\mathrm{\Lambda }}_n}{sup}\mid L(\lambda )-\widehat{L}(\lambda )-{\delta }_n\mid =O_P\left(\frac{k_n}{n^{1-c_2}}\right)+O_P\left(\frac{k_n}{\sqrt{n}}\right).$$ (9)

4. Multi-Stage Methods

The multi-stage methods use the following steps. As mentioned earlier, we randomly split the data into three parts , and which we take to be of equal size.

1. Stage I. Use to find Ŝ _n (λ) for each λ.

2. Stage II. Use to find λ̂ by cross-validation and let Ŝ _n = Ŝ _n (λ̂ )

3. Stage III. Use to find the least squares estimate β̂ for the model Ŝ _n. Let

$${\widehat{D}}_n=\{j\in {\widehat{S}}_n:\mid T_j\mid >c_n\}$$

where T_j is the usual t-statistic, c_n = z_α/2m and m = |Ŝ _n|

4.1. The Lasso

The lasso estimator (Tibshirani 1996) β̃(λ) minimizes

$$M_{\lambda }(\lambda )=\sum _{i=1}^n{(Y_i-X_i^T\beta )}^2+\lambda \sum _{j=1}^p\mid {\beta }_j\mid$$

and let Ŝ _n(λ) = {j: β̃_j (λ) ≠ 0}. Recall that β̂ (λ) is the least squares estimator using the covariates in Ŝ _n (λ).

Let k_n = A log n where A > 0 is a positive constant.

Theorem 4.1

Assume that (A1)–(A5) hold. Let Λ_n = {λ: |Ŝ _n(λ)| ≤ k_n}. Then:

1. The true loss overfits: ℙ(D ⊂ Ŝ _n(λ_*)) → 1 where λ_* = argmin_λ∈Λn L(λ).

2. Cross-validation also overfits: ℙ(D ⊂ Ŝ _n(λ̂ )) →1 where λ̂ = argmin_λ∈ΛnL̂ (λ).

3. Type I error is controlled: lim sup_{n→ ∞} ℙ(D^c ∩ D̂ _n ≠ ∅) ≤ α

If we let α = α_n → 0 then D̂ _n is consistent for variable selection.

Theorem 4.2

Assume that (A1)–(A5) hold. Let α_n → 0 and $\sqrt{n}{\alpha }_n\to \infty$. Then, the multi-stage lasso is consistent,

$$\mathbb{P}({\widehat{D}}_n=D)\to 1.$$ (10)

The next result follows directly. The proof is thus omitted.

Theorem 4.3

Assume that (A1)–(A5) hold. Let α be fixed. Then (D̂ _n; Ŝ _n) forms a confidence sandwich:

$$\underset{n\to \infty }{liminf}\mathbb{P}({\widehat{D}}_n\subset D\subset {\widehat{S}}_n)\ge 1-\alpha .$$ (11)

Remark 4.4

This confidence sandwich is expected to be conservative in the sense that the coverage can be much larger than 1 − α.

4.2. Stepwise Regression

The version of stepwise regression we consider is as follows. Let k_n = A log n for some A > 0.

1. Initialize: Res = Y, λ = 0, Ŷ = 0, and Ŝ _n (λ) = ∅.

2. Let λ ← λ+ 1. Compute μ̂ _j = n⁻¹〈X_j, Res〉 for j = 1, …, p.

3. Let J= argmax_j |μ̂ _j|. Set Ŝ _n(λ) = {Ŝ _n(λ −1), J}. Set Ŷ = X_λβ̂ (λ) where ${\widehat{\beta }}_{\lambda }={(X_{\lambda }^T{X}_{\lambda })}^{-1}{X}_{\lambda }^{T}Y$ and let Res = Y − Ŷ .

4. If λ = k_n stop. Otherwise, go to step 2.

For technical reasons, we assume that the final estimator x^Tβ̂ is truncated to be no larger than B. Note that λ is discrete and Λ_n = {0, 1, …, k_n}.

Theorem 4.5

With Ŝ _n(λ) defined as above, the statements of Theorems 4.1, 4.2 and 4.3 hold.

4.3. Marginal Regression

This is probably the oldest, simplest and most common method. It is quite popular in gene expression analysis. It is used to be regarded with some derision but has enjoyed a revival. A version appears in a recent paper by Fan and Lv (2008). Let Ŝ _n(λ) = {j: |μ̂ _j| ≥ λ} where μ̂ _j =n⁻¹ 〈Y, X_•j〉.

Let μ_j = (μ̂ _j) and let μ_(j) denote the value of μ ordered by their absolute values:

$$\mid {\mu }_{(1)}\mid \ge \mid {\mu }_{(2)}\mid \ge \cdots$$

Theorem 4.6

Let k_n → ∞ with $k_n=o(\sqrt{n})$. Let Λ_n = {λ: |Ŝ _n(λ)| ≤ k_n}. Assume that

$$\underset{j\in D}{min}\mid {\mu }_j\mid >\mid {\mu }_{(k_n)}\mid .$$ (12)

Then, the statements of Theorems 4.1, 4.2 and 4.3 hold.

The assumption (12) limits the degree of unfaithfulness (small partial correlations induced by cancellation of parameters). Large values of k_n weaken assumption (12) thus making the method more robust to unfaithfulness, but at the expense of lower power. Fan and Lv (2008) make similar assumptions. They assume that there is a C > 0 such that |μ_j| ≥ C|β_j| for all j which rules out unfaithfulness. However, they do not explicitly related the values of μ_j for j ∈ D to the values outside D as we have done. On the other hand, they assume that Z = Σ^−1/2 X has a spherically symmetric distribution. Under this assumption and their faithfulness assumption, they deduce that the μ_j’s outside D cannot strongly dominate the μ_j’s within D. We prefer to simply make this an explicit assumption without placing distributional assumptions on X. At any rate, any method that uses marginal regressions as a starting point must make some sort of faithfulness assumptions to succeed.

4.4. Modifications

Let us now discuss a few modifications of the basic method. First, consider splitting the data only into two groups and . Then do these steps:

1. Stage I. Find Ŝ _n(λ) for λ ∈ Λ_n where |Ŝ _n(λ)| ≤ k_n for each λ ∈ Λ_n using .

2. Stage II. Find λ̂ by cross-validation and let Ŝ _n = Ŝ _n(λ̂ ) using .

3. Stage III. Find the least squares estimate β̂ _{Ŝ n} using . Let D̂ _n = {j ∈ Ŝ _n: |T_j| > c_n} where T_j is the usual t-statistic.

Theorem 4.7

Choosing

$$c_n=\frac{loglog\mathit{n}\sqrt{2k_{n}log(2p_n)}}{\alpha }$$ (13)

controls asymptotic type I error.

The critical value in (13) is hopelessly large and it does not appear it can be substantially reduced. We present this mainly to show the value of the extra data-splitting step. It is tempting to use the same critical value as in the tri-split case, namely, c_n = z_α/2m where m = |Ŝ _n| but we suspect this will not work in general. However, it may work under extra conditions.

5. Application

As an example we illustrate an analysis based on part of the Osteoporotic Fractures in Men Study (MrOS, Orwoll et al. 2005). A sample of 860 men were measured at a large number of genes and outcome measures. We consider only 296 SNPs which span 30 candidate genes for bone mineral density. An aim of the study was to identify genes associated with bone mineral density that could help in understanding the genetic basis of osteoporosis in men. Initial analyses of this subset of the data revealed no SNPs with a clear pattern of association with the phenotype; however, three SNPs, numbered (67, 277, 289) exhibited some association in the screening of the data. To further explore the effacacy of the lasso screen and clean procedure we modified the phenotype to enhance this weak signal and then reanalyzed the data to see if we could detect this planted signal.

We were interested in testing for main effects and pairwise interactions in these data; however, including all interactions results in a model with 43,660 additional terms, which is not practical for this sample size. As a compromise we selected 2 SNPs per gene to model potential interaction effects. This resulted in a model with a total of 2066 potential coefficients, including 296 main effects and 1770 interaction terms. With this model our initial screen detected 10 terms, including the three enhanced signals, 2 other main effects and 5 interactions. After cleaning, the final model detected the 3 enhanced signals, and no other terms.

6. Simulations

To further explore the screen and clean procedures, we conducted simulation experiments with four models. For each model $Y_i=X_i^T\beta +{\epsilon }_i$ where the measurement errors, ε_i and ${\epsilon }_{ij}^{\ast }$, are iid Normal(0, 1) and the covariates X_ij’s are Normal(0, 1) (except for model D). Models differ in how Y_i is linked to X_i and the dependence structure of the X_i’s. Models A, B and C explore scenarios with moderate and large p, while Model D focuses on confounding and unfaithfullness.

1. Null model: β = (0,…,0) and the X_ij’s are iid.

2. Triangle model: β_j = δ(10 − j), j = 1,…, 10, β_j = 0, j > 10 and X_ij’s are iid.

3. Correlated Triangle model: as B, but with $X_{ij(+1)}=\rho X_{ij}+{(1-{\rho }^2)}^{1/2}{\epsilon }_{ij}^{\ast }$ for j > 1, and ρ = 0.5.

4. Unfaithful model: Y_i = β₁X_i1 + β₂X_i2 + ε_i, for β₁ = − β₂ = 10, where the X_ij’s are iid for j = {1, 5, 6, 7, 8, 9, 10}, but $X_{i2}=\rho X_{i1}+\tau {\epsilon }_{i2}^{\ast },X_{i3}=\rho X_{i1}+\tau {\epsilon }_{i10}^{\ast }$, and $X_{i4}=\rho X_{i2}+\tau {\epsilon }_{i11}^{\ast }$, for τ = 0.01 and ρ = 0.95.

We used a maximum model size of k_n = n^1/2 which technically goes beyond the theory but works well in practice. Prior to analysis the covariates are scaled so that each has mean 0 and variance 1. The tests were initially performed using a third of the data for each of the three stages of the procedure (Table 1, top half, 3 splits). For models A, B and C each approach has Type I error less than ρ, except the stepwise procedure which has trouble with model C when n = p = 100. We also calculated the false positive rate and found it to be very low (about 10⁻⁴ when p = 100 and 10⁻⁵ when p = 1000) indicating that even when a Type I error occurs, only a very small number of terms are included erroneously. The lasso screening procedure exhibited a slight power advantage over the stepwise procedure. Both methods dominated the marginal approach. The Markov dependence structure in model C clearly challenged the marginal approach. For Model D none of the approaches controlled the Type I error.

Table 1.

Size and Power of Screen and Clean Procedures using Lasso, Stepwise and Marginal regression for the screening step. For all procedures α = 0.05. For p = 100, δ = 0.5 and for p = 1000, δ = 1.5. Reported power is π_av. The top 8 rows of simulations were conducted using three stages as described in section 4, with a third of the data used for each stage. The bottom 8 rows of simulations were conducted splitting the data in half, using the first portion with leave-one-out cross validation for stages 1 and 2 and the second portion for cleaning.

Splits	n	p	model	Lasso	Size Step	Marg	Lasso	Power Step	Marg
2	100	100	A	0.005	0.001	0.004	0.00	0.00	0.00
2	100	100	B	0.01	0.02	0.03	0.62	0.62	0.31
2	100	100	C	0.001	0.01	0.01	0.77	0.57	0.21
2	100	10	D	0.291	0.283	0.143	0.08	0.08	0.04
2	100	1000	A	0.001	0.002	0.010	0.00	0.00	0.00
2	100	1000	B	0.002	0.020	0.010	0.17	0.09	0.11
2	100	1000	C	0.02	0.14	0.01	0.27	0.15	0.11
2	1000	10	D	0.291	0.283	0.143	0.08	0.08	0.04
3	100	100	A	0.040	0.050	0.030	0.00	0.00	0.00
3	100	100	B	0.02	0.01	0.02	0.91	0.90	0.56
3	100	100	C	0.03	0.04	0.03	0.91	0.88	0.41
3	100	10	D	0.382	0.343	0.183	0.16	0.18	0.09
3	100	1000	A	0.035	0.045	0.040	0.00	0.00	0.00
3	100	1000	B	0.045	0.020	0.035	0.57	0.66	0.29
3	100	1000	C	0.06	0.070	0.020	0.74	0.65	0.19
3	1000	10	D	0.481	0.486	0.187	0.17	0.17	0.13

To determine the sensitivity of the approach to using distinct data for each stage of the analysis, simulations were conducted screening on the first half of the data and cleaning on the second half (2 splits). The tuning parameter was selected using leave-one-out cross validation (Table 1, bottom half). As expected this approach lead to a dramatic increase in the power of all the procedures. More surprising is the fact that the Type I error was near α or below for models A, B and C. Clearly this approach has advantages over data splitting and merits further investigation.

A natural competitor to screen and clean procedure is a two-stage adaptive lasso (Zou, 2006). In our implementation we split the data and used one half for each stage of the analysis. At stage one, leave-one-out cross validation lasso screens the data. In stage two, the adaptive lasso, with weights w_j = |β̂ _j|⁻¹, cleans the data. The tuning parameter for the lasso was again chosen using leave-one-out cross validation. Table 2 provides the size, power and false positive rate (FPR) for this procedure. Naturally, the adaptive lasso does not control the size of the test, but the FPR is small. The power of the test is greater than we found for our lasso screen and clean procedure, but this extra power comes at the cost of a much higher Type I error rate.

Table 2.

Size, Power and False Positive Rate (FPR) of Two-stage Adaptive Lasso Procedure

n	p	model	Size	Power	FPR
100	100	A	0.93	0	0.032
100	100	B	0.84	0.97	0.034
100	100	C	0.81	0.96	0.031
100	10	D	0.67	0.21	0.114
100	1000	A	0.96	0	0.004
100	1000	B	0.89	0.65	0.004
100	1000	C	0.76	0.77	0.002
1000	10	D	0.73	0.24	0.013

7. Proofs

Recall that if A is a square matrix then φ(A) and Φ(A) denote the smallest and largest eigenvalues of A. Throughout the proofs we make use of the following fact. If v is a vector and A is a square matrix then

$$\phi (A)\parallel v\mid {\mid }^2\le v^{T}Av\le \mathrm{\Phi }(A)\parallel v\mid {\mid }^2.$$ (14)

We use the following standard tail bound: if Z ~ N(0, 1) then ℙ(|Z| > t) ≤ t⁻¹e^−t2/2. We will also use the following results about the lasso from Meinshausen and Yu (2008). Their results are stated and proved for fixed X but, under the conditions (A1)–(A5), it is easy to see that their conditions hold with probability tending to one and so their results hold for random X as well.

Theorem 7.1 (Meinshausen and Yu, 2008)

Let β̃(λ) be the lasso estimator.

1. The squared error satisfies:

   $$\mathbb{P}\left(\parallel \stackrel{\sim }{\beta }(\lambda )-\beta \mid {\mid }_2^2\le \frac{2{\lambda }^{2}s}{n^2{\kappa }^2}+\frac{\mathit{cm}log{p}_n}{n{\phi }_n^2(m)}\right)\to 1$$ (15)

   where m = |Ŝ _n(λ)| and c > 0 is a constant.

2. The size of Ŝ _n(λ) satisfies

   $$\mathbb{P}\left(\mid {\widehat{S}}_n(\lambda )\mid \le \frac{{\tau }^{2}C{n}^2}{{\lambda }^2}\right)\to 1$$ (16)

   where ${\tau }^2=\mathbb{E}(Y_i^2)$.

Proof of Lemma 3.1

Let D ⊂ M and $\phi =\phi (n^{-1}{X}_M^T{X}_M)$. Then

$$L({\widehat{\beta }}_M)=\frac{1}{n}{\epsilon }^T{X}_M{(X_M^T{X}_M)}^{-1}{X}_M^T\epsilon \le \frac{1}{n^2\phi }\parallel X_M^T\epsilon \mid {\mid }^2=\frac{1}{n\phi }\sum _{j\in M}{Z}_j^2$$

where $Z_j=n^{-1/2}{X}_{•j}^T\epsilon$. Conditional on X, $Z_i\sim N(0,a_j^2)$ where $a_j^2=n^{-1}{\sum }_{i=1}^n{X}_{ij}^2$. Let $A_n^2={max}_{1\le j\le p_n}{a}_j^2$. By Hoeffding’s inequality, (A2) and (A5), ℙ(E_n) → 1 where $E_n=\{A_n\le \sqrt{2}\}$. So

$$\begin{array}{l}\mathbb{P}(\underset{1\le j\le p_n}{max}\mid Z_j\mid >\sqrt{4log{p}_n})=\mathbb{P}(\underset{1\le j\le p_n}{max}\mid Z_j\mid >\sqrt{4log{p}_n},E_n)+\mathbb{P}(\underset{1\le j\le p_n}{max}\mid Z_j\mid >\sqrt{4log{p}_n},E_n^c)\\ \le \mathbb{P}(\underset{1\le j\le p_n}{max}\mid Z_j\mid >\sqrt{4log{p}_n},E_n)+\mathbb{P}(E_n^c)\\ \le \mathbb{P}\left(A_n\underset{1\le j\le p_n}{max}\frac{\mid Z_j\mid }{a_j}>\sqrt{4log{p}_n},E_n\right)+o(1)\\ \le \mathbb{P}\left(\underset{1\le j\le p_n}{max}\frac{\mid Z_j\mid }{a_j}>\sqrt{2log{p}_n}\right)+o(1)\\ =\mathbb{E}\left(\mathbb{P}\left(\underset{1\le j\le p_n}{max}\frac{\mid Z_j\mid }{a_j}>\sqrt{2log{p}_n}\right)\mid X\right)+o(1)\\ \le O\left(\frac{1}{\sqrt{2log{p}_n}}\right)+o(1)=o(1).\end{array}$$

But ${\sum }_{j\in M}{Z}_j^2\le m{max}_{1\le j\le p_n}{Z}_j^2$ and (6) follows.

Now we lower bound L(β̂ _M). Let M be such that D ⊄ M. Let A = {j: β̂ (j) ≠ 0} ∪ D. Then |A| ≤ m + s. Therefore, with probability tending to 1,

$$\begin{array}{l}L({\widehat{\beta }}_M)=\frac{1}{n}{({\widehat{\beta }}_M-\beta )}^T{X}^{T}X({\widehat{\beta }}_M-\beta )=\frac{1}{n}{({\widehat{\beta }}_M-\beta )}^T{X}_A^T{X}_A({\widehat{\beta }}_M-\beta )\\ \ge {\phi }_n(m+s)\parallel {\widehat{\beta }}_M-\beta \mid {\mid }^2={\phi }_n(m+s)\sum _{j\in A}{({\widehat{\beta }}_M(j)-\beta (j))}^2\\ \ge {\phi }_n(m+s)\sum _{j\in D\cap M^c}{(0-\beta (j))}^2\ge {\phi }_n(m+s){\psi }^2.\end{array}$$

Proof of Theorem 3.2

Let Ỹ denote the responses, and X̃ the design matrix, for the second half of the data. Then Ỹ = X̃β + ε̃. Now

$$L(\lambda )=\frac{1}{n}{(\widehat{\beta }(\lambda )-\beta )}^T{X}^{T}X(\widehat{\beta }(\lambda )-\beta )={(\widehat{\beta }(\lambda )-\beta )}^T{\widehat{\mathrm{\sum }}}_n(\widehat{\beta }(\lambda )-\beta )$$

and

$$\widehat{L}(\lambda )=n^{-1}\parallel \stackrel{\sim }{Y}-\stackrel{\sim }{X}\widehat{\beta }(\lambda )\mid {\mid }^2={(\widehat{\beta }(\lambda )-\beta )}^T{\mathrm{\sum }^{\sim }}_n(\widehat{\beta }(\lambda )-\beta )+{\delta }_n+\frac{2}{n}\langle \stackrel{\sim }{\epsilon },\stackrel{\sim }{X}(\widehat{\beta }(\lambda )-\beta )\rangle$$

where δ_n = ||ε̃||²/n, and ${\widehat{\mathrm{\sum }}}_n=n_1^{-1}{X}^{T}X$ and Σ̃_n = n⁻¹ X̃^T X̃. By Hoeffding’s inequality

$$\mathbb{P}(\mid {\widehat{\mathrm{\sum }}}_n(j,k)-{\mathrm{\sum }^{\sim }}_n(j,k)\mid >\epsilon )\le e^{-nc{\epsilon }^2}$$

for some c > 0 and so

$$\mathbb{P}(\underset{jk}{max}\mid {\widehat{\mathrm{\sum }}}_n(j,k)-{\mathrm{\sum }^{\sim }}_n(j,k)\mid >\epsilon )\le p_n^2{e}^{-nc{\epsilon }^2}.$$

Choose ε_n = 4/(cn^1−c₂). It follows that

$$\mathbb{P}\left(\underset{jk}{max}\mid {\widehat{\mathrm{\sum }}}_n(j,k)-{\mathrm{\sum }^{\sim }}_n(j,k)\mid >\frac{4}{c{n}^{1-c_2}}\right)\le e^{-2n^{c_2}}\to 0.$$

Note that

$$\mid \{j:{\widehat{\beta }}_j(\lambda )\ne 0\}\cup \{j:{\beta }_j\ne 0\}\mid \le k_n+s.$$

Hence, with probability tending to 1,

$$\mid L(\lambda )-\widehat{L}(\lambda )-{\delta }_n\mid \le \frac{4}{c{n}^{1-c_2}}\parallel \widehat{\beta }(\lambda )-\beta \mid {\mid }_1^2+2{\xi }_n(\lambda )$$

for all λ ∈ Λ_n, where

$${\xi }_n(\lambda )=\frac{1}{n}\sum _{i\in I_2}{\stackrel{\sim }{\epsilon }}_i{\mu }_i(\lambda )$$

and ${\mu }_i(\lambda )={\stackrel{\sim }{X}}_i^T(\widehat{\beta }(\lambda )-\beta )$. Now $\parallel \widehat{\beta }(\lambda )-\beta )\mid {\mid }_1^2=O_P({(k_n+s)}^2)$ since ||β̂ (λ)||² = O_P (k_n/φ(k_n)). Thus, ||β̂ (λ) − β||₁ ≤ C(k_n + s) with probability tending to 1, for some C > 0. Also, |μ_i(λ)| ≤ B||β̂ (λ) − β||₁ ≤ BC(k_n + s) with probability tending to 1. Let W ~ N (0, 1). Conditional on ,

$$\mid {\xi }_n(\lambda )\mid \stackrel{d}{=}\frac{\sigma }{\sqrt{n}}\sqrt{\sum _{i=1}^n{\mu }_i^2(\lambda )}\mid W\mid \le \frac{\sigma }{\sqrt{n}}BC(k_n+s)\mid W\mid$$

so ${sup}_{\lambda \in {\mathrm{\Lambda }}_n}\mid {\xi }_n(\lambda )\mid =O_P(k_n/\sqrt{n})$.

Proof of Theorem 4.1

(1) Let ${\lambda }_n=\tau n\sqrt{C/k_n}$, M = Ŝ _n(λ_n) and m = |M |. Then, ℙ(m ≤ k_n) → 1 due to (16). Hence, ℙ(λ_n ∈ Λ_n) → 1. From (15),

$$\parallel \stackrel{\sim }{\beta }({\lambda }_n)-\beta \mid {\mid }_2^2\le O\left(\frac{1}{k_n}\right)+O_P\left(\frac{k_{n}log{p}_n}{n}\right)=o_P(1).$$

Hence, $\parallel \stackrel{\sim }{\beta }({\lambda }_n)-\beta \mid {\mid }_{\infty }^2=o_P(1)$. So, for each j ∈ D,

$$\mid {\stackrel{\sim }{\beta }}_j({\lambda }_n)\mid \ge \mid {\beta }_j\mid -\mid {\stackrel{\sim }{\beta }}_j({\lambda }_n)-{\beta }_j\mid \ge \psi +o_P(1)$$

and hence ℙ(min_j∈D|β̃_j(λ_n)| > 0) → 1. Therefore, Γ_n = {λ ∈ Λ_n: D ⊂ Ŝ _n(λ)} is nonempty. By Lemma 3.1,

$$L({\lambda }_n)\le \mathit{cm}log{p}_n/(n\phi (m))=O_P(k_{n}log{p}_n/n).$$ (17)

On the other hand, from Lemma 3.1,

$$\mathbb{P}\left(\underset{\lambda \in {\mathrm{\Lambda }}_n\cap {\mathrm{\Gamma }}_n^c}{inf}L({\widehat{\beta }}_{\lambda })>{\psi }^2\phi (k_n)\right)\to 1.$$ (18)

Now, nφ_n(k_n)/(k_n log p_n) → ∞ and so, (17) and (18) imply that

$$\mathbb{P}\left(\underset{\lambda \in {\mathrm{\Lambda }}_n\cap {\mathrm{\Gamma }}_n^c}{inf}L({\widehat{\beta }}_{\lambda })>L({\lambda }_n)\right)\to 1.$$

Thus, if λ_* denotes the minimizer of L(λ) over Λ_n, we conclude that ℙ(λ_* ∈ Γ_n) → 1 and hence, ℙ(D ⊂Ŝ _n(λ_*)) → 1.

(2) This follows from part (1) and Theorem 3.2.

(3) Let A = Ŝ _n ∩ D^c. We want to show that

$$\mathbb{P}(\underset{j\in A}{max}\mid T_j\mid >c_n)\le \alpha +o(1).$$

Now,

$$\begin{array}{l}\mathbb{P}(\underset{j\in A}{max}\mid T_j\mid >c_n)=\mathbb{P}(\underset{j\in A}{max}\mid T_j\mid >c_n,D\subset {\widehat{S}}_n)+\mathbb{P}(\underset{j\in A}{max}\mid T_j\mid >c_n,D\not\subset {\widehat{S}}_n)\\ \le \mathbb{P}(\underset{j\in A}{max}\mid T_j\mid >c_n,D\subset {\widehat{S}}_n)+\mathbb{P}(D\not\subset {\widehat{S}}_n)\\ \le \mathbb{P}(\underset{j\in A}{max}\mid T_j\mid >c_n,D\subset {\widehat{S}}_n)+o(1).\end{array}$$

Conditional on ( , ), β̂ _A is Normally distributed with mean 0 and variance matrix ${\sigma }^2{(X_A^T{X}_A)}^{-1}$ when D ⊂ Ŝ _n. Recall that

$$T_j(M)=\frac{e_j^T{(X_M^T{X}_M)}^{-1}{X}_M^{T}Y}{\widehat{\sigma }\sqrt{e_j^T{(X_M^T{X}_M)}^{-1}{e}_j}}=\frac{{\widehat{\beta }}_{M,j}}{s_j}$$

where M = Ŝ _n, $s_j^2={\widehat{\sigma }}^2{e}_j^T{(X_M^T{X}_M)}^{-1}{e}_j$ and e_j= (0, …, 0, 1, 0, …, 0)^T where the 1 is in the j^th coordinate. When D ⊂ Ŝ _n, each T_j, for j ∈ A, has a t-distribution with n − m degrees of freedom where m = |Ŝ _n|. Also, c_n/t_α/2m → 1 where t_u denotes the upper tail critical value for the t-distribution. Hence,

$$\mathbb{P}(\underset{j\in A}{max}\mid T_j\mid >c_n,D\subset {\widehat{S}}_n\mid {\mathcal{D}}_1,{\mathcal{D}}_2)=\mathbb{P}(\underset{j\in A}{max}\mid T_j\mid >t_{\alpha \mid 2m},D\subset {\widehat{S}}_n\mid {\mathcal{D}}_1,{\mathcal{D}}_2)+a_n\le \alpha +a_n$$

where a_n = o(1), since |A| ≤ m. It follows that

$$\mathbb{P}(\underset{j\in A}{max}\mid T_j\mid >c_n,D\subset {\widehat{S}}_n)\le \alpha +o(1).$$

Proof of Theorem 4.2

From Theorem 4.1, ℙ(D̂ _n ∩ D^c ≠ ∅) ≤ α_n and so ℙ(D̂ _n ∩ D^c ≠ ∅) → 0. Hence, ℙ(D̂ _n ⊂ D) → 1. It remains to be shown that

$$\mathbb{P}(D\subset {\widehat{D}}_n)\to 1.$$ (19)

The test statistic for testing β_j = 0 when Ŝ _n = M is

$$T_j(M)=\frac{e_j^T{(X_M^T{X}_M)}^{-1}{X}_M^{T}Y}{\widehat{\sigma }\sqrt{e_j^T{(X_M^T{X}_M)}^{-1}{e}_j}}.$$

For simplicity in the proof, let us take σ̂ = σ, the extension to unknown σ being straightforward. Let j ∈ D, ℳ = {M: |M| ≤ k_n, D ⊂ M}. Then,

$$\begin{array}{l}\mathbb{P}(j\notin {\widehat{D}}_n)=\mathbb{P}(j\notin {\widehat{D}}_n,D\subset {\widehat{S}}_n)+\mathbb{P}(j\notin {\widehat{D}}_n,D\not\subset {\widehat{S}}_n)\\ \le \mathbb{P}(j\notin {\widehat{D}}_n,D\subset {\widehat{S}}_n)+\mathbb{P}(D\not\subset {\widehat{S}}_n)\\ =\mathbb{P}(j\notin {\widehat{D}}_n,D\subset {\widehat{S}}_n)+o(1)\\ =\sum _{M\in \mathcal{M}}\mathbb{P}(j\notin {\widehat{D}}_n,{\widehat{S}}_n=M)+o(1)\\ \le \sum _{M\in \mathcal{M}}\mathbb{P}(\mid T_j(M)\mid <c_n,{\widehat{S}}_n=M)+o(1)\\ \le \sum _{M\in \mathcal{M}}\mathbb{P}(\mid T_j(M)\mid <c_n)+o(1).\end{array}$$

Conditional on ∪ , for each M ∈ ℳ, T_j(M) = (β_j/s_j) + Z where Z ~ N (0, 1). Without loss of generality assume that β_j > 0. Hence,

$$\mathbb{P}(\mid T_j(M)\mid <c_n\mid {\mathcal{D}}_1\cup {\mathcal{D}}_2)=\mathbb{P}\left(-c_n-\frac{{\beta }_j}{s_j}<Z<c_n-\frac{{\beta }_j}{s_j}\right).$$

Fix a small ε > 0. Note that $s_j^2\le {\sigma }^2/(n\kappa )$. It follows that, for all large n, $c_n-{\beta }_j/s_j<-\epsilon \sqrt{n}$. So,

$$\mathbb{P}(\mid T_j(M)\mid <c_n\mid {\mathcal{D}}_1\cup {\mathcal{D}}_2)\le \mathbb{P}(Z<-\epsilon \sqrt{n})\le e^{-n{\epsilon }^2/2}.$$

The number of models in ℳ is

$$\sum _{j=0}^{k_n}\left(\begin{array}{c}{p}_n-s\\ j-s\end{array}\right)\le k_n\left(\begin{array}{c}{p}_n-s\\ k_n-s\end{array}\right)\le k_n{\left(\frac{(p_n-s)e}{k_n-s}\right)}^{k_n-s}\le k_n{p}_n^{k_n}$$

where we used the inequality

$$\left(\begin{array}{c}n\\ k\end{array}\right)\le {\left(\frac{ne}{k}\right)}^k.$$

So,

$$\sum _{M\in \mathcal{M}}\mathbb{P}(\mid T_j(M)\mid <c_n\mid {\mathcal{D}}_1\cup {\mathcal{D}}_2)\le k_n{p}_n^{k_n}{e}^{-n{\epsilon }^2}\to 0$$

by (A2). We have thus shown that ℙ(j ∉ D̂ _n) → 0 for each j ∈ D. Since |D| is finite, it follows that ℙ(j ∉ D̂ _n for some j ∈ D) → 0 and hence (19).

Proof of Theorem 4.5

A simple modification of Theorem 3.1 of Barron, Cohen, Dahmen and DeVore (2008) shows that

$$L(k_n)=\frac{1}{n}\parallel {\widehat{Y}}_{k_n}-X\beta \mid {\mid }^2=o_P(1).$$

(The modification is needed because Barron, Cohen, Dahmen and DeVore (2008) require Y to be bounded while we have assumed that Y is Normal. By a truncation argument, we can still derive the bound on L(k_n).) So

$$\parallel {\widehat{\beta }}_{k_n}-\beta \mid {\mid }^2\le \frac{L(k_n)}{{\phi }_n(k_n+s)}\le \frac{L(k_n)}{\kappa }=o_P(1).$$

Hence, for any ε > 0, with probability tending to 1, ||β̂ (k_n) − β||² < ε so that |β̂ _j| > ψ/2 > 0 for all j ∈ D. Thus, ℙ(D ⊂ Ŝ _n(k_n)) → 1. The remainder of the proof of part 1 is the same as in Theorem 4.1. Part 2 follows from the previous result together with Theorem 3.2. The proof of Part 3 is the same as for Theorem 4.1.

Proof of Theorem 4.6

Note that $\widehat{{\mu }_j}-{\mu }_j=n^{-1}{\sum }_{i=1}^n{X}_{ij}{\epsilon }_i$. Hence, $\widehat{{\mu }_j}-{\mu }_j\sim N(0,1/n)$. So, for any δ > 0,

$$\begin{array}{l}\mathbb{P}(\underset{j}{max}\mid {\widehat{\mu }}_j-{\mu }_j\mid >\delta )\le \sum _{j=1}^{p_n}\mathbb{P}(\mid {\widehat{\mu }}_j-{\mu }_j\mid >\delta )\\ \le \frac{p_n}{\delta \sqrt{n}}{e}^{-n{\delta }^2/2}\le \frac{c_1{e}^{n^{c_2}}}{\delta \sqrt{n}}{e}^{-n{\delta }^2/2}\to 0.\end{array}$$

By (12), conclude that D ⊂ Ŝ _n(λ) when λ = μ̂ _(kn). The remainder of the proof is the same as the proof of Theorem 4.5.

Proof of Theorem 4.7

Let A = Ŝ _n ∩ D^c. We want to show that

$$\mathbb{P}(\underset{j\in A}{max}\mid T_j\mid >c_n)\le \alpha +o(1).$$

For fixed A, β̂ _A is Normal with mean 0 but this is not true for random A. Instead we need to bound T_j. Recall that

$$T_j(M)=\frac{e_j^T{(X_M^T{X}_M)}^{-1}{X}_M^{T}Y}{\widehat{\sigma }\sqrt{e_j^T{(X_M^T{X}_M)}^{-1}{e}_j}}=\frac{{\widehat{\beta }}_{M,j}}{s_j}$$

where M = Ŝ_n, $s_j^2={\widehat{\sigma }}^2{e}_j^T{(X_M^T{X}_M)}^{-1}{e}_j$ and e_j = (0, …, 0, 1, 0, …, 0)^T where the 1 is in the j^th coordinate. The probabilities that follow are conditional on but this is supressed for notational convenience. First, write

$$\begin{array}{l}\mathbb{P}(\underset{j\in A}{max}\mid T_j\mid >c_n)=\mathbb{P}(\underset{j\in A}{max}\mid T_j\mid >c_n,D\subset {\widehat{S}}_n)+\mathbb{P}(\underset{j\in A}{max}\mid T_j\mid >c_n,D\not\subset {\widehat{S}}_n)\\ \le \mathbb{P}(\underset{j\in A}{max}\mid T_j\mid >c_n,D\subset {\widehat{S}}_n)+\mathbb{P}(D\not\subset {\widehat{S}}_n)\\ \le \mathbb{P}(\underset{j\in A}{max}\mid T_j\mid >c_n,D\subset {\widehat{S}}_n)+o(1).\end{array}$$

When D ⊂ Ŝ _n,

$${\widehat{\beta }}_{{\widehat{S}}_n}={\beta }_{{\widehat{S}}_n}+{\left(\frac{1}{n}{X}_{{\widehat{S}}_n}^T{X}_{{\widehat{S}}_n}\right)}^{-1}\frac{1}{n}{X}_{{\widehat{S}}_n}^T\epsilon ={\beta }_{{\widehat{S}}_n}+Q_{{\widehat{S}}_n}{\gamma }_{{\widehat{S}}_n}$$

where $Q_{{\widehat{S}}_n}={((1/n)X_{{\widehat{S}}_n}^T{X}_{{\widehat{S}}_n})}^{-1},{\gamma }_{{\widehat{S}}_n}=n^{-1}{X}_{{\widehat{S}}_n}^T\epsilon$, and β_{Ŝ n} (j) = 0 for j ∈A. Now, $s_j^2\ge {\widehat{\sigma }}^2/(nC)$ so that

$$\mid T_j\mid \le \frac{\sqrt{nC}\mid {\widehat{\beta }}_{{\widehat{S}}_n,j}\mid }{\widehat{\sigma }}\le \frac{\sqrt{\mathit{n}loglog\mathit{n}}\mid {\widehat{\beta }}_{{\widehat{S}}_n,j}\mid }{\widehat{\sigma }}$$

for j ∈ Ŝ _n. Therefore,

$$\mathbb{P}(\underset{j\in A}{max}\mid T_j\mid >c_n,D\subset {\widehat{S}}_n)\le \mathbb{P}\left(\underset{j\in A}{max}\mid {\widehat{\beta }}_{{\widehat{S}}_n,j}\mid >\frac{\widehat{\sigma }{c}_n}{\sqrt{nC}},D\subset {\widehat{S}}_n\right).$$

Let γ = n⁻¹X^Tε. Then,

$$\parallel {\widehat{\beta }}_A\mid {\mid }^2\le {\gamma }_{{\widehat{S}}_n}^T{Q}_{{\widehat{S}}_n}^2{\gamma }_{{\widehat{S}}_n}\le \frac{\parallel {\gamma }_{{\widehat{S}}_n}\mid {\mid }^2}{{\kappa }^2}\le \frac{k_n{max}_{1\le j\le p_n}{\gamma }_j^2}{{\kappa }^2}.$$

It follows that

$$\underset{j\in A}{max}\mid {\widehat{\beta }}_{{\widehat{S}}_n,j}\mid \le \frac{\sqrt{k_n}{max}_{1\le j\le p_n}\mid {\gamma }_j\mid }{\kappa }\le \sqrt{k_{\mathit{n}}loglog\mathit{n}}\underset{1\le j\le p_n}{max}\mid {\gamma }_j\mid$$

since κ > 0. So,

$$\mathbb{P}\left(\underset{j\in A}{max}\mid {\widehat{\beta }}_{{\widehat{S}}_n,j}\mid >\frac{\widehat{\sigma }{c}_n}{\sqrt{\mathit{n}loglog\mathit{n}}},D\subset {\widehat{S}}_n\right)\le \mathbb{P}\left(\underset{1\le j\le p_n}{max}\mid {\gamma }_j\mid >\frac{\widehat{\sigma }{c}_n}{loglog\mathit{n}\sqrt{n{k}_n}}\right).$$

Note that γ_j ~ N (0, σ²/n) and hence

$$\mathbb{E}(\underset{j}{max}\mid {\gamma }_j\mid )\le \sqrt{\frac{2{\sigma }^{2}log(2p_n)}{n}}.$$

There exists ε_n → 0 such that ℙ(B_n) → 1 where B_n = {(1 − ε_n) ≤ σ̂ /σ ≤ (1 + ε)}. So,

$$\begin{array}{l}\mathbb{P}\left(\underset{1\le j\le p_n}{max}\mid {\gamma }_j\mid >\frac{\widehat{\sigma }{c}_n}{loglog\mathit{n}\sqrt{n{k}_n}}\right)\le \mathbb{P}\left(\underset{1\le j\le p_n}{max}\mid {\gamma }_j\mid >\frac{\widehat{\sigma }{c}_n(1-{\epsilon }_n)}{loglog\mathit{n}\sqrt{n{k}_n}},B_n\right)\\ \le \frac{\sqrt{n{k}_n}}{\sigma (1-{\epsilon }_n)c_n\sqrt{loglog\mathit{n}}}\mathbb{E}(\underset{j}{max}\mid {\gamma }_j\mid )\le \alpha +o(1).\end{array}$$

8. Discussion

The multi-stage method presented in this paper successfully controls type I error while giving reasonable power. The lasso and stepwise have similar performance. Although theoretical results assume independent data for each of the three stages, simulations suggest that leave-one-out cross-validation leads to valid Type I error rates and greater power. Screening the data in one phase of the experiment and cleaning in a followup phase leads to an efficient experimental design. Certainly this approach deserves further theoretical investigation. In particular, the question of optimality is an open question.

The literature on high dimensional variable selection is growing quickly. The most important deficiency in much of this work, including this paper, is the assumption that the model Y = X^Tβ + ε is correct. In reality, the model is at best an approximation. It is possible to study linear procedures when the linear model is not assumed to hold as in Greenshtein and Ritov (2004). We discuss this point in the appendix. Nevertheless, it seems useful to study the problem under the assumption of linearity to gain insight into these methods. Future work should be directed at exploring the robustness of the results when the model is wrong.

Other possible extensions include: dropping the Normality of the errors, permitting non-constant variance, investigating the optimal sample sizes for each stage, and considering other screening methods besides cross-validation.

Finally let us note that the example involving unfaithfulness, that is, cancellations of parameters to make the marginal correlation much different than the regression coefficient, pose a challenge for all the methods and deserve more attention even in cases of small p.

Appendix

Prediction

Realistically, there is little reason to believe that the linear model is correct. Even if we drop the assumption that the linear model is correct, sparse methods like the lasso can still have good properties as shown in Greenshtein and Ritov (2004). In particular, they showed that the lasso satisfies a risk consistency property. In this appendix we show that this property continues to hold if λ is chosen by cross-validation.

The lasso estimator is the minimizer of ${\sum }_{i=1}^n{(Y_i-X_i^T\beta )}^2+\lambda \parallel \beta \mid {\mid }_1$. This is equivalent to minimizing ${\sum }_{i=1}^n{(Y_i-X_i^T\beta )}^2$ subject to ||β||₁ ≤ Ω, for some Ω. (More precisely, the set of estimators as λ varies is the same as the set of estimators as Ω varies.) We use this second version throughout this section.

The predictive risk of a linear predictor ℓ(x) = x^Tβ is R(β) = (Y − ℓ(x))² where (X, Y) denotes a new observation. Let γ = γ(β) = (−1, β₁, …, β_p)^T and let Γ = (ZZ^T) where Z = (Y, X₁, …, X_p). Then we can write R(β) = γ^TΓγ. The lasso estimator can now be written as β̂ (Ω_n) = argmin_β∈B(Ωn) R̂ (β) where R̂ (β) = γ^TΓ̂ γ and $\widehat{\mathrm{\Gamma }}=n^{-1}{\sum }_{i=1}^n{Z}_i{Z}_i^T$.

Define

$${\beta }_{\ast }={\text{argmin}}_{\beta \in B({\mathrm{\Omega }}_n)}R(\beta )$$

where

$$B({\mathrm{\Omega }}_n)=\left\{\beta :\parallel \beta \mid {\mid }_1\le {\mathrm{\Omega }}_n\right\}.$$

Thus, ℓ_*(x) = x^T β_* is the best linear predictor in the set B(Ω_n). The best linear predictor is well defined even though (Y | X) is no longer assumed to be linear. Greenshtein and Ritov (2004) call an estimator β̂ _n persistent, or predictive risk consistent, if

$$R({\widehat{\beta }}_n)-R({\beta }_{\ast })\stackrel{P}{\to }0$$

as n → ∞.

The assumptions we make in this section are:

• (B1) p_n ≤ e^nξ for some 0 ≤ ξ < 1 and

• (B2) The elements of Γ̂ satisfy an exponential inequality:

  $$\mathbb{P}(\mid {\widehat{\mathrm{\Gamma }}}_{jk}-{\mathrm{\Gamma }}_{jk}\mid >\epsilon )\le c_3{e}^{-n{c}_4{\epsilon }^2}$$

  for some c₃, c₄ > 0 and

• (B3) There exists B₀ < ∞ such that, for all n, max_j;k (|Z_jZ_k|) ≤ B₀.

Condition (A2) can easily be deduced from more primitive assumptions as in Greenshtein and Ritov (2004) but for simplicity we take (A2) as an assumption. Let us review one of the results in Greenshtein and Ritov (2004). For the moment, replace (A1) with the assumption that p_n ≤ n^b for some b. Under these conditions, it follows that

$${\mathrm{\Delta }}_n\equiv \underset{j,k}{max}\mid {\widehat{\mathrm{\Gamma }}}_{jk}-{\mathrm{\Gamma }}_{jk}\mid =O_P\left(\sqrt{\frac{logn}{n}}\right).$$

Hence,

$$\begin{array}{l}\underset{\beta \in B({\mathrm{\Omega }}_n)}{sup}\mid R(\beta )-\widehat{R}(\beta )\mid =\underset{\beta \in B({\mathrm{\Omega }}_n)}{sup}\mid {\gamma }^T(\mathrm{\Gamma }-\widehat{\mathrm{\Gamma }})\gamma \mid \\ \le {\mathrm{\Delta }}_n\underset{\beta \in B({\mathrm{\Omega }}_n)}{sup}\parallel \gamma \mid {\mid }_1^2={\mathrm{\Omega }}_n^2{O}_P\left(\sqrt{\frac{logn}{n}}\right).\end{array}$$

The latter term is o_P (1) as long as Ω_n = o((n/log n)^1/4). Thus we have:

Theorem 8.1 (Greenshtein and Ritov 2004)

If Ω_n = o((n/log n)^1/4) then the lasso estimator is persistent.

For future reference, let us state a slightly different version of their result that we will need. We omit the proof.

Theorem 8.2

Let γ > 0 be such that ξ + γ < 1. Let Ω_n = O(n^{(1−ξ−γ)/4}). Then, under (B1) and (B2),

$$\mathbb{P}\left(\underset{\beta \in B({\mathrm{\Omega }}_n)}{sup}\mid \widehat{R}(\beta )-R(\beta )\mid >\frac{1}{n^{\gamma /4}}\right)=O(e^{-c{n}^{\gamma /2}})$$ (20)

for some c > 0.

The estimator β̂ (Ω_n) lies on the boundary of the ball B(Ω_n) and is very sensitive to the exact choice of Ω_n. A potential improvement—and something that reflects actual practice—is to compute the set of lasso estimators β̂ (ℓ) for 0 ≤ ℓ ≤ Ω_n and then select from that set based on cross validation. We now confirm that the resulting estimator preserves persistence. As before we split the data into and . Construct the lasso estimators {β̂ (ℓ): 0 ≤ ℓ ≤ Ω_n}. Choose ℓ̂ by cross validation using . Let β̂ = β̂ (ℓ̂ ).

Theorem 8.3

Let γ > 0 be such that ξ + γ < 1. Under (A1), (A2) and (A3), if Ω_n = O(n^{(1−ξ−γ)/4}). then the cross validated lasso estimator β̂ is persistent. Moreover,

$$R(\widehat{\beta })-\underset{0\le \ell \le {\mathrm{\Omega }}_n}{inf}R(\widehat{\beta }(\ell ))\stackrel{P}{\to }0.$$ (21)

Proof

Let β_*(ℓ) = argmin_β∈B(ℓ)R(β). Define h(ℓ) = R(β_*(ℓ)), g(ℓ) = R(β̂ (ℓ)) and c(ℓ) = L̂ (β̂ (ℓ)). Note that, for any vector b, we can write R(b) = τ² + b^TΣb −2b^T ρ where ρ = ((Y X₁), …, (Y X_p))^T.

Clearly, h is monotone nonincreasing on [0, Ω_n]. We claim that |h(ℓ + δ) − h(ℓ)| ≤ cΩ_nδ where c depends only on Γ. To see this, let u = β_*(ℓ), v = β_*(ℓ + δ) and a = ℓ β_*(ℓ + δ)/(ℓ + δ) so that a ∈ B(ℓ). Then,

$$\begin{array}{l}h(\ell +\delta )\le h(\ell )=R(u)\le R(a)\\ =R(v)+R(a)-R(v)=h(\ell +\delta )+\frac{2\delta }{\ell +\delta }{\rho }^{T}v-\frac{\delta (2\ell +\delta )}{{(\ell +\delta )}^2}{v}^T\mathrm{\sum }v\\ \le h(\ell +\delta )+2\delta C+\delta (2{\mathrm{\Omega }}_n+\delta )C\end{array}$$

where C = max_j,k |Γ_j,k| = O(1).

Next we claim that g(ℓ) is Lipschitz on [0, Ω_n] with probability tending to 1. Let β̂ (ℓ) = argmin_{β∈B̂ (ℓ)}R̂ (β) denote the lasso estimator and set û = β̂ (ℓ) and v̂ = β̂ (ℓ + δ). Let ε_n = n^−γ/4. From (20), the following chain of equations hold except on a set of exponentially small probability:

$$\begin{array}{l}g(\ell +\delta )=R(\widehat{v})\le \widehat{R}(\widehat{v})+{\epsilon }_n\le \widehat{R}(v)+{\epsilon }_n\\ \le R(v)+2{\epsilon }_n=h(\ell +\delta )+2{\epsilon }_n\\ \le h(\ell )+c{\mathrm{\Omega }}_n\delta +2{\epsilon }_n=R(u)+c{\mathrm{\Omega }}_n\delta +2{\epsilon }_n\\ \le R(\widehat{u})+c{\mathrm{\Omega }}_n\delta +2{\epsilon }_n=g(\ell )+c{\mathrm{\Omega }}_n\delta +2{\epsilon }_n.\end{array}$$

A similar argument can be applied in the other direction. Conclude that

$$\mid g(\ell +\delta )-g(\ell )\mid \le c{\mathrm{\Omega }}_n\delta +2{\epsilon }_n$$

except on a set of small probability.

Now let A = {0, δ, 2δ, …, mδ} where m is the smallest integer such that mδ ≥ Ω_n. Thus, m ~ Ω_n/δ_n. Choose δ = δ_n = n^{−3(1−ξ−γ)/8}. Then Ω_nδ_n → 0 and Ω_n/δ_n ≤ n^{3(1−ξ−γ)/4}. Using the same argument as in the proof of Theorem 3.2,

$$\underset{\ell \in A}{max}\mid \widehat{L}(\widehat{\beta }(\ell ))-R(\widehat{\beta }(\ell ))\mid ={\sigma }_n$$

where σ_n = o_P (1). Then,

$$\begin{array}{l}R({\beta }_{\ast }({\mathrm{\Omega }}_n))\le R(\widehat{\beta })\le \widehat{L}(\widehat{\beta }(\widehat{\ell }))+{\sigma }_n\\ \le \widehat{L}(m{\delta }_n)+{\sigma }_n\le g(m{\delta }_n)+2{\sigma }_n\le g({\mathrm{\Omega }}_n)+2{\sigma }_n+c{\mathrm{\Omega }}_n{\delta }_n\\ \le h({\mathrm{\Omega }}_n)+2{\sigma }_n+{\epsilon }_n+c{\mathrm{\Omega }}_n{\delta }_n=R({\beta }_{\ast }({\mathrm{\Omega }}_n))+2{\sigma }_n+{\epsilon }_n+c{\mathrm{\Omega }}_n{\delta }_n\end{array}$$

and persistence follows. To show the second result, let β̃ = argmin_{0≤ℓ≤Ωn} g(ℓ) and β̄ = argmin_ℓ∈A g(ℓ). Then,

$$\begin{array}{l}R(\stackrel{\sim }{\beta })\le \widehat{L}(\stackrel{\sim }{\beta })+{\sigma }_n\le \widehat{L}(\overline{\beta })+{\sigma }_n\\ \le R(\overline{\beta })+2{\sigma }_n\le R(\stackrel{\sim }{\beta })+2{\sigma }_n+c{\delta }_n{\mathrm{\Omega }}_n\end{array}$$

and the claim follows.