A Perturbation Method for Inference on Regularized Regression Estimates

Abstract

Analysis of high dimensional data often seeks to identify a subset of important features and assess their effects on the outcome. Traditional statistical inference procedures based on standard regression methods often fail in the presence of high-dimensional features. In recent years, regularization methods have emerged as promising tools for analyzing high dimensional data. These methods simultaneously select important features and provide stable estimation of their effects. Adaptive LASSO and SCAD for instance, give consistent and asymptotically normal estimates with oracle properties. However, in finite samples, it remains difficult to obtain interval estimators for the regression parameters. In this paper, we propose perturbation resampling based procedures to approximate the distribution of a general class of penalized parameter estimates. Our proposal, justified by asymptotic theory, provides a simple way to estimate the covariance matrix and confidence regions. Through finite sample simulations, we verify the ability of this method to give accurate inference and compare it to other widely used standard deviation and confidence interval estimates. We also illustrate our proposals with a data set used to study the association of HIV drug resistance and a large number of genetic mutations.

1. INTRODUCTION

Accurate prediction of disease outcomes is fundamental for successful disease prevention and treatment selection. Recent advancement in biological and genomic research has led to the discovery of a vast number of new markers that can potentially be used to develop molecular disease prevention and intervention strategies. For example, gene expression analyses have identified molecular subtypes that are associated with differential prognosis and response to treatment for breast cancer patients (Perou et al. 2000; Dent et al. 2007). For non-small cell lung cancer patients, a composite score consisting of several biological markers including cyclin E and Ki-67 was shown to be highly predictive of patient survival (Dosaka-Akita et al. 2001). However, construction of accurate prediction models with a panel of markers is a difficult task in general. For example, statistical models for calculating individual cancer risk have been developed for a few types of cancer in the past two decades (Gail et al. 1989; Thompson et al. 2006; Cassidy et al. 2008; Freedman et al. 2009). However, much refinement is needed even for the best of these models due to their limited discriminatory accuracy (Spiegelman et al. 1994; Gail and Costantino 2001).

The increasing availability of new potential markers, while holding great promise for better prediction of disease outcomes, imposes challenges to model development due to the high dimensionality in the feature space and the relatively small sample size. To improve prediction with a large number of promising genomic or biological markers, an important step is to build a parsimonious model that only includes important markers. Such a model could reduce the cost associated with unnecessary marker measurements and improve the prediction precision for future patients. For such purposes, various regularization procedures such as the LASSO (Tibshirani 1996; Knight and Fu 2000), the SCAD (Fan and Li 2001, 2002, 2004; Zhang et al. 2006), the adaptive LASSO (ALASSO; Zou 2006; Wang and Leng 2007), the Elastic Net (Zou and Hastie 2005; Zou and Zhang 2009), and one-step local linear approximation (LLA; Zou and Li 2008) have been developed in recent years. These procedures simultaneously identify non-informative variables and produce coefficient estimates for the selected variables to induce a model for prediction.

These regularization procedures, while effective for variable selection and stable estimation, yield estimators whose distributions are difficult to approximate. LASSO type estimators have a non-standard limiting distribution that depends on which components of the coefficient vector are zero. Since the LASSO type estimator is not consistent in variable selection, the limiting distribution cannot be estimated directly. Furthermore, standard bootstrap methods fail when the true coefficient vector is sparse (Knight and Fu 2000). Recently, Chatterjee and Lahiri (2010) proposed a truncated LASSO estimator whose distribution can be approximated using a residual bootstrap procedure. To overcome the difficulties in LASSO estimators, other regularized procedures such as the SCAD and ALASSO have been proposed. These estimators possess asymptotic oracle properties including perfect variable selection and super efficiency. However, our simulation results suggest that in finite samples, such oracle properties are far from being true and inference procedures based on asymptotic properties such as those given in Zou (2006) perform poorly especially when the signal to noise ratio (SNR) is high and the between covariate correlations are not low. Recently, Pötscher and Schneider (2009, 2010) developed theory on the coverage probabilities of the confidence intervals for ALASSO type estimators under the orthogonal design. It was shown that estimating the distribution function of the ALASSO estimator is not feasible when the true parameter is of similar magnitude to $n^{-\frac{1}{2}}$, where n is the sample size. It is thus generally difficult to develop well performed confidence regions (CRs) and hypothesis testing procedures based on these regularized estimators. Such difficulties limit applicability to clinical studies where confidence in statistical evidence is crucial for clinical decision making.

In this paper, we propose resampling methods to derive CR and testing procedures for marker effects estimated from regularized procedures such as the ALASSO and one-step SCAD estimator when the true parameter is fixed. Our preliminary studies suggest that CRs constructed from such resampling procedures perform much better than their asymptotic based counterparts. When the fitted model is merely a working model, many frequently used estimation procedures may fail to produce stable parameter estimates. Procedures that can provide stable parameter estimates and valid interval estimates under a possibly misspecified working model are highly valuable when building a prediction model with high dimensional data. Our proposed procedures remain valid even if the fitted model fails to hold, provided that the employed objective function satisfies mild regularity conditions. The rest of the paper is organized as follows. In Section 2, we introduce the proposed perturbation resampling procedures and describe various methods for constructing confidence regions. In Section 3, we demonstrate the validity of the proposed procedures in finite samples via simulation studies. In Section 4, we illustrate our proposed procedure with an HIV drug resistance study where the goal is to predict phenotypic drug resistance levels using genotypic viral mutations.

2. RESAMPLING PROCEDURES

Suppose that y = (y₁, … y_n)^⊤ is the n × 1 vector of response variables and x_j = (x_1i, …, x_pi)^⊤, i = 1 … n, are the predictors. Let X = [x₁, …, x_n]^⊤ be the n × p matrix of these covariates. Assume that the effect of x on y is determined via an objective function (θ; ) = ℓ(y, α + β^⊤x), where θ = (α, β^⊤)^⊤, α is an unknown location parameter, β is an unknown p × 1 vector of covariate effects, and = (y, x^⊤)^⊤. To assess the association between x and y, let $\stackrel{\sim }{\mathcal{L}}(\mathit{\theta })=n^{-1}{\sum }_{i=1}^n\mathcal{L}(\mathit{\theta };{\mathcal{D}}_i)$ be the objective function used to fit a regression model and θ̃ = (α̃, β̃^⊤)^⊤ = argmin_θ (θ). To obtain a regularized estimator for θ₀, we minimize the regularized objective function

$$\widehat{\mathcal{L}}(\mathit{\theta })=\stackrel{\sim }{\mathcal{L}}(\mathit{\theta })+\sum _{j=1}^p{p}_{{\lambda }_{n}j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j\mid )\mid {\beta }_j\mid$$ (1)

where $p_{{\lambda }_{n}j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j\mid )$ is the derivative of a penalty p_λnj (|β_j|) evaluated at the initial estimate of β_0j. We consider the cases where p_λnj (|β_j|) is the concave SCAD penalty or the L_q penalty for 0 < q < 1, and utilize a one-step estimator of these penalties with the local linear approximation (LLA) method proposed by Zou and Li (2008). Additionally, we consider the ALASSO penalty of Zou (2006) that arises when $p_{{\lambda }_{n}j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j\mid )=n^{-\frac{1}{2}}{\lambda }_n{\mid {\stackrel{\sim }{\beta }}_j\mid }^{-1}$.

2.1 Regularity Conditions

To ensure the asymptotic oracle properties of the regularized estimators and the validity of the proposed resampling procedures, we require the following set of conditions:

• C1ℙ{ (θ; )} has a unique minimum at θ₀ and a continuous secondary derivative with a positive definite = ∂²ℙ{ (θ; )}/∂θθ^⊤|_{θ = θ0} > 0, where ℙ is the probability measure generated by the data = { , i = 1, …, n}.
• C2The class of functions indexed by θ, { (θ; ) | θ ∈ Ω}, is Glivenko-Cantelli (Kosorok 2008), where = (y, x^T )^T and Ω is the compact parameter space containing θ₀.
• C3There exists a “qausi-derivative” function (θ; ) for (θ; ) such that for any positive sequence δ_n → 0

  1. ℙ{ (θ₀; )} = , a positive definite matrix.

  2. $\mathbb{P}\{\mathcal{L}(\mathit{\theta };\mathcal{D})-\mathcal{L}({\mathit{\theta }}_0;\mathcal{D})-\mathcal{U}({\mathit{\theta }}_0;\mathcal{D})(\mathit{\theta }-{\mathit{\theta }}_0)\}=\frac{1}{2}{(\mathit{\theta }-{\mathit{\theta }}_0)}^T\mathbb{A}(\mathit{\theta }-{\mathit{\theta }}_0)+o({\left|\right|\mathit{\theta }-{\mathit{\theta }}_0\left|\right|}^2)$, where ||θ − θ₀|| ≤ δ_n.

  3. ${\mathbb{P}}_n\{\mathcal{L}({\mathit{\theta }}_1;\mathcal{D})-\mathcal{L}({\mathit{\theta }}_2;\mathcal{D})-\mathcal{U}({\mathit{\theta }}_2;\mathcal{D})({\mathit{\theta }}_1-{\mathit{\theta }}_2)\}=\frac{1}{2}{({\mathit{\theta }}_1-{\mathit{\theta }}_2)}^T\mathbb{A}({\mathit{\theta }}_1-{\mathit{\theta }}_2)+o({\left|\right|{\mathit{\theta }}_1-{\mathit{\theta }}_2\left|\right|}^2+n^{-1/2}||{\mathit{\theta }}_1-{\mathit{\theta }}_2||)$, almost surely, uniformly in ||θ₁ − θ₀|| ≤ δ_n, ||θ₂ − θ₀|| ≤ δ_n.

These conditions are parallel to the conditions required in Proposition A1–A3 in Jin et al. (2001). These regularity conditions hold for commonly used L₂ minimization with (β; ) = (y − β^⊤x)² and L₁ minimization with (β; ) = |y − β^⊤x|. Details of the justification for these two cases can be found in Section 3 of Jin et al. (2001). These conditions also guarantee that θ̃ is a consistent estimator of θ₀ and $n^{-\frac{1}{2}}(\stackrel{\sim }{\mathit{\theta }}-{\mathit{\theta }}_0)$ converges in distribution to N(0, ). Let = {j : β_0j ≠ 0} of size q and = {j : β_0j = 0}, where a_j denotes the jth component of a vector a.

Following similar arguments to those given in Zou (2006), Zou and Li (2008) and the unconditional arguments given in the appendix, θ̂ = argmin_θ (θ) has ‘good’ properties for certain choices of λ_n, including the oracle property,

Lemma 1: (Oracle properties)

Suppose that λ_n → 0 and ${\lambda }_n{n}^{\frac{1}{2}}\to \infty$. Then the regularized estimates must satisfy the following:

1. Consistency in variable selection: lim_nℙ{I ( = ) = 1} = 1, where = {j : β_j ≠ 0}.

2. Asymptotic normality: $n^{\frac{1}{2}}({\widehat{\mathit{\theta }}}_{\mathcal{A}}-{\mathit{\theta }}_{0\mathcal{A}}){\to }_d{}^{\ast }\text{N}(\mathbf{0},{\mathbb{A}}_{11}^{-1}{\mathbb{B}}_{11}{\mathbb{A}}_{11}^{-1})$, where and are the respective q × q submatrices of and corresponding to .

This lemma guarantees that the regularized estimate asymptotically chooses the correct model and has the optimal estimation rate. However, estimating the distribution of $n^{\frac{1}{2}}(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)$ in finite samples remains difficult. To estimate the standard errors of the SCAD estimates $\widehat{\mathit{\theta }}={\text{argmin}}_{\mathit{\theta }}\{\stackrel{\sim }{\mathcal{L}}(\mathit{\theta })+{\sum }_{j=1}^p{p}_{{\lambda }_{n}j}(\mid {\beta }_j\mid )\}$ when $\stackrel{\sim }{\mathcal{L}}(\mathit{\theta })=n^{-1}{\sum }_{i=1}^n\mathcal{L}(\mathit{\theta };{\mathcal{D}}_i)$ is smooth in θ, Fan and Li (2001) proposed a local quadratic approximation (LQA) method. This gives a sandwich estimator for the covariance matrix of the estimated nonzero parameters:

$$\widehat{cov}\left({\widehat{\mathit{\theta }}}_{\widehat{\mathcal{A}}}\right)={\{{\nabla }^2\stackrel{\sim }{\mathcal{L}}({\widehat{\mathit{\theta }}}_{\widehat{\mathcal{A}}})+{\mathrm{\sum }}_{\lambda }({\widehat{\mathit{\theta }}}_{\widehat{\mathcal{A}}})\}}^{-1}\widehat{cov}\{\nabla \stackrel{\sim }{\mathcal{L}}({\widehat{\mathit{\theta }}}_{\widehat{\mathcal{A}}})\}{\{{\nabla }^2\stackrel{\sim }{\mathcal{L}}({\widehat{\mathit{\theta }}}_{\widehat{\mathcal{A}}})+{\mathrm{\sum }}_{\lambda }({\widehat{\mathit{\theta }}}_{\widehat{\mathcal{A}}})\}}^{-1}$$ (2)

where ∇ (θ̂) = ∂ (θ̂/∂θ, ∇² (θ̂) = ∂² (θ̂)/∂θ∂θ^T, and Σ_λ(θ̂) is a diagonal matrix with the (j, j)th element being $I({\widehat{\beta }}_j\ne 0)p_{{\lambda }_{n}1}^{\prime }(\mid {\widehat{\beta }}_j\mid )/\mid {\widehat{\beta }}_j\mid$. The LQA approach can also be used to construct a covariance estimate for the ALASSO estimates where $p_{{\lambda }_{n}j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j\mid )=n^{-\frac{1}{2}}{\lambda }_n{\mid {\stackrel{\sim }{\beta }}_j\mid }^{-1}$. Similar to covariance estimates in Tibshirani (1996) and Fan and Li (2001) for penalized estimates, this procedure estimates the standard errors for variables with β̂_j = 0 as 0. Although this sandwich estimator has been proven to be consistent (Fan and Peng 2004) under the linear regression model, it tends to underestimate the standard errors, and normal confidence regions (CRs) using this estimate often do not provide acceptable coverage in finite sample.

To approximate the covariance of θ̂ more accurately, we propose a perturbation method to estimate the distribution of $n^{\frac{1}{2}}(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)$ for a general class of objective functions and penalties. Let = {G_i, i = 1, …, n} be a set of independent and identically distributed (i.i.d.) positive random variables with mean and variance equal to one. We first perturb the initial objective function and obtain

$${\stackrel{\sim }{\mathcal{L}}}^{\ast }(\mathit{\theta })=n^{-1}\sum _{i=1}^n\stackrel{\sim }{\mathcal{L}}(\mathit{\theta },{\mathcal{D}}_i)G_i,\text{and}{\stackrel{\sim }{\mathit{\theta }}}^{\ast }=\underset{\mathit{\theta }}{\text{argmin}}{\stackrel{\sim }{\mathcal{L}}}^{\ast }(\mathit{\theta }).$$ (3)

Then with the same set , we obtain the minimizer of a stochastically perturbed version of the regularized objective function:

$${\widehat{\mathcal{L}}}^{\ast }(\mathit{\theta })={\stackrel{\sim }{\mathcal{L}}}^{\ast }(\mathit{\theta })+\sum _{j=1}^p{p}_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )\mid {\beta }_j\mid$$ (4)

where ${\lambda }_n^{\ast }$ satisfies the same order constraints as λ_n as discussed in the Lemma 1. In practice, one may select λ_n and ${\lambda }_n^{\ast }$ based on the BIC criterion detailed in the appendix with the corresponding objective functions. In the appendix we first show that $n^{\frac{1}{2}}({\widehat{\mathit{\theta }}}_{\mathcal{A}}^{\ast }-{\mathit{\theta }}_{0\mathcal{A}})$ converges in distribution to $N(\mathbf{0},{\mathbb{A}}_{11}^{-1}{\mathbb{B}}_{11}{\mathbb{A}}_{11}^{-1})$, the same limiting distribution of $n^{\frac{1}{2}}(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)$. Furthermore, ${\mathbb{P}}^{\ast }({\widehat{\mathit{\theta }}}_{{\mathcal{A}}^c}^{\ast }=0)\to 1$, where ℙ^* is the probability measure generated by both and . In addition, we show that the distribution of $n^{\frac{1}{2}}({\widehat{\mathit{\theta }}}_{\mathcal{A}}^{\ast }-{\widehat{\mathit{\theta }}}_{\mathcal{A}})$ conditional on the data can be used to approximate the unconditional distribution of $n^{\frac{1}{2}}({\widehat{\mathit{\theta }}}_{\mathcal{A}}-{\mathit{\theta }}_{\mathcal{A}0})$ and that ${\mathbb{P}}^{\ast }({\widehat{\mathit{\theta }}}_{{\mathcal{A}}^c}^{\ast }=0\mid \mathcal{X})\to 1$. In practice, these results allow us to estimate the distribution of $n^{\frac{1}{2}}(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)$ by generating a large number, M, say, of random samples . We obtain ${\widehat{\mathit{\theta }}}_m^{\ast }$ by minimizing the perturbed objective function for each sample m = 1, … M, and then approximate the theoretical distribution of θ̂ by the empirical distribution { ${\widehat{\mathit{\theta }}}_m^{\ast }$, m = 1, … M}. Specifically, the covariance matrix of θ̂ can be estimated by the sample covariance matrix constructed from { ${\widehat{\mathit{\theta }}}_m^{\ast }$, m = 1, … M}.

Estimating the distribution of $n^{\frac{1}{2}}(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)$ based on the distribution of $n^{\frac{1}{2}}({\widehat{\mathit{\theta }}}^{\ast }-\widehat{\mathit{\theta }})\mid \mathcal{X}$ leads to the construction of three possible (1 − α)100% confidence regions for θ₀. For the first, let ${\widehat{\sigma }}_j^2=M^{-1}{\sum }_{m=1}^M{({\widehat{\beta }}_{mj}^{\ast }-{\widehat{\beta }}_j)}^2$. We construct a normal CR for β_0j, ${\text{CR}}_j^{\ast \text{N}}$, centered at β̂_j with standard deviation ${\widehat{\sigma }}_j^{\ast }$. Since $n^{\frac{1}{2}}({\widehat{\mathit{\theta }}}^{\ast }-\widehat{\mathit{\theta }})\mid \mathcal{X}$ and $n^{\frac{1}{2}}(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)$ converge to the same normal distribution, $n{\widehat{\sigma }}_j^2$ consistently estimates the variance of $n^{\frac{1}{2}}({\widehat{\beta }}_j-{\beta }_{0j})$. This method is in contrast to CR^Asym obtained with standard deviations ${\widehat{\sigma }}_j^{\text{Asym}}$ estimated with the asymptotically consistent LQA sandwich estimator in Fan and Li (2001) and Zou (2006). In contrast to setting the standard error to 0 when β̂_j = 0, we set ${\text{CR}}_j^{\ast \text{N}}=\{0\}$ if the proportion of ${\widehat{\beta }}_j^{\ast }$ being 0 is larger than a threshold p̂_high, such that p̂_high → p_high < 1. This method accounts for the superefficiency due to the oracle property and results in a shorter interval with valid coverage. Secondly, we simply take the (α/2)100th and (1 − α/2)100th quantiles of ${\widehat{\beta }}_j^{\ast }$ as the upper and lower bounds of ${\text{CR}}_j^{\ast \text{Q}}$. For the third, we estimate the density of ${\widehat{\beta }}_j^{\ast }$ with a kernel density estimator and choose the (1 − α)100% highest density region, ${\text{CR}}_j^{\ast \text{HDR}}$. We estimate the density of ${\widehat{\beta }}_j^{\ast }\mid \mathcal{X}$ as a mixed density with distribution $f_j^{\ast }(\beta )={\widehat{\mathcal{P}}}_{0j}I(\beta =0)+(1-{\widehat{\mathcal{P}}}_{0j})f_j^{\ast }(\beta )$, where is the proportion of ${\widehat{\beta }}_j^{\ast }$ set to 0, and $f_j^{\ast }(\beta )$ is the unknown distribution of ${\widehat{\beta }}_j^{\ast }$ given that it is not set to 0. To construct a highest density confidence region that has accurate coverage of this mixed distribution, we adjust the definition of the region depending on thresholds that reflect the strength of evidence for β_0j = 0. Our highest density confidence region ${\text{CR}}_j^{\ast \text{HDR}}$ is defined as

$${\text{CR}}_j^{\ast \text{HDR}}=\{\begin{array}{ll}\{0\}\hfill & \text{if}{\widehat{\mathcal{P}}}_{0j}\ge {\widehat{p}}_{\mathit{high}}\hfill \\ \{\beta :f_j^{\ast }(\beta )\ge {\widehat{c}}_1\}\cup \{0\}\hfill & \text{if}{\widehat{p}}_{\mathit{low}}\le {\widehat{\mathcal{P}}}_{0j}<{\widehat{p}}_{\mathit{high}}\hfill \\ \{\beta :f_j^{\ast }(\beta )\ge {\widehat{c}}_2\}\cup \{0\}\hfill & \text{if}\alpha \le {\widehat{\mathcal{P}}}_{0j}<max(\alpha ,{\widehat{p}}_{\mathit{low}})\hfill \\ \{\beta :f_j^{\ast }(\beta )\ge {\widehat{c}}_3\}\hfill & \text{if}{\widehat{\mathcal{P}}}_{0j}\le \alpha \hfill \end{array}$$

where ĉ₁, ĉ₂, and ĉ₃ are chosen such that for $H(c)=\int I\{f_j^{\ast }(\beta )\ge c\}{f}_j^{\ast }(\beta )d\beta$, we have H(ĉ₁) = (1 − α − )/(1 − ), H (ĉ₂) = 1 − α + α( + p̂_low), H (ĉ₃) = 1 − α, while p̂_low → 0 and p̂_high → p_high = 1 − α. When , the proportion of ${\widehat{\beta }}_j^{\ast }$ set to zero, is greater than the upper thresholding value p̂_high, we have strong evidence that β_0j = 0 and thus take {0} as the confidence interval. When is between the high and low thresholding p̂ values, we have moderately strong evidence that β_0j = 0 and thus take the mass at 0 and a 1 − α − highest density region from the ${\widehat{\beta }}_j^{\ast }\mid {\widehat{\beta }}_j^{\ast }\ne 0$ samples. The occurrence of α ≤ < max(α, p̂_low) suggests that β_0j is likely to be a weak signal. For such cases, it would be difficult to make inference about β_0j due to shrinkage. Thus, we inflate the highest density region from the ${\widehat{\beta }}_j^{\ast }\mid {\widehat{\beta }}_j^{\ast }\ne 0$ samples. Finally, when < α, we have strong evidence that β_0j is nonzero and so we take the 1 − α highest density region of the continuous empirical distribution of nonzero ${\widehat{\beta }}_j^{\ast }$ samples. The justification of this method and the choices of p̂_high and p̂_low are relegated to the appendix.

In practice, when assessing the effects of multiple features, it is often important to adjust for multiple comparisons. For interval estimation, we may construct a (1 − α)100% simultaneous confidence region to cover the entire parameter vector θ₀. We may then make statements about the importance of each of the covariates in the presence of other covariates while maintaining a type I error of α. For the regularized estimator, we define the Normal simultaneous region as ${\text{CR}}^{\ast \text{Sim}}={\prod }_{j\notin {\widehat{\mathcal{A}}}^{\ast }}\{0\}\times {\prod }_{j\in {\widehat{\mathcal{A}}}^{\ast }}({\widehat{\beta }}_j-{\gamma }_{\alpha }{\widehat{\sigma }}_j^{\ast },{\widehat{\beta }}_j+{\gamma }_{\alpha }{\widehat{\sigma }}_j^{\ast })$ where = {j : < p̂_high} and γ_α is the (1 − α) 100% quantile of ${\left\{max{\left\{\mid {\widehat{\beta }}_{jm}^{\ast }-{\widehat{\beta }}_j\mid /{\widehat{\sigma }}_j^{\ast }\right\}}_{j\in {\widehat{\mathcal{A}}}^{\ast }}\right\}}_{m=1}^M$. We define the (1 − α)100% HDR simultaneous region as ${\text{CR}}^{\ast \text{SimHDR}}={\prod }_j{\text{CR}}_{j,{\alpha }_s}^{\ast \text{HDR}}$ where ${\text{CR}}_{j,{\alpha }_s}^{\ast \text{HDR}}$ is the $1-{\alpha }_s{\text{CR}}_j^{\ast \text{HDR}}$ for β̂_j and α_s = 2(1 − Φ(γ_α)). We compare the performance of these confidence regions with numerical examples in Sections 3 and 4.

3. SIMULATION STUDIES

To examine the validity of our procedures in finite samples, we performed simulation studies to assess the performance of the corresponding confidence regions. For each setting, we simulated 1500 data sets with n observations generated under the linear model, y = Xβ + ε, where x_ij ~ (0, 1), the pairwise correlation between x_i and x_j was set to cor(x_i, x_j) = ρ, ε_i ~ (0, σ²), and β, ρ, and σ were varied between settings. In each setting, β was sparse and included medium and high signals. We obtained ALASSO estimators via LARS (Efron et al. 2004) for each simulated data set with OLS initial estimates and λ chosen by the BIC as described in the appendix, and then M = 500 perturbed samples using our proposed method with generated from a mean 1 exponential distribution. The sample size n was set to 100, 200, 400, or 1000, while ρ was 0, 0.2, or 0.5, and σ was 1 or 2. To compute the highest density regions CR^*HDR we utilized the hdrcde package in R with the “ndr” bandwidth estimator as presented in Scott (1992) based on Silverman’s rule of thumb (Silverman 1986). We chose ${\widehat{p}}_{\mathit{low}}=min\{\sqrt{2/\pi }exp(-n\lambda /(4{\widehat{\sigma }}^2)),0.49\}$ and ${\widehat{p}}_{\mathit{high}}=min\{1-\sqrt{2/\pi }exp(-n\lambda /{\widehat{\sigma }}^2),0.95\}$ as justified in the appendix for CR^*HDR, CR^*N, CR^*Sim, and CR^*SimHDR. We substituted the σ used in the standard deviation estimate from Zou (2006) analogous to equation (2) with the known σ from the simulations. We present the results for simulations with n = 100, 200 and 400 when σ = 1 or 2 and p = 10 or 20. In these cases, the true β₀ contains two large effects of β_0j = 1, two moderate effects of β_0j = 0.5, and six (for p = 10) or sixteen (for p = 20) noise parameters where β_0j = 0. To examine the effect of regularization we compare our CRs for the regularized estimators to CR^OLS, the normal CR based on the empirical standard error of the perturbed ordinary least squares (OLS) estimates.

In Tables 1 and 2 we see that when σ = 1, most regions perform similarly for nonzero parameters. When σ = 2, the perturbation regions usually have higher coverage than CR^Asym and sacrifice very little in length. The asymmetric CR^*HDR has the shortest length when β_0j = 0 for all settings. Coverage for CR^*HDR and simultaneous confidence regions can be low when n = 100 due to the difficulty of estimating at such a small sample size, but coverage reaches nominal levels by n = 200. The standard deviation estimate from Zou (2006), σ̂^Asym (also see Table 3), is not large enough to cover β_0j sufficiently, and while the coverage probability of the CR^OLS is not extremely low, it is notably outperformed by the other confidence regions when β_0j = 0. We omit the results from the settings where n = 1000 because the results have similar patterns as those with n = 400. For these large sample cases with n greater than or equal to 400 we saw convergence to 95% coverage for the normal CRs, highest density regions, and OLS CRs in all settings when the true parameter was nonzero. For true zero parameters, the coverage probabilities of our confidence regions converged to 1, while the OLS CR converged to 0.95. A tradeoff associated with our method is that while the coverage of our perturbation confidence regions tends to be higher than CR^OLS and CR^Asym, some power is sacrificed for moderate signals of β_0j = 0.5. This loss is minimal, however, and only appears in difficult cases when sample size is low and ρ and σ are high. When β_0j = 0, CR^OLS has coverage lower than 95% for small samples while our methods produce regions with coverage probability near 1 and very short lengths reflecting the oracle properties. Overall, the most disparity between our methods and previous methods is seen when the SNR is low.

Table 1.

Coverage probabilities (lengths) of confidence regions when σ = 1.

p	β0		n = 100			n = 200			n = 400
			ρ = 0	ρ = 0.2	ρ = 0.5	ρ = 0	ρ = 0.2	ρ = 0.5	ρ = 0	ρ = 0.2	ρ = 0.5
10	1	CR*N	91.6 (38)	92.7 (41)	92.5 (52)	92.9 (27)	94.1 (29)	94.2 (37)	93.9 (19)	95.1 (21)	94.9 (26)
		CR*HDR	91.4 (38)	91.7 (40)	91.5 (51)	92.4 (27)	93.7 (29)	93.9 (36)	93.4 (19)	94.5 (21)	94.1 (26)
		CR*Q	91.7 (38)	91.0 (40)	91.5 (51)	92.5 (27)	93.6 (29)	93.8 (36)	93.5 (19)	94.3 (20)	93.9 (26)
		CRAsym	93.9 (41)	94.1 (43)	93.7 (53)	94.1 (28)	94.2 (30)	94.1 (36)	94.4 (20)	95.0 (21)	94.9 (25)
		CROLS	91.4 (38)	91.7 (40)	90.6 (51)	93.0 (27)	93.6 (29)	93.9 (36)	93.0 (19)	94.3 (21)	93.7 (26)
	0.5	CR*N	93.0 (40)	93.3 (43)	92.0 (54)	93.9 (28)	94.6 (30)	95.1 (38)	93.7 (20)	94.2 (21)	95.2 (27)
		CR*HDR	91.9 (38)	92.5 (41)	93.4 (51)	93.5 (27)	94.0 (29)	93.8 (37)	93.7 (19)	93.8 (21)	94.7 (26)
		CR*Q	91.7 (39)	92.3 (42)	90.7 (53)	93.3 (27)	93.7 (29)	93.9 (37)	93.8 (19)	93.6 (21)	94.7 (26)
		CRAsym	93.3 (41)	93.5 (43)	91.5 (52)	94.5 (28)	94.3 (30)	93.7 (36)	95.0 (20)	94.0 (21)	94.1 (25)
		CROLS	92.4 (38)	91.6 (41)	90.9 (50)	93.5 (27)	94.3 (29)	93.8 (36)	94.3 (19)	93.7 (21)	94.9 (26)
	0	CR*N	97.6 (23)	98.5 (25)	98.1 (31)	98.4 (17)	98.3 (19)	97.9 (23)	98.7 (13)	98.7 (13)	98.7 (16)
		CR*HDR	99.1 (17)	99.3 (18)	99.4 (23)	99.6 (12)	99.3 (13)	99.0 (16)	99.7 (8)	99.3 (8)	99.5 (11)
		CR*Q	99.5 (31)	99.7 (33)	99.8 (43)	99.7 (22)	99.7 (24)	99.7 (30)	99.8 (16)	99.7 (17)	99.8 (21)
		CROLS	92.9 (38)	92.9 (40)	91.7 (51)	93.4 (27)	93.0 (29)	92.6 (36)	93.4 (19)	93.7 (21)	93.6 (26)
		CR*SimHDR	91.9 (36)	92.5 (39)	91.7 (49)	93.1 (26)	94.9 (28)	93.8 (36)	94.9 (19)	94.8 (20)	96.0 (25)
		CR*Sim	92.5 (42)	92.9 (46)	91.9 (58)	93.7 (31)	95.5 (33)	95.2 (42)	95.5 (23)	95.7 (24)	96.6 (30)
		CR*SimOLS	87.1 (54)	86.5 (58)	85.9 (72)	89.8 (38)	90.0 (41)	90.9 (52)	92.3 (28)	91.6 (30)	92.6 (37)
20	1	CR*N	91.7 (38)	92.4 (42)	92.5 (53)	93.3 (27)	92.9 (30)	94.3 (38)	95.4 (19)	94.6 (21)	94.5 (27)
		CR*HDR	90.2 (37)	90.9 (41)	90.8 (51)	92.4 (26)	91.9 (29)	91.9 (36)	95.1 (19)	93.6 (21)	93.3 (26)
		CR*Q	90.2 (37)	90.7 (41)	90.7 (51)	92.3 (26)	92.3 (29)	91.9 (36)	95.1 (19)	93.2 (21)	93.1 (26)
		CRAsym	93.9 (41)	94.5 (44)	93.3 (54)	95.1 (28)	93.9 (30)	94.0 (37)	95.9 (20)	94.7 (21)	93.3 (26)
		CROLS	90.3 (38)	89.9 (42)	90.0 (52)	92.6 (27)	92.1 (29)	92.0 (37)	95.3 (19)	93.4 (21)	93.3 (26)
	0.5	CR*N	91.1 (40)	91.5 (44)	91.0 (56)	93.7 (28)	93.5 (30)	92.7 (39)	93.1 (20)	94.7 (21)	95.0 (27)
		CR*HDR	89.7 (38)	90.3 (42)	92.5 (52)	93.1 (27)	93.1 (30)	91.7 (38)	92.9 (19)	93.3 (21)	94.4 (27)
		CR*Q	89.7 (39)	89.7 (43)	89.2 (54)	92.7 (27)	92.5 (30)	91.5 (38)	92.7 (19)	93.5 (21)	94.5 (27)
		CRAsym	91.7 (41)	92.0 (44)	89.7 (53)	94.5 (28)	93.2 (30)	92.3 (37)	93.7 (20)	94.3 (21)	94.2 (26)
		CROLS	89.7 (38)	89.7 (42)	89.6 (52)	92.9 (27)	92.9 (29)	91.8 (37)	92.9 (19)	92.7 (21)	94.2 (26)
	0	CR*N	96.6 (29)	97.3 (32)	96.8 (40)	98.6 (21)	98.5 (23)	98.7 (29)	98.8 (15)	99.1 (17)	99.0 (21)
		CR*HDR	97.7 (25)	98.3 (28)	98.3 (34)	98.9 (17)	99.3 (19)	99.1 (24)	99.3 (12)	99.5 (13)	99.4 (16)
		CR*Q	99.0 (31)	99.4 (34)	99.2 (43)	99.5 (22)	99.9 (24)	99.5 (30)	99.8 (15)	99.9 (17)	99.7 (21)
		CROLS	89.5 (38)	90.1 (41)	90.1 (52)	92.4 (27)	92.2 (29)	92.6 (37)	94.7 (19)	93.8 (21)	94.2 (26)
		CR*SimHDR	90.5 (42)	92.0 (47)	92.1 (58)	95.4 (31)	95.7 (34)	95.3 (44)	96.9 (22)	96.9 (25)	96.9 (32)
		CR*Sim	92.5 (51)	91.9 (57)	91.7 (71)	96.5 (38)	97.5 (42)	96.5 (54)	97.7 (28)	97.8 (31)	98.1 (40)
		CR*SimOLS	80.1 (59)	78.4 (64)	77.9 (80)	87.5 (41)	87.1 (45)	84.9 (57)	90.6 (29)	90.0 (32)	91.1 (41)

NOTE: We multiply values by 100. The lengths of the simultaneous confidence regions are averaged over the number of parameters.

Table 2.

Coverage probabilities (lengths) of confidence regions when σ = 2.

p	β0		n = 100			n = 200			n = 400
			ρ = 0	ρ = 0.2	ρ = 0.5	ρ = 0	ρ = 0.2	ρ = 0.5	ρ = 0	ρ = 0.2	ρ = 0.5
10	1	CR*N	92.6 (79)	94.3 (85)	92.9 (110)	93.4 (55)	94.4 (59)	94.9 (76)	94.4 (39)	94.2 (42)	94.7 (53)
		CR*HDR	91.7 (76)	93.9 (82)	94.0 (104)	92.7 (55)	94.2 (59)	94.4 (74)	94.0 (39)	93.6 (42)	93.7 (52)
		CR*Q	91.9 (77)	93.7 (84)	91.4 (107)	92.7 (55)	93.8 (59)	94.3 (75)	93.9 (39)	93.4 (42)	93.8 (52)
		CRAsym	80.7 (57)	82.7 (60)	78.8 (73)	82.3 (40)	83.6 (42)	80.5 (51)	83.7 (28)	83.5 (30)	80.9 (36)
		CROLS	91.7 (75)	93.4 (81)	91.1 (102)	92.7 (54)	93.4 (58)	94.0 (73)	94.0 (39)	93.7 (42)	93.4 (52)
	0.5	CR*N	87.5 (80)	87.6 (86)	81.3 (100)	93.5 (59)	94.2 (63)	90.3 (79)	94.7 (41)	95.7 (44)	94.5 (57)
		CR*HDR	90.3 (71)	91.4 (76)	83.9 (88)	95.9 (54)	96.5 (58)	92.3 (69)	93.5 (39)	94.5 (42)	96.5 (52)
		CR*Q	91.5 (76)	92.3 (81)	90.5 (97)	93.4 (57)	93.9 (61)	92.8 (75)	94.0 (40)	94.6 (43)	94.0 (56)
		CRAsym	76.5 (51)	76.3 (53)	67.3 (57)	78.8 (39)	78.8 (42)	78.3 (48)	79.9 (28)	80.9 (29)	78.1 (36)
		CROLS	91.1 (75)	92.1 (81)	90.7 (101)	93.0 (54)	93.5 (58)	93.1 (72)	94.1 (38)	94.5 (41)	93.5 (52)
	0	CR*N	97.1 (47)	97.1 (50)	97.7 (65)	98.1 (33)	98.0 (37)	98.1 (47)	98.3 (25)	98.4 (27)	98.8 (34)
		CR*HDR	98.3 (40)	98.2 (40)	99.1 (52)	99.0 (25)	99.4 (28)	99.5 (36)	99.4 (17)	99.5 (19)	99.6 (24)
		CR*Q	99.1 (64)	98.9 (68)	99.4 (86)	99.7 (45)	99.7 (49)	99.7 (61)	99.9 (32)	99.7 (34)	99.9 (43)
		CROLS	91.3 (75)	91.2 (81)	91.7 (102)	92.9 (54)	92.5 (58)	92.3 (73)	94.1 (39)	94.7 (42)	94.2 (52)
		CR*SimHDR	85.3 (71)	85.7 (77)	75.1 (96)	94.1 (51)	94.4 (56)	90.8 (70)	96.3 (38)	95.2 (41)	95.2 (52)
		CR*Sim	84.1 (83)	84.1 (91)	73.9 (116)	93.4 (60)	93.5 (66)	90.1 (84)	96.6 (45)	95.9 (49)	95.8 (62)
		CR*SimOLS	85.2 (108)	86.9 (116)	87.5 (146)	90.9 (77)	90.7 (83)	91.1 (104)	92.7 (55)	92.5 (59)	92.6 (74)
20	1	CR*N	91.2 (80)	91.7 (87)	90.0 (112)	92.1 (55)	93.9 (60)	92.9 (78)	94.3 (39)	93.7 (43)	94.6 (54)
		CR*HDR	90.2 (76)	91.2 (83)	92.1 (104)	92.1 (54)	93.1 (59)	91.7 (75)	94.1 (38)	93.1 (42)	93.9 (53)
		CR*Q	90.1 (77)	90.7 (84)	89.6 (107)	92.1 (54)	93.3 (59)	91.9 (76)	94.4 (38)	92.6 (42)	93.8 (53)
		CRAsym	79.2 (58)	78.9 (62)	75.9 (76)	80.3 (40)	82.2 (43)	77.3 (53)	83.5 (28)	80.5 (30)	81.9 (37)
		CROLS	89.7 (76)	90.1 (83)	89.7 (104)	92.3 (54)	92.9 (58)	91.3 (74)	94.4 (38)	92.9 (42)	93.8 (53)
	0.5	CR*N	86.1 (81)	82.7 (86)	80.7 (103)	91.6 (59)	91.9 (64)	88.4 (81)	93.9 (41)	94.9 (45)	93.1 (58)
		CR*HDR	91.0 (74)	87.8 (79)	84.7 (94)	95.7 (55)	94.7 (59)	92.1 (73)	93.3 (39)	94.3 (43)	96.1 (54)
		CR*Q	89.1 (75)	88.8 (80)	88.6 (97)	92.0 (57)	92.1 (61)	90.9 (75)	93.3 (40)	94.1 (44)	92.5 (56)
		CRAsym	72.5 (50)	71.6 (51)	64.2 (57)	77.2 (39)	76.6 (41)	73.8 (48)	81.3 (28)	79.2 (30)	76.8 (36)
		CROLS	89.5 (76)	88.9 (82)	89.3 (104)	92.5 (54)	92.5 (59)	91.8 (74)	94.1 (38)	94.4 (42)	93.1 (53)
	0	CR*N	97.3 (57)	96.8 (61)	97.0 (79)	98.5 (40)	97.3 (45)	97.7 (58)	98.8 (29)	98.9 (33)	98.9 (41)
		CR*HDR	97.9 (53)	97.5 (55)	97.7 (70)	99.1 (35)	98.2 (39)	98.7 (50)	99.3 (24)	99.3 (27)	99.3 (33)
		CR*Q	98.9 (63)	98.7 (69)	98.8 (87)	99.6 (44)	99.2 (49)	99.3 (62)	99.6 (31)	99.7 (34)	99.7 (43)
		CROLS	90.1 (76)	90.2 (83)	89.8 (104)	92.8 (54)	91.9 (59)	91.9 (74)	93.5 (38)	93.5 (42)	93.7 (53)
		CR*SimHDR	87.1 (81)	83.6 (89)	78.1 (112)	95.1 (60)	94.4 (66)	93.7 (84)	97.1 (45)	97.1 (50)	97.3 (62)
		CR*Sim	86.1 (99)	82.8 (109)	78.3 (138)	94.3 (73)	94.4 (81)	94.1 (103)	97.8 (55)	98.3 (61)	97.1 (77)
		CR*SimOLS	77.7 (117)	76.3 (128)	76.6 (160)	86.1 (82)	85.4 (90)	86.9 (114)	90.5 (59)	89.8 (64)	90.3 (81)

NOTE: We multiply values by 100. The lengths of the simultaneous confidence regions are averaged over the number of parameters.

Table 3.

Empirical s.d. of the parameter estimates (σ̃) and average s.e. estimates (σ̂).

p	β0		n = 100			n = 200			n = 400
			ρ= 0	ρ = 0.2	ρ = 0.5	ρ = 0	ρ = 0.2	ρ = 0.5	ρ = 0	ρ = 0.2	ρ = 0.5
10	1	σ̃	21.7	22.2	29.9	14.7	15.2	19.3	10.1	10.8	13.7
		σ̃OLS	21.1	21.6	28.5	14.4	15.2	19.1	10.1	10.9	13.9
		σ̂*	20.0	21.7	28.1	14.1	15.2	19.4	10.0	10.8	13.5
		σ̂OLS	19.1	20.6	26.0	13.8	14.8	18.5	9.8	10.6	13.2
		σ̂Asym	14.7	15.4	18.7	10.2	10.8	13.1	7.1	7.5	9.2
	0.5	σ̃	24.2	25.5	32.3	16.1	16.8	21.8	10.6	11.2	14.7
		σ̃OLS	21.6	22.8	29.2	14.8	15.5	19.4	10.2	10.8	13.9
		σ̂*	21.1	22.8	27.9	15.1	16.2	20.5	10.3	11.2	14.5
		σ̂OLS	19.1	20.7	25.9	13.8	14.8	18.4	9.8	10.6	13.2
		σ̂Asym	13.0	13.6	14.5	10.0	10.6	12.1	7.1	7.5	9.1
	0	σ̃	17.0	17.2	20.3	10.7	11.5	14.2	6.9	7.2	9.2
		σ̃OLS	22.4	23.5	28.4	14.9	16.2	19.9	10.2	10.9	13.8
		σ̂*	18.6	19.9	25.1	13.2	14.2	17.8	9.4	10.1	12.6
		σ̂OLS	19.2	20.6	26.1	13.8	14.9	18.5	9.9	10.6	13.3
		σ̂Asym	5.0	4.5	5.4	2.6	2.9	3.5	1.5	1.5	2.0
20	1	σ̃	23.2	24.8	33.4	15.4	16.2	21.6	9.9	11.4	14.0
		σ̃OLS	23.0	24.6	31.7	15.4	16.1	21.3	9.8	11.4	13.9
		σ̂*	20.3	22.2	28.5	14.1	15.3	19.9	9.9	10.9	13.9
		σ̂OLS	19.3	21.1	26.5	13.7	14.9	18.9	9.7	10.7	13.4
		σ̂Asym	14.9	15.9	19.4	10.3	10.9	13.5	7.1	7.6	9.4
	0.5	σ̃	24.7	27.1	32.8	16.4	18.1	23.1	10.5	11.7	15.7
		σ̃OLS	22.8	25.3	31.6	15.1	16.6	20.8	10.0	11.2	14.4
		σ̂*	21.2	22.8	28.1	15.2	16.5	20.8	10.5	11.5	14.9
		σ̂OLS	19.3	21.0	26.4	13.7	14.9	18.8	9.8	10.7	13.5
		σ̂Asym	12.8	13.1	14.5	10.1	10.5	12.2	7.2	7.6	9.2
	0	σ̃	15.3	16.7	21.1	9.3	10.8	13.7	6.1	6.6	8.1
		σ̃OLS	22.5	25.0	31.7	14.7	16.6	21.2	10.2	11.4	14.1
		σ̂*	18.5	20.2	25.6	12.9	14.2	18.1	9.1	10.2	12.7
		σ̂Asym	4.7	5.1	6.1	2.6	2.8	3.5	1.4	1.6	1.9

NOTE: We present results for settings when σ, the standard deviation of ε, is 2. All values are multiplied by 100. Note that ${\widehat{\sigma }}_j^{\text{Asym}}=0$ when β̂_j = 0, but β̂_j and ${\widehat{\beta }}_j^{\ast }$ are not always 0 in the simulations, and therefore the average ${\widehat{\sigma }}_j^{\text{Asym}}$ is nonzero.

The coverage probabilities and lengths of our simultaneous confidence regions are also displayed in Tables 1 and 2. We compared our methods to CR^*SimOLS, constructed analogously to CR^*Sim except = {j|j = 1, …, p} and CR^*SimOLS is centered at the OLS estimates and the standard error is the sample standard deviation of the perturbed OLS estimates. Our regularized CR^*Sim and CR^*SimHDR have the advantage of shrinking the dimension of the region by reducing some CRs to the point {0} when is large. We see that our CR^*Sim and CR^*SimHDR outperform CR^*SimOLS in coverage and have shorter lengths. For large sample settings when n = 1000, CR^*SimOLS converges further to 95% coverage with levels around 90% for p = 20 and CR^*Sim and CR^*SimHDR have coverage almost always over 95%.

In Table 3 we also present the standard error estimates when σ = 2. For notation, let the empirical standard deviations of the estimators β̂_j and β̃_j be denoted as σ̃_j and ${\stackrel{\sim }{\sigma }}_j^{\text{OLS}}$, respectively. We see that our estimate of the standard error from the perturbed samples, ${\widehat{\sigma }}_j^{\ast }$, does well in estimating σ̃_j. However, the standard error proposed by Zou (2006) underestimates the true standard error of the parameter estimates, especially when σ = 2 and β_0j = 0.5 or 0. When the SNR is higher, ${\stackrel{\sim }{\sigma }}_j^{\text{Asym}}$ estimates σ̃_j well except when β_0j = 0 because ${\widehat{\sigma }}_j^{\text{Asym}}=0$ whereas σ̃_j and ${\widehat{\sigma }}_j^{\ast }$ are clearly nonzero.

4. EXAMPLE: HIV DRUG RESISTANCE

We illustrate our methods in a real example using the HIV antiretroviral drug susceptibility data described in Rhee et al. (2003). This dataset was refined from the Stanford HIV Drug Resistance Database (available at http://hivdb.stanford.edu/), and is used to study the association of protease mutations with susceptibility to the protease inhibitor anti-retroviral (ARV) drug amprenavir. The data consist of mutation information at 99 protease codons in the viral genome, of which 79 contain mutations, and ARV drug resistance assays for n = 702 HIV infected patients. Drug resistance was measured in units of IC₅₀, the amount of drug needed to inhibit viral replication by 50% in units of fold increase compared to drug-sensitive wildtype virus. Researchers are interested in determining which protease mutations are associated with ARV resistance so that they may develop a genotype test for resistance that looks for these mutations in the patient’s infecting HIV strain. Therefore, we aim to examine the effect of the presence of any of the mutations at 79 codons on IC₅₀, where higher IC₅₀ measurements indicate higher levels of drug resistance. We chose to log-transform the non-negative IC₅₀ outcome and represented the presence of each of the mutations as a binary predictor in our regression model. We removed the fifteen mutations that occurred less than 0.5% in the data set. Recently, Wu (2009) analyzed these data with a permutation test for regression coefficients of LASSO. In this paper, we will analyze the data using ALASSO and gain inference by using our perturbation methods to construct CRs and standard errors.

For this analysis, we used LARS to fit an ALASSO linear model with initial parameters β̃ estimated by OLS and λ and λ^* chosen to minimize the BIC as described in the appendix. We generated M=500 perturbation variable sets , consisting of n = 702 i.i.d. variables from an exponential distribution with mean and variance equal to 1, and for each we minimized the perturbed objective function to obtain ${\widehat{\mathit{\beta }}}_m^{\ast }$. We constructed 95% CRs using our perturbation method and compared inference gained from CR^*N, CR^*HDR, and CR^*Q to the inference from CR^Asym and CR^OLS. We estimated the σ used in the standard deviation estimate from Zou (2006) analogous to equation (2) with the residual variance from the nonregularized linear regression model and chose p̂_high and p̂_low as described in the simulation studies.

We present a graphical summary of the analysis results in Figure 1. Previous studies by Prado et al. (2002) and results collected by Johnson et al. (2005) found that mutations at codons 10, 32, 46, 47, 50, 54, 73, 82, 84 and 90 emerge in amprenavir resistant viral genomes. Using a permutation based p-value adjusted for multiple testing, Wu (2009) determined these mutations (except 73 and 82) as well as additional codon mutations to be significantly associated with amprenavir susceptibility at the α = 0.05 level for a total of thirteen significant associations. The ALASSO estimator obtained with λ = 0.56 from BIC estimated 36 coefficients as nonzero. The confidence region from nonregularized estimates CR^OLS was significant for twenty-six mutations. Our perturbation based CR^*N, CR^*HDR, and CR^*Q for the mutations found significant by Wu (2009) did not include zero and three new mutations (37, 64, 93) had significant perturbation confidence regions. We see in Figure 2 that the parameter for codons 71 and 89 have marginally significant Normal and HDR confidence regions and marginally nonsignificant quantile confidence regions and note that is marginally close to 0.05.

Figure 1.

Perturbation methods results denoting significant associations between genetic mutations and drug susceptibility.

Figure 2.

95% perturbation CRs (CR^*N and CR^*HDR) for the association between genetic mutations and antiretroviral drug susceptibility. Estimated coefficients β̂_j are represented with a circle on each CR line and a star at zero signifies that the CR includes the point mass at zero. The shaded region denotes the simultaneous confidence regions CR^*Sim and CR^*SimHDR. Note that even coefficients estimated as zero may have CRs around their estimates and that CR^*HDR may be asymmetrical and noncontiguous.

Our use of ALASSO provides estimates of the effects of each mutation while adjusting for the presence of other mutations. Several studies have shown that mutations associated with resistance to protease inhibitors can have varying effects when combined with other mutations (Schumi and DeGruttola 2008; Van Marck et al. 2009). For instance, the mutation at codon 32 has been found to have no effect on resistance of the protease inhibitor drug darunavir when a mutation at codon 84 is present (Van Marck et al. 2009). Our method allows us to determine the size of associations without orthogonalizing predictors and we adjust for multiple testing with the simultaneous confidence region CR^*Sim. Results could be impacted by studies summarized in Johnson et al. (2005) that may not have adjusted for other mutations, and the use of LASSO estimators that do not have oracle properties in Wu (2009). Our methods highlight three new mutations that have not been found to be associated with drug susceptibility. Furthermore, our methods produce CRs for the coefficients of mutations that were estimated as zero. These CRs quantify the uncertainty in our estimation and can aid scientists who wish to conduct future drug therapy studies involving the codons.

5. DISCUSSION

In this paper, we address the problem of constructing a covariance estimate for parameter estimates obtained with a general objective function and concave penalty functions including adaptive LASSO and SCAD. The proposed methods for covariance estimates are simple to implement and possess the attractive property that parameters estimated as zero have nonzero standard errors. We may then construct confidence regions for each parameter estimate and obtain more meaningful inference.

We have shown through extensive simulation studies using the ALASSO penalty that our perturbation method results in confidence regions with accurate coverage probability. The perturbation based normal CR does not sacrifice much in length and has reasonable coverage for small sample sizes. We set the CR to {0} when the proportion of perturbed estimates set to 0 is higher than a threshold, and therefore shorten the length by utilizing the oracle property. The perturbation based highest density region has even shorter length and good coverage probability, especially for the moderate signal β_0j = 0.5 in comparison to all other confidence regions. The asymptotic based Normal interval that uses the standard error estimate presented in Zou (2006) fails to reach nominal coverage levels due to the underestimation of the standard error, most notably when the standard error is estimated as 0 when β̂ = 0. However, our estimate of the standard error of the parameter estimates based on our perturbation samples is close to the empirical standard error of the ALASSO estimates, even for parameters estimated as 0. Additionally, we propose two types of simultaneous CRs that adjust for multiple comparisons. We again utilize the oracle property and reduce the dimension of our region by setting intervals to {0} when the proportion of zero perturbed parameter estimates is high. Therefore, the average length of our Normal simultaneous region will often be shorter than the simultaneous OLS region. For instance, when all covariates are independent, the OLS length is asymptotically proportional to ${\gamma }_{\text{OLS}}=max{\left\{\left|({\stackrel{\sim }{\beta }}_j-{\beta }_{0j})/\sigma \right|\right\}}_{j=1}^p$ whereas the perturbation region length is asymptotically proportional to (q/p)γ where γ = max {|(β̂_j − β_0j)/σ|}_{β0j ≠ 0}. Note that γ ≤ γ_OLS and so the length of the perturbation region will be shorter than the OLS length when the true model is sparse. Similarly, when the covariates are not independent, ${\left\{({\stackrel{\sim }{\beta }}_j-{\beta }_{0j})/\sigma \right\}}_{j=1}^p~\mathcal{N}(\mathbf{0},\text{Corr}(\widehat{\mathit{\beta }}))$ and the perturbation region generally has shorter average length than the OLS region. Simple simulations show that when q parameters are estimated as nonzero, we expect the perturbation region length to be approximately 0.36 times the OLS region length when p = 10 and q = 4 and approximately 0.16 times the OLS region length when p = 20 and q = 4 for both the independent case and the compound symmetry case when ρ = 0.5 and σ = 1. However, in finite sample, the gain in interval length for the shrinkage estimators may be substantially less than the theoretical gain as oracle properties may be far from being true and the intervals may need to be enlarged to ensure proper coverage levels.

When the SNR is low, much larger sample sizes may be required for the resampling procedure to yield confidence intervals with proper coverage levels. We conducted further simulations for the case when β = (1, 1, 0.1, 0.1, 0_1×(p−4))^⊤. In general, we find that the standard error estimates perform well even with sample sizes around 100. The confidence intervals have reasonable coverage levels for β₃ when σ = 1 and the correlation ρ is low with sample size 400 or larger. For example, when σ = 1, ρ = 0.2, and p = 20, the coverage level of the 95% CR^*HDR of β₃ is about 90% for n = 400 and 94% for n = 1000. As we increase the correlation ρ and σ, the interval estimation of β₃ becomes more difficult. For example, for the most difficult case with σ = 2, ρ = 0.5, and p = 20, the empirical coverage level of CR^*HDR is about 60%, 84% and 90% when n = 400, 1000, and 2000, respectively. This is a particularly difficult case as it has been shown that estimating the distribution function of the ALASSO type estimator is not feasible when the effect size is of similar magnitude to $n^{-\frac{1}{2}}$ (Pötscher and Schneider 2009). Note that when σ = 2, the effect size corresponding to β₃ is 0.05 whereas $n^{-\frac{1}{2}}=0.1$ when n is 100 and $n^{-\frac{1}{2}}\approx 0.032$ for n = 1000.

Additionally, it is well known that regularized estimators, while possessing asymptotic oracle properties, are prone to bias in finite samples. Bias correction for the ALASSO estimator can be achieved based on our perturbation samples. We present the technical details of the estimation of the bias in the appendix. We find that this bias correction works well in practice, especially when the signal is small or moderate, as when β_0j = 0.5. For example, in our simulations when p = 20, n = 200, ρ = 0.2, σ = 2, and β_0j = 0.5, the bias of β̂_j is −0.067 while the bias of ${\widehat{\beta }}_{0j}^{\text{BC}}$ is −0.034. Similar gains are seen for most settings. The bias corrected estimator has empirical standard error similar to that of the original ALASSO estimator but with smaller bias. We could construct analogous bias-corrected estimators based on other penalties and objective functions. The model size with the ALASSO and bias-corrected ALASSO estimator in our simulations is close to 5 when σ = 1, except for the difficult cases when n = 100 and p = 20 for which the average model size is closer to 5.5. For the settings where the SNR is low with σ = 2, the oracle property is weak in finite samples and so the model size is between 5 and 6 when p = 10 and between 6 and 9 when p = 20.

We note that when p is large relative to n, initial parameter estimates obtained with ridge regression can produce more stable results. Furthermore, our results may be extended to the case where p tends to ∞ at some rate slower than n. We expect that the theory could be derived using similar arguments as given in Fan and Peng (2004) and Zou and Zhang (2009). Lastly, we note that our methods are robust to misspecification of the model and are valid provided that regularity conditions given in Section 2 hold.

APPENDIX

A.1 Justification for the resampling method

To show that the distribution of $n^{\frac{1}{2}}(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)$ can be estimated by that of $n^{\frac{1}{2}}({\widehat{\mathit{\theta }}}^{\ast }-\widehat{\mathit{\theta }})\mid \mathcal{X}$ under conditions C1–C3, we first consider the distribution of $n^{\frac{1}{2}}({\widehat{\mathit{\theta }}}^{\ast }-{\mathit{\theta }}_0)$ under the product probability measure ℙ^* generated by the data, , and the perturbation variables = {G_i, i = 1, …, n}. Throughout, we assume that the parameter space for θ, denoted by Ω, is a compact set and θ₀ is an interior point of Ω. Note that this compactness condition may be nontrivial in practice. This condition is necessary for this proof of our proposed method, and validity of the method without this condition warrants further investigation. We let ℙ_n denote the empirical measure generated by and ${\mathbb{G}}_n=n^{-\frac{1}{2}}({\mathbb{P}}_n-\mathbb{P})$. We use notation →_p to denote convergence in probability.

We first show that θ̃^* →_p θ₀, where θ̃^* is the perturbed initial parameter estimate obtained by minimizing the perturbed un-regularized likelihood in (3). For f ∈ { (θ; )}, denote the empirical perturbation measure as ${\mathbb{P}}_n^{\ast }f=n^{-1}{\sum }_{i=1}^n{G}_{i}f(X_i)$. Since { (θ; ) : θ ∈ Ω} is ℙ-Glivenko-Cantelli, by Corollary 10.14 of Kosorok (2008) $\mid {\stackrel{\sim }{\mathcal{L}}}^{\ast }(\mathit{\theta })-\mathbb{P}\{\mathcal{L}(\mathit{\theta }:\mathcal{D})\}\mid \le \mid {\stackrel{\sim }{\mathcal{L}}}^{\ast }(\mathit{\theta })-\stackrel{\sim }{\mathcal{L}}(\mathit{\theta })\mid +\mid \stackrel{\sim }{\mathcal{L}}(\mathit{\theta })-\mathbb{P}\{\stackrel{\sim }{\mathcal{L}}(\mathit{\theta };\mathcal{D})\}\mid =\mid ({\mathbb{P}}_n^{\ast }-{\mathbb{P}}_n)\{\mathcal{L}(\mathit{\theta };\mathcal{D})\}\mid +\mid ({\mathbb{P}}_n-\mathbb{P})\{\stackrel{\sim }{\mathcal{L}}(\mathit{\theta };\mathcal{D})\}\mid$ uniformly converges to zero. Then, under condition C1, ℙ{ (θ; )} has a unique minimum at θ₀, and so θ̃^* →_p θ₀ (Newey and McFadden 1994; Theorem 2.1).

We now show that θ̂^* →_p θ₀. First note that ${\sum }_{j=1}^p{p}_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )\mid {\beta }_j\mid \to 0$ in probability. When the penalty is $L_q,p_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )={\lambda }_n{\mid {\beta }_j\mid }^q,p^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid ){\to }_p{p}^{\prime }(\mid {\beta }_{0j}\mid )$ by the continuous mapping theorem and λ_n → 0. For the SCAD penalty, $p_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )={\lambda }_{n}I(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid \le {\lambda }_n)+{(a{\lambda }_n-\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )}_{+}I(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid >{\lambda }_n)/(a-1)$. We consider two cases: (i) β_0j ≠ 0, and (ii) β_0j = 0. For case (i), λ_n → 0 and $\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid {\to }_p\mid {\beta }_{0j}\mid$. Thus, $I(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid \le {\lambda }_n){\to }_{p}0$ and ${(a{\lambda }_n-\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )}_{+}{\to }_{p}0$. For case (ii), λ_n → 0 and ${(a{\lambda }_n-\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )}_{+}{\to }_{p}0$. Finally, for the ALASSO penalty, $p_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )={\lambda }_n{\mid n^{\frac{1}{2}}{\stackrel{\sim }{\beta }}_j^{\ast }\mid }^{-1},\mid n^{\frac{1}{2}}{\stackrel{\sim }{\beta }}_j^{\ast }\mid =O_{\mathbb{P}}(1)$, and λ_n → 0. Then, since θ lies in a compact space, ${\sum }_{j=1}^p{p}_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )\mid {\beta }_j\mid \le \tau {\sum }_{j=1}^p{p}_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )\le \left|\right|\mathit{\beta }\left|\right|B_n$, where τ = max{|β_j|}, B_n = o_ℙ (1) since $p_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid ){\to }_{\mathbb{P}}0$ for each j, and hence ${sup}_{\mathit{\theta }}\left|{\sum }_{j=1}^p{p}_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )\mid {\beta }_j\mid \right|{\to }_{p}0$ (Newey and McFadden 1994; Lemma 2.9). Now, with similar arguments as above for the proof of θ̃^* →_p θ₀, we have that $\mid {\widehat{\mathcal{L}}}^{\ast }(\mathit{\theta })=\mathbb{P}\{\mathcal{L}(\mathit{\theta };\mathcal{D})\}\mid \le \mid {\stackrel{\sim }{\mathcal{L}}}^{\ast }(\mathit{\theta })-\mathbb{P}\{\mathcal{L}(\mathit{\theta };\mathcal{D})\}\mid +{\sum }_{j=1}^p{p}_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )\mid {\beta }_j\mid$ uniformly converges to zero. This implies the convergence of θ̂^* →_p θ₀.

We next show that $\left|\right|{\widehat{\mathit{\theta }}}^{\ast }-{\mathit{\theta }}_0\left|\right|=O_{{\mathbb{P}}^{\ast }}(n^{-\frac{1}{2}})$. It is sufficient to show that for any ε > 0, there exits C > 0 such that

$${\mathbb{P}}^{\ast }\left(\underset{\left|\right|\mathit{\theta }-{\mathit{\theta }}_{\text{o}}\left|\right|\ge {Cn}^{-\frac{1}{2}}}{inf}{\widehat{\mathcal{L}}}^{\ast }(\mathit{\theta })>{\widehat{\mathcal{L}}}^{\ast }({\mathit{\theta }}_0)\right)>1-\epsilon$$ (A.1)

Consider $\mathit{\theta }={\mathit{\theta }}_0+n^{-\frac{1}{2}}\mathbf{u}$. Condition C3(c) implies

$$\frac{{\mathbb{P}}_n\{\mathcal{L}({\mathit{\theta }}_0+n^{-\frac{1}{2}}\mathbf{u})-\mathcal{L}({\mathit{\theta }}_0)-n^{-\frac{1}{2}}\mathcal{U}{({\mathit{\theta }}_0;\mathcal{D})}^{\top }\mathbf{u}\}-\frac{1}{2}{n}^{-1}{\mathbf{u}}^{\top }\mathbb{A}\mathbf{u}}{\left|\right|n^{-\frac{1}{2}}\mathbf{u}\left|\right|}=o_{\mathbb{P}}(1)$$

uniformly in u. By the multiplier central limit theorem (Kosorok 2008; Theorem 10.1),

$$\frac{{\mathbb{P}}_n^{\ast }\{\mathcal{L}({\mathit{\theta }}_0+n^{-\frac{1}{2}}\mathbf{u})G-\stackrel{\sim }{\mathcal{L}}({\mathit{\theta }}_0)G-n^{-\frac{1}{2}}\mathcal{U}{({\mathit{\theta }}_0;\mathcal{D})}^{\top }\mathbf{u}G\}-\frac{1}{2}{n}^{-1}{\mathbf{u}}^{\top }\mathbb{A}\mathbf{u}}{\left|\right|n^{-\frac{1}{2}}\mathbf{u}\left|\right|}=o_{{\mathbb{P}}^{\ast }}(1)$$

uniformly in u. It follows that uniformly for $\mathit{\theta }\in \{\mathit{\theta }:||\mathit{\theta }-{\mathit{\theta }}_0||\le n^{-\frac{1}{2}}\mathbf{u}\}$,

$${\stackrel{\sim }{\mathcal{L}}}^{\ast }({\mathit{\theta }}_0+n^{-\frac{1}{2}}\mathbf{u})-{\stackrel{\sim }{\mathcal{L}}}^{\ast }({\mathit{\theta }}_0)=n^{-\frac{1}{2}}{\mathbb{P}}_n\{\mathcal{U}({\mathit{\theta }}_0;\mathcal{D})G\}\mathbf{u}+\frac{1}{2}{n}^{-1}{\mathbf{u}}^{\top }\mathbb{A}\mathbf{u}+o_{{\mathbb{P}}^{\ast }}(n^{-1}|\left|u\right||)$$ (A.2)

and thus we may approximate $n\{{\widehat{\mathcal{L}}}^{\ast }({\mathit{\theta }}_0+n^{-\frac{1}{2}}\mathbf{u})-{\widehat{\mathcal{L}}}^{\ast }({\mathit{\theta }}_0)\}$ with ${\mathbb{G}}_n\{\mathcal{U}({\mathit{\theta }}_0;\mathcal{D})G\}\mathbf{u}+\frac{1}{2}{\mathbf{u}}^{\top }\mathbb{A}\mathbf{u}+n{\sum }_{j=1}^p{p}_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )\left(\left|{\beta }_{0j}+n^{-\frac{1}{2}}{u}_j\right|-\mid {\beta }_{0j}\mid \right)+o_{{\mathbb{P}}^{\ast }}({\left|\right|\mathbf{u}\left|\right|}^2+|\left|\mathbf{u}\right||)$.

Now we show the “consistency” of variable selection, i.e., ${\mathbb{P}}^{\ast }({\widehat{\mathit{\theta }}}_{{\mathcal{A}}^c}^{\ast }=0)\to 1$ as n → ∞. It suffices to to show that for any constant C and given θ̃ such that $\left|\right|{\stackrel{\sim }{\mathit{\theta }}}_{\mathcal{A}}-{\mathit{\theta }}_{0\mathcal{A}}\left|\right|=O_{{\mathbb{P}}^{\ast }}(n^{-\frac{1}{2}})$

$${\mathbb{P}}^{\ast }\left[{\text{argmin}}_{\left|\right|{\mathit{\theta }}_{{\mathcal{A}}^c}\left|\right|\le {Cn}^{-\frac{1}{2}}}{\widehat{\mathcal{L}}}^{\ast }\left\{{\left({\stackrel{\sim }{\mathit{\theta }}}_{\mathcal{A}}^{\top },{\theta }_{{\mathcal{A}}^c}^{\top }\right)}^{\top }\right\}=0\right]\to 1.$$ (A.3)

Let ũ and u denote $n^{\frac{1}{2}}({\stackrel{\sim }{\mathit{\theta }}}_{\mathcal{A}}-{\mathit{\theta }}_{0\mathcal{A}})$ and $n^{\frac{1}{2}}{\mathit{\theta }}_{{\mathcal{A}}^c}$, respectively. It follows from (A.2)

$$\begin{array}{l}n\left[{\widehat{\mathcal{L}}}^{\ast }\left\{{\left({\mathit{\theta }}_{0\mathcal{A}}^{\top }+n^{-\frac{1}{2}}{\stackrel{\sim }{\mathbf{u}}}_{\mathcal{A}}^{\top },n^{-\frac{1}{2}}{\mathbf{u}}_{{\mathcal{A}}^c}^{\top }\right)}^{\top }\right\}-{\widehat{\mathcal{L}}}^{\ast }\left\{{\left({\mathit{\theta }}_{0\mathcal{A}}^{\top }+n^{-\frac{1}{2}}{\stackrel{\sim }{\mathbf{u}}}_{\mathcal{A}}^{\top },0^{\top }\right)}^{\top }\right\}\right]\\ =[{\mathbb{G}}_n\{\mathcal{U}{({\mathit{\theta }}_0;\mathcal{D})}_{{\mathcal{A}}^c}^{\top }G\}+{\stackrel{\sim }{\mathbf{u}}}_{\mathcal{A}}^{\top }{\mathbb{A}}_{12}]{\mathbf{u}}_{{\mathcal{A}}^c}+\frac{1}{2}{\mathbf{u}}_{{\mathcal{A}}^c}^{\top }{\mathbb{A}}_{22}{\mathbf{u}}_{{\mathcal{A}}^c}+n\sum _{j\in {\mathcal{A}}^c}{p}_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )\mid n^{-\frac{1}{2}}{u}_j\mid \\ +o_{{\mathbb{P}}^{\ast }}({\left|\right|{\mathbf{u}}_{{\mathcal{A}}^c}\left|\right|}^2+|\left|u_{{\mathcal{A}}^c}\right||)=\sum _{j\in {\mathcal{A}}^c}{n}^{\frac{1}{2}}{p}_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )\mid u_j\mid +R_n({\mathbf{u}}_{{\mathcal{A}}^c}).\end{array}$$

where sup_{||u|| ≤ C} R_n(u)/(||u||² + ||u||) = o_ℙ^* (1). Zou and Li (2008) consider the limiting behavior of $n^{\frac{1}{2}}{p}_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )$ for SCAD and L_q penalties in their proof of the oracle properties of the one-step LLA estimator. They show that for both cases, when j ∈ , $n^{\frac{1}{2}}{p}_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid ){\to }_p\infty$. Additionally, for the ALASSO penalty, $n^{\frac{1}{2}}{p}_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )=n^{-\frac{1}{2}}{\lambda }_n{\mid n^{\frac{1}{2}}{\stackrel{\sim }{\beta }}_j^{\ast }\mid }^{-1}$, when j ∈ , we have $n^{-\frac{1}{2}}{\lambda }_n\to \infty$ and $\mid n^{\frac{1}{2}}{\stackrel{\sim }{\beta }}_j^{\ast }\mid =O_{{\mathbb{P}}^{\ast }}(1)$. Hence, for all three types of penalties, $n^{\frac{1}{2}}{p}_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid ){\to }_p\infty$. Thus, for any ε > 0, there exist C₁ > C₀ > 0 and N₀ such that ${\mathbb{P}}^{\ast }\left\{{\sum }_{j\in {\mathcal{A}}^c}{n}^{\frac{1}{2}}{p}_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )\mid u_j\mid \ge C_1{\sum }_{j\in {\mathcal{A}}^c}\mid u_j\mid \right\}\ge 1-\epsilon$ and ℙ^*{C₀ Σ_{j ∈} |u_j| ≥ |R_n(u)|} ≥ 1 − ε for ||u|| ≤ C and n ≥ N₀. This implies that with probability greater than 1 − 2ε, $n\left[{\widehat{\mathcal{L}}}^{\ast }\left\{{\left({\stackrel{\sim }{\mathit{\theta }}}_{\mathcal{A}}^{\top },n^{-\frac{1}{2}}{\mathbf{u}}_{{\mathcal{A}}^c}^{\top }\right)}^{\top }\right\}-{\widehat{\mathcal{L}}}^{\ast }\left\{{\left({\stackrel{\sim }{\mathit{\theta }}}_{\mathcal{A}}^{\top },0^{\top }\right)}^{\top }\right\}\right]\ge (C_1-C_0){\sum }_{j\in {\mathcal{A}}^c}\mid u_j\mid \ge 0$, which implies (A.3).

Lastly, we will justify the oracle property of ${\widehat{\mathit{\theta }}}_{\mathcal{A}}^{\ast }$. Since ${\mathbb{P}}^{\ast }({\widehat{\mathit{\theta }}}_{{\mathcal{A}}^c}^{\ast }=0)\to 1,{\widehat{\mathit{\theta }}}_{\mathcal{A}}^{\ast }$ can be considered as the minimizer of ${\widehat{\mathcal{L}}}_{\mathcal{A}}^{\ast }({\mathit{\theta }}_{\mathcal{A}})={\widehat{\mathcal{L}}}^{\ast }\{{({\mathit{\theta }}_{\mathcal{A}}^{\top },0^{\top })}^{\top }\}$. Following the approach of Zou (2006), we consider the reparametrization

$${\widehat{\mathcal{L}}}_{\mathcal{A}}^{\ast }({\mathit{\theta }}_{0\mathcal{A}}+n^{-\frac{1}{2}}{\mathbf{u}}_{\mathcal{A}})={\mathbb{P}}_n\mathcal{L}\left\{{\left({\mathit{\theta }}_{0\mathcal{A}}^{\top }+n^{-\frac{1}{2}}{\mathbf{u}}_{\mathcal{A}}^{\top },0^{\top }\right)}^{\top },{\mathcal{D}}_i\right\}{G}_i+\sum _{j\in \mathcal{A}}{p}_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )\left|{\beta }_{0j}+n^{-\frac{1}{2}}{u}_j\right|.$$ (A.4)

Let ${\widehat{\mathbf{u}}}_{\mathcal{A}}^{(n)}=arg{min}_{{\mathbf{u}}_{\mathcal{A}}}{\widehat{\mathcal{L}}}_{\mathcal{A}}^{\ast }({\mathit{\theta }}_{0\mathcal{A}}+n^{-\frac{1}{2}}{\mathbf{u}}_{\mathcal{A}})$. Note ${\widehat{\mathbf{u}}}_{\mathcal{A}}^{(n)}=n^{\frac{1}{2}}({\widehat{\mathit{\theta }}}_{\mathcal{A}}^{\ast }-{\mathit{\theta }}_{0\mathcal{A}})$ is also the minimizer of $V_n^{\ast }({\mathbf{u}}_{\mathcal{A}})\equiv {\widehat{\mathcal{L}}}_{\mathcal{A}}^{\ast }({\mathit{\theta }}_{0\mathcal{A}}+n^{-\frac{1}{2}}{\mathbf{u}}_{\mathcal{A}})-{\widehat{L}}^{\ast }({\mathit{\theta }}_0)$, as (θ₀) is a constant. Again, it follows from (A.2)

$$\begin{array}{l}{V}_n^{\ast }({\mathbf{u}}_{\mathcal{A}})=n^{\frac{1}{2}}{\mathbf{u}}_{\mathcal{A}}^{\top }{\mathbb{P}}_n\{{\mathcal{U}}_{\mathcal{A}}({\mathit{\theta }}_0,\mathcal{D})G\}+\frac{1}{2}{\mathbf{u}}_{\mathcal{A}}^{\top }{\mathbb{A}}_{11}{\mathbf{u}}_{\mathcal{A}}\\ +n\sum _{j\in \mathcal{A}}{p}_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )\left(\left|{\beta }_{0j}+n^{\frac{1}{2}}{u}_j\right|-\mid {\beta }_{0j}\mid \right)+o_{{\mathbb{P}}^{\ast }}({\left|\right|{\mathbf{u}}_{\mathcal{A}}\left|\right|}^2+|\left|{\mathbf{u}}_{\mathcal{A}}\right||)\end{array}$$

To examine the limiting behavior of the third term of $V_n^{\ast }(\mathbf{u})$, we have β_0j ≠ 0, $n^{\frac{1}{2}}(\mid {\beta }_{j0}+n^{-\frac{1}{2}}{u}_j\mid -\mid {\beta }_{j0}\mid ){\to }_p{u}_{j}sgn({\beta }_{0j})$, since j ∈ . Also, as Zou and Li (2008) proved in their appendix, $n^{\frac{1}{2}}{p}_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid ){\to }_{p}0$ when j ∈ for the SCAD and L_q penalties. For the ALASSO penalty, $n^{\frac{1}{2}}{p}_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )={\lambda }_n{\mid {\stackrel{\sim }{\beta }}_j\mid }^{-1}$, λ_n → 0, and |β̃_j|⁻¹ →_p |β_0j|⁻¹ for β_0j ≠ 0. Therefore, by Slutsky’s theorem, we have ${np}_{{\lambda }_n^{\ast }j}^{\prime }(\mid {\stackrel{\sim }{\beta }}_j^{\ast }\mid )\left(\left|{\beta }_{0j}+n^{\frac{1}{2}}{u}_j\right|-\mid {\beta }_{0j}\mid \right)=o_{{\mathbb{P}}^{\ast }}(1)$ and

$$V_n^{\ast }({\mathbf{u}}_{\mathcal{A}})={\mathbf{u}}_{\mathcal{A}}^{\top }{\mathbb{G}}_n\{{\mathcal{U}}_{\mathcal{A}}({\mathit{\theta }}_0,\mathcal{D})G\}+\frac{1}{2}{\mathbf{u}}_{\mathcal{A}}^{\top }{\mathbb{A}}_{11}{\mathbf{u}}_{\mathcal{A}}+o_{{\mathbb{P}}^{\ast }}(1+{\left|\right|{\mathbf{u}}_{\mathcal{A}}\left|\right|}^2+|\left|{\mathbf{u}}_{\mathcal{A}}\right||).$$

Thus, ${\widehat{\mathbf{u}}}_{\mathcal{A}}^{(n)}=-{\mathbb{A}}_{11}^{-1}{\mathbb{G}}_n\{{\mathcal{U}}_{\mathcal{A}}({\mathit{\theta }}_0,\mathcal{D})G\}+o_{{\mathbb{P}}^{\ast }}(1)$. Since { (θ₀, )G} converges to N(0, ) in distribution, $n^{\frac{1}{2}}({\widehat{\mathit{\theta }}}_{\mathcal{A}}^{\ast }-{\mathit{\theta }}_{0,\mathcal{A}}){\to }_{d}N(\mathbf{0},{\mathbb{A}}_{11}^{-1}{\mathbb{B}}_{11}{\mathbb{A}}_{11}^{-1})$ and ${\mathbb{P}}^{\ast }({\widehat{\mathit{\theta }}}_{{\mathcal{A}}^c}^{\ast }=0)\to 1$. Then the perturbed regularized estimator θ̂^* is asymptotically normal in the true nonzero parameter set.

Similar arguments as given above, along with the conditional multiplier central limit theorem (Kosorok 2008; Theorem 10.4), can be used to justify that the distribution of $n^{\frac{1}{2}}({\widehat{\mathit{\theta }}}^{\ast }-\widehat{\mathit{\theta }})\mid \mathcal{X}$ approximates that of $n^{\frac{1}{2}}(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)$. Specifically, we can similarly obtain $n^{\frac{1}{2}}({\widehat{\mathit{\theta }}}_{\mathcal{A}}-{\mathit{\theta }}_{0\mathcal{A}})=-{\mathbb{A}}_{11}^{-1}{\mathbb{G}}_n\{{\mathcal{U}}_{\mathcal{A}}({\mathit{\theta }}_0,\mathcal{D})\}+o_{\mathbb{P}}(1)$ and ℙ^*(θ̂= 0) → 1. Therefore, $n^{\frac{1}{2}}({\widehat{\mathit{\theta }}}_{\mathcal{A}}^{\ast }-{\widehat{\mathit{\theta }}}_{\mathcal{A}})=-{\mathbb{A}}_{11}^{-1}{\mathbb{G}}_n\{{\mathcal{U}}_{\mathcal{A}}({\mathit{\theta }}_0,\mathcal{D})(G-1)\}+o_{{\mathbb{P}}^{\ast }}(1)$. Since $-{\mathbb{A}}_{11}^{-1}{\mathbb{G}}_n\{{\mathcal{U}}_{\mathcal{A}}({\mathit{\theta }}_0,\mathcal{D})(G-1)\}\mid \mathcal{X}{\to }_{d}N(\mathbf{0},{\mathbb{A}}_{11}^{-1}{\widehat{\mathbb{B}}}_{11}{\mathbb{A}}_{11}^{-1})$ and →_ℙ , $n^{\frac{1}{2}}({\widehat{\mathit{\theta }}}_{\mathcal{A}}^{\ast }-{\widehat{\mathit{\theta }}}_{\mathcal{A}})\mid \mathcal{X}$ and $n^{\frac{1}{2}}({\widehat{\mathit{\theta }}}_{\mathcal{A}}-{\mathit{\theta }}_{0\mathcal{A}})$ converge in distribution to the same limit. Furthermore, ${\mathbb{P}}^{\ast }({\widehat{\mathit{\theta }}}_{{\mathcal{A}}^C}^{\ast }=0\mid \mathcal{X})\to 1$.

A.2 Choice of thresholding values p̂_high and p̂_low for confidence regions

We choose values for p̂_high and p̂_low to converge at rates relative to the order of the tuning parameter λ and bounded by the probability that our perturbation samples are set to zero. For illustration, consider the univariate β case with one predictor under orthonormal design. Consider standardized parameters γ̂ = β̂/σ, γ = β/σ and λ̃_n = λ_n/σ², where λ_n → 0 and $n^{\frac{1}{2}}{\lambda }_n\to \infty$. Then

$$\widehat{\gamma }\stackrel{.}{\sim }N({\gamma }_0,n^{-1}),{\widehat{\gamma }}_1=\widehat{\gamma }{\left(1-\frac{{\stackrel{\sim }{\lambda }}_n}{{\mid \widehat{\gamma }\mid }^2}\right)}_{+},{\gamma }_1^{\ast }\stackrel{.}{\sim }{\gamma }^{\ast }{\left(1-\frac{{\stackrel{\sim }{\lambda }}_n}{{\mid {\gamma }^{\ast }\mid }^2}\right)}_{+}$$

where γ^* ∼̇ N(γ̂, 1/n). Thus ${\gamma }_1^{\ast }=0$ with probability

$${\widehat{P}}_0=P\{\mid {\gamma }^{\ast }\mid <{\stackrel{\sim }{\lambda }}_n^{1/2}\}=\mathrm{\Phi }\{n^{1/2}({\stackrel{\sim }{\lambda }}_n^{1/2}-\widehat{\gamma })\}-\mathrm{\Phi }\{-n^{1/2}({\stackrel{\sim }{\lambda }}_n^{1/2}+\widehat{\gamma })\}$$

First consider non-zero parameters. Without loss of generality, assume that γ₀ > 0. Let $\epsilon =2{\stackrel{\sim }{\lambda }}_n^{\frac{1}{2}}$ and assume that γ₀ > 2ε. Then

$$\begin{array}{l}E({\widehat{P}}_0)=E\left[\mathrm{\Phi }\left\{n^{\frac{1}{2}}({\stackrel{\sim }{\lambda }}_n^{\frac{1}{2}}-\widehat{\gamma })\right\}-\mathrm{\Phi }\left\{n^{\frac{1}{2}}(-{\stackrel{\sim }{\lambda }}_n^{\frac{1}{2}}-\widehat{\gamma })\right\}\right]\\ \le \left[\mathrm{\Phi }\left\{n^{\frac{1}{2}}({\stackrel{\sim }{\lambda }}_n^{\frac{1}{2}}-\epsilon )\right\}-\mathrm{\Phi }\left\{n^{\frac{1}{2}}(-{\stackrel{\sim }{\lambda }}_n^{\frac{1}{2}}-\epsilon )\right\}\right]P(\widehat{\gamma }>\epsilon )+P(\widehat{\gamma }\le \epsilon )\\ \le 2\mathrm{\Phi }(-n^{\frac{1}{2}}\epsilon )⪷\sqrt{2/\pi }exp\{-n{\stackrel{\sim }{\lambda }}_n\}=\sqrt{2/\pi }exp\{-n{\lambda }_n/{\sigma }^2\}\end{array}$$

Thus, we propose to choose the lower bound ${\widehat{p}}_{\mathit{low}}=min(0.49,\sqrt{2/\pi }exp\{-n{\lambda }_n/(4{\sigma }^2)\})$ such that ${\widehat{p}}_{\mathit{low}}\gg \sqrt{2/\pi }exp(-n{\lambda }_n/{\sigma }^2)$. On the other hand, if γ₀ = 0, then

$$\begin{array}{l}E(1-{\widehat{P}}_0)=E\left[\mathrm{\Phi }\left\{n^{\frac{1}{2}}(-{\stackrel{\sim }{\lambda }}_n^{\frac{1}{2}}+\widehat{\gamma })\right\}+\mathrm{\Phi }\left\{n^{\frac{1}{2}}(-{\stackrel{\sim }{\lambda }}_n^{\frac{1}{2}}-\widehat{\gamma })\right\}\right]\\ \approx 2\mathrm{\Phi }\left\{-n^{\frac{1}{2}}{\stackrel{\sim }{\lambda }}_n^{\frac{1}{2}}/\sqrt{2}\right\}⪷\sqrt{2/\pi }exp\{-n{\stackrel{\sim }{\lambda }}_n/4\}=\sqrt{2/\pi }exp\{-n{\lambda }_n/(4{\sigma }^2)\}\end{array}$$

Thus, we chose ${\widehat{p}}_{\mathit{high}}^{\ast }=1-\sqrt{2/\pi }exp(-n{\lambda }_n/{\sigma }^2)$ such that ${\widehat{p}}_{\mathit{high}}^{\ast }\gg 1-\sqrt{2/\pi }exp\{-n{\lambda }_n/(4{\sigma }^2)\}$. Note that we chose p̂_low and p̂_high such that p̂_low goes to 0 at a much slower rate than P̂₀ for γ₀ ≠ 0. On the other hand, when γ₀ = 0, ${\widehat{p}}_{\mathit{high}}^{\ast }$ goes to 1 at a much faster rate than P̂₀ and thus ${\widehat{P}}_0>{\widehat{p}}_{\mathit{high}}=min(1-\alpha ,{\widehat{p}}_{\mathit{high}}^{\ast })$ occurs with probability approaching 1 as n → ∞, for any fixed α > 0. Consequently, this indicates a strong evidence of γ = 0 when P̂₀ > p̂_high. When σ is unknown, it is replaced with a consistent estimate σ̂.

A.3 Justification of highest density region and bias estimate

For j ∈ , ${\mathbb{P}}^{\ast }({\widehat{\beta }}_j^{\ast }=0)\to 1$ and thus for any α > 0, ℙ^*( > α) → 1, and ℙ( < p̂_high) + ℙ( < p̂_low) → 0. Hence, ${\mathbb{P}}^{\ast }(0\in {\text{CR}}_j^{\ast \text{HDR}})\to 1$ and so we include {0} in our CR when > p̂_low and the coverage of ${\text{CR}}_j^{\ast \text{HDR}}$ converges to 1 when β_0j = 1. For j ∈ , →_p 0, and our estimates converge to a continuous distribution, specifically $n^{\frac{1}{2}}({\widehat{\beta }}_j^{\ast }-{\widehat{\beta }}_j)\mid \mathcal{X}{\to }_{d}N(0,{\sigma }_j^2)$, where ${\sigma }_j^2$ is the asymptotic variance of $n^{\frac{1}{2}}({\widehat{\beta }}_j-{\beta }_{0j})$. It follows that ${sup}_x\mid n^{-\frac{1}{2}}{f}_j^{\ast }({\widehat{\beta }}_{0j}+n^{\frac{1}{2}}x)-{\phi }_{{\sigma }_j}(x)\mid {\to }_{p}0$ where φ_σ(x) = φ(x/σ)/σ and φ(·) is the density function of the standard normal. Therefore, ${sup}_{\beta }\mid n^{-\frac{1}{2}}{f}_j^{\ast }(\beta )-{\phi }_{{\sigma }_j}\{n^{\frac{1}{2}}(\beta -{\widehat{\beta }}_{0j})\}\mid {\to }_{p}0$ and $n^{-\frac{1}{2}}{\widehat{c}}_3{\to }_p{c}_{30}$, where c₃₀ is the solution to ∫ I{φ_σj (x) > c₃₀}φ_σj (x)dx = 1 − α. It follows that the coverage of our CR converges to nominal levels since, with respect to probability measure ℙ^*, $\text{pr}({\beta }_{0j}\in {\text{CR}}_j^{\ast \text{HDR}})=\text{pr}\{f_j^{\ast }({\beta }_{0j})\ge {\widehat{c}}_3\}+o_{{\mathbb{P}}^{\ast }}(1)=\text{pr}\{n^{-\frac{1}{2}}{f}_j^{\ast }({\beta }_{0j})\ge n^{-\frac{1}{2}}{\widehat{c}}_3\}+o_{{\mathbb{P}}^{\ast }}(1)=\text{pr}\left[{\phi }_{{\sigma }_j}\{n^{\frac{1}{2}}({\beta }_{0j}-{\widehat{\beta }}_{0j})\}\ge c_{30}\right]+o_{{\mathbb{P}}^{\ast }}(1)\to 1-\alpha$.

Here we define our bias corrected estimator for β_0j, ${\widehat{\beta }}_j^{BC}={\widehat{\beta }}_j+I({\widehat{\beta }}_j\ne 0){\widehat{\text{bias}}}_j$, where ${\widehat{\text{bias}}}_j=(\frac{1}{M}{\sum }_{m=1}^M{\widehat{\beta }}_{j,m}^{\ast }){(-1)}^{I\left[{\sum }_{m=1}^M\{I({\widehat{\beta }}_{j,m}^{\ast }>0)-I({\widehat{\beta }}_{j,m}^{\ast }<0)\}<0\right]}{\left({\widehat{\mathbb{A}}}_{\lambda }^{-1}\right)}_{jj}/\{nmax(\mid {\widehat{\xi }}_{7.5}\mid ,\mid {\widehat{\xi }}_{97.5}\mid )\},{\widehat{\mathbb{A}}}_{\lambda }=n^{-1}\left({\mathbf{X}}_{\widehat{\mathcal{A}}}^{\top }{\mathbf{X}}_{\widehat{\mathcal{A}}}+n^{-\frac{1}{2}}{\lambda }_n\text{diag}{\left\{1/{\stackrel{\sim }{\beta }}_j^2\right\}}_{j=1}^p\right)$ and ξ̂_r is the r percentile of { ${\stackrel{\sim }{\beta }}_{j,m}^{\ast }$, m = 1, … M}. We estimate for ALASSO with following the methods of Cai et al. (2009) where a stabilized estimate of the covariance of coefficients from an accelerated failure time model is used.

A.4 Selection of λ with Bayes Information Criterion

In Section 2, we suggest choosing the tuning parameter λ_n by minimizing the BIC. Here we explicitly present the BIC for the linear regression objective function and ALASSO penalty that we utilized in the simulations and data example in Sections 3 and 4. First, assume that the data has been centered so there is no intercept. We implement a least squares approximation of the likelihood for BIC(λ) as in Wang and Leng (2007). For a given λ,

$$\text{BIC}(\lambda )={(\widehat{\mathit{\beta }}(\lambda )-\stackrel{\sim }{\mathit{\beta }})}^T{\widehat{\mathbf{\sum }}}_{\lambda }^{-1}(\widehat{\mathit{\beta }}(\lambda )-\stackrel{\sim }{\mathit{\beta }})+{\widehat{q}}_{\lambda }{\omega }_n$$

where β̂ (λ) minimizes the least squares objective function $\widehat{\mathcal{L}}(\mathit{\beta })={(\mathbf{y}-\mathbf{X}\mathit{\beta })}^T(\mathbf{y}-\mathbf{X}\mathit{\beta })+{\sum }_{j=1}^p{p}_{\lambda }^{\prime }(\mid {\stackrel{\sim }{\beta }}_j\mid )\mid {\beta }_j\mid$, based on (1), ${\widehat{\mathbf{\sum }}}_{\lambda }^{-1}={({\widehat{\sigma }}^{2}n)}^{-1}\{{\mathbf{X}}^T\mathbf{X}+\lambda \text{diag}{\{I({\widehat{\beta }}_j(\lambda )\ne 0)/\mid {\stackrel{\sim }{\beta }}_j{\stackrel{\sim }{\beta }}_j(\lambda )\mid \}}_{j=1}^p\}$ is a stabilized estimate of Σ similar to that in Zou (2006), σ̂² is the consistent estimate of σ from the linear regression model based on the residual variance, and q̂_λ estimates the degrees of freedom of ALASSO with the number of nonzero elements of β̂ (λ) (Zou et al. 2007). We choose ω_n = min(n^0.1, log(n)) because numerical results suggest that log(n) is much greater than n^0.1 and leads to excessive shrinkage of moderately sized parameters in finite sample.