Confidence Intervals for Sparse Penalized Regression with Random Designs

Abstract

With the abundance of large data, sparse penalized regression techniques are commonly used in data analysis due to the advantage of simultaneous variable selection and estimation. A number of convex as well as non-convex penalties have been proposed in the literature to achieve sparse estimates. Despite intense work in this area, how to perform valid inference for sparse penalized regression with a general penalty remains to be an active research problem. In this paper, by making use of state-of-the-art optimization tools in stochastic variational inequality theory, we propose a unified framework to construct confidence intervals for sparse penalized regression with a wide range of penalties, including convex and non-convex penalties. We study the inference for parameters under the population version of the penalized regression as well as parameters of the underlying linear model. Theoretical convergence properties of the proposed method are obtained. Several simulated and real data examples are presented to demonstrate the validity and effectiveness of the proposed inference procedure.

1. Introduction

With the advantage of simultaneous variable selection and estimation, sparse penalized regression techniques have been widely used. By introducing biases on the estimators, sparse penalized regression can often select a simpler model and produce estimators with smaller mean square errors than unpenalized regression. One of the well-known representatives is the L₁ penalized technique LASSO (Donoho and Johnstone, 1994; Tibshirani, 1996). LASSO has become a popular variable selection method due to its good selection performance and computational efficiency. Many other extensions with different penalties have been studied in the literature, see for example Fan and Li (2001); Zou and Hastie (2005); Candes and Tao (2007); Liu and Wu (2007); Lv and Fan (2009); Zhang (2010).

For computational implementation of these methods, there is a large literature on efficient algorithms. The LARS algorithm by Efron et al. (2004) and the Coordinate-Descent algorithm by Wu and Lange (2008) are two popular examples. Mazumder et al. (2011) proposed the SparseNet algorithm to deal with non-convex penalties. In terms of inference, much less development has been made, especially for estimators from non-convex penalized regression. For the LASSO, one common approach is to first perform model selection and then carry out inference based on some pivot distributions conditional on the selected model. For example, see Lee et al. (2016); Lockhart et al. (2014). This approach does not sufficiently account for the stochastic errors in the model selection step. Another popular approach is to achieve valid inference by adjusting the bias introduced by the L₁ regularization term. Papers along this line include Javanmard and Montanari (2014); Van de Geer et al. (2014); Zhang and Zhang (2014). Recently, Lu et al. (2017) suggested using a variational inequality formulation to establish an asymptotic distribution of LASSO estimators that can be used to construct confidence intervals (CI) for the population LASSO parameters as well as the true model parameters. Some other methods about high dimensional inference include Voorman et al. (2014); Ning et al. (2017); Zhao et al. (2017).

For sparse penalized regression with penalties more complex than the LASSO, not much work has been done on inference. In particular, with a non-convex penalty, the regression problem may have multiple local optimal solutions. This brings up the question of whether one can use a local solution to construct meaningful confidence intervals. The goal of this paper is to provide a unified framework to perform valid inference for sparse penalized regression with general penalties, based on a local solution. Our assumptions about the penalties are in line with the desired properties for regularized penalty functions given in Fan and Li (2001), namely, sparsity, unbiasness, and continuity. For example, this framework can be applied to the adaptive LASSO penalty (Zou, 2006), the non-convex log penalty (Friedman, 2012), SCAD (Fan and Li, 2001), MCP (Zhang, 2010), the transformed ${\mathcal{l}}_1$ penalty (Nikolova, 2000), and so on.

In this paper, we consider a general random-design penalized regression problem

$$\underset{{\beta }_0,\beta }{\min }E{[Y-{\beta }_0-\sum _{i=1}^p{\beta }_i{X}_i]}^2+\sum _{j=1}^p{P}_{{\lambda }_j}(|{\beta }_j|),$$ (1)

where $X={(X_1,\cdots ,X_p)}^T\in {ℝ}^p$ is an explanatory random vector with mean 0, and $Y\in ℝ$ is a response random variable. Here ${\beta }_0\in ℝ\text{and}\beta =({\beta }_1,\cdots ,{\beta }_p)\in {ℝ}^p$ are the regression parameters. For $j=1,2,\cdots ,p,P_{{\lambda }_j}(|\cdot |)$ is a general penalty for β_j with the regularization parameter λ_j. This general penalty covers many convex and non-convex penalties.

The solution of (1) can be estimated by the solution of the corresponding sample average approximation (SAA) problem

$$\underset{{\beta }_0,\beta }{\min }\frac{1}{N}||\mathbf{y}-{\beta }_0{1}_N-\mathbf{X}\beta {\Vert }_2^2+\sum _{j=1}^p{P}_{{\lambda }_j}(|{\beta }_j|),$$ (2)

where

$$\mathbf{y}=\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_N\end{array}\right],\mathbf{X}=\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1p}\\ x_{21}& x_{22}& \cdots & x_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ x_{N1}& x_{N2}& \cdots & x_{Np}\end{array}\right]=\left[\begin{array}{c}{\mathbf{x}}_1^T\\ {\mathbf{x}}_2^T\\ ⋮\\ {\mathbf{x}}_N^T\end{array}\right],1_N=\left[\begin{array}{c}1\\ 1\\ ⋮\\ 1\end{array}\right]\in {ℝ}^N,$$

and (x₁, y₁), ⋯, (x_N, y_N) are independent samples of (X, Y). For each i = 1, ⋯, N and j = 1, ⋯, p, $x_{ij}\in ℝ,y_i\in ℝ\text{and}{\mathbf{x}}_i\in {ℝ}^{p\times 1}.$ We refer to a local solution to the population penalized problem (1) as a population penalized parameter, and denote it as $({\tilde{\beta }}_0,\tilde{\beta }).$ We refer to a local solution to the SAA problem (2) as a penalized estimator, and denote it as $({\widehat{\beta }}_0,\widehat{\beta }).$.

The population penalized parameter $({\tilde{\beta }}_0,\tilde{\beta })$ is closely related to the traditional least squares parameter. When $P_{{\lambda }_j}\left(|{\beta }_j|\right)=0\text{for}j=1,\cdots ,p,$ the problem (1) becomes the following population least squares problem:

$$\underset{{\beta }_0,\beta }{\min }E{[Y-{\beta }_0-\sum _{i=1}^p{\beta }_i{X}_i]}^2,$$ (3)

which has a unique minimizer (E[XX^T])⁻¹ E[XY] when E[XX^T] is invertible. If additionally X and Y are related by the following linear model

$$Y={\beta }_0^{true}+X^T{\beta }^{true}+\epsilon$$ (4)

with E[ε|X] = 0, then the solution to the population least squares problem (3) is exactly $({\beta }_0^{true},{\beta }^{true})$, which we refer to as the true model parameter. When $P_{{\lambda }_j}(|{\beta }_j|)>0,({\tilde{\beta }}_0,\tilde{\beta })$ is not exactly $({\beta }_0^{true},{\beta }^{true})$, but there is a relation between $({\tilde{\beta }}_0,\tilde{\beta })$ and $({\beta }_0^{true},{\beta }^{true})$, that will be described in Section 3.4.

The idea of our proposed method is to use the penalized estimator $({\widehat{\beta }}_0,\widehat{\beta })$ to derive confidence intervals for the population penalized parameter $({\tilde{\beta }}_0,\tilde{\beta })$, and then exploit the relation between $({\tilde{\beta }}_0,\tilde{\beta })$ and $({\beta }_0^{true},{\beta }^{true})$ to derive confidence intervals for the true model parameter $({\beta }_0^{true},{\beta }^{true})$ in the linear model (4). Therefore, our proposed method could construct confidence intervals for both $({\tilde{\beta }}_0,\tilde{\beta })$ and $({\beta }_0^{true},{\beta }^{true})$. Note that the valid inference of $({\tilde{\beta }}_0,\tilde{\beta })$ is also useful for some problems such as the cost-effective linear regression taking into account the cost of collecting variables. For each sample, suppose we need to spend c_j dollars on collecting the value of the jth variable X_j where j = 1, 2,...,p. For this special linear regression problem, in order to find a relatively cheap linear model with good prediction performance, we need to estimate the regression coefficient vector that minimizes an objective function which balances between the expected prediction accuracy and the data collation cost, that is,

$$({\tilde{\beta }}_0,\tilde{\beta })\in \arg \underset{{\beta }_0,\beta }{\min }E{(Y-{\beta }_0-\sum _{j=1}^p{X}_j{\beta }_j)}^2+\lambda \sum _{j=1}^p{c}_{j}P(|{\beta }_j|),$$

where P(x) is a continuously differentiable non-convex function, and it is an approximation to the indicator function $\mathcal{I}(x)$ that equals to 1 if x > 0 and 0 if x = 0. The parameter λ can be selected according to the budget. In this example, the population penalized parameter becomes a reasonable target of the inference and our proposed method could deliver its asymptotically exact confidence intervals. On the other hand, even when a model on the relation between X and Y is not available, the confidence intervals of $({\tilde{\beta }}_0,\tilde{\beta })$ can still provide measures on the randomness of the penalized estimators.

The main techniques to construct confidence intervals for $({\tilde{\beta }}_0,\tilde{\beta })$ and $({\beta }_0^{true},{\beta }^{true})$, consist of three steps. First, we transform problems (1) and (2) into their corresponding variational inequality and normal map formulations and obtain an asymptotic distribution of a solution to the normal map formulation of (2). Next, by finding reliable estimates for quantities that describe the asymptotic distribution, we provide methods to compute confidence intervals for $({\tilde{\beta }}_0,\tilde{\beta })$ based on a solution to the normal map formulation of (2). Finally, we establish the connection between $({\tilde{\beta }}_0,\tilde{\beta })$ and $({\beta }_0^{true},{\beta }^{true})$, from which we obtain the bias-corrected estimator of $({\beta }_0^{true},{\beta }^{true})$, the asymptotic distribution and confidence intervals. The methodology in this paper is developed for a fixed dimension p, based on a local solution $({\widehat{\beta }}_0,\widehat{\beta })$ to (2). The confidence intervals we obtain for $({\tilde{\beta }}_0,\tilde{\beta })$ are for the local solution to (1) close to $({\widehat{\beta }}_0,\widehat{\beta })$. On the other hand, for any local solution $({\widehat{\beta }}_0,\widehat{\beta })$ of (2), we can always obtain confidence intervals for the true model parameter $({\beta }_0^{true},{\beta }^{true})$. Indeed, under the setting we consider in Section 3.4, a local solution $({\widehat{\beta }}_0,\widehat{\beta })$ almost surely converges to $({\beta }_0^{true},{\beta }^{true})$.

Although our method and the method proposed in Lu et al. (2017) use similar techniques, there are important new contributions. First, we propose a unified framework to construct confidence intervals for a large class of penalties, including the method using the LASSO penalty (Lu et al. (2017)) as a special case. Second, for non-convex penalties, the construction of the confidence intervals and the theoretical studies are more complicated. We propose a new transformation of the original optimization problem to its corresponding variational inequality and normal map formulations. Special technical conditions and theoretical results for general penalties are studied. Third, the proposed method based on non-convex penalties could deliver better confidence intervals than the methods using convex penalties. For example, in our numerical studies, we compare the method using the MCP penalty with a = 2 and the method using the MCP penalty with a = 2000 (the case that MCP penalty is very close to the LASSO penalty). Our numerical results indicate that the lengths of the confidence intervals for a = 2 are generally shorter than those for a = 2000 with similar coverage rates. This is possibly due to the smaller bias imposed by the MCP penalty with a small a.

The rest of this paper is organized as follows. In Section 2, we show some background on variational inequality, and present the problem transformations. Section 3 discusses how to obtain the confidence intervals for the population penalized parameters as well as the true model parameters in the linear model. Some theoretical results about convergence properties are shown in this section. In Section 4, we present numerical results to illustrate the performance of the proposed method. Section 5 contains some discussion. The technical details of variational inequalities and proofs are shown in the supplementary materials.

Throughout this paper, we use $\mathcal{N}(0,\mathrm{\Sigma })$ to denote a normal random vector with mean zero and covariance matrix $\mathrm{\Sigma },\text{and}{Y}_n\Rightarrow Y$ to represent weak convergence of a sequence of random variables {Y_n} to Y. The inner product between two vectors x and y is denoted as 〈x, y〉. For a convex set S and a vector $z\in {ℝ}^n,\text{we use}{\mathrm{\Pi }}_S(z)$ to denote the Euclidean projection onto S. A function $f:{ℝ}^n\to {ℝ}^m$ is said to be B-differentiable at a point $x_0\in {ℝ}^n$ if there exists a positively homogeneous function $df(x_0):{ℝ}^n\to {ℝ}^m,$ such that $f(x_0+v)=f(x_0)+df(x_0)(v)+o(v).$ The function df(x₀) is called the B-derivative of f at x₀.

2. Background and problem transformations

In this section, we first introduce some background on variational inequalities and normal maps. Then we introduce how to transform the problems (1) and (2) to their corresponding variational inequality and normal map formulations. Some assumptions for our theoretical analysis are also given in this section.

2.1. Background on variational inequalities and normal maps

We start with the definition of a variational inequality. Given a function $f:{ℝ}^n\to {ℝ}^n$ and a closed, convex set S in ${ℝ}^n$, the variational inequality associated with (f, S) is the problem of finding $x\in S$ such that

$$0\in f(x)+N_S(x),$$ (5)

where N_s(x) is the normal cone to S at x defined as

$$N_S(x)=\{v\in {ℝ}^n|\langle v,s-x\rangle ⩽0\text{for each}s\in S\}.$$

Variational inequalities are closely related to optimization problems. Consider a problem of minimizing an objective function $F:{ℝ}^n\to ℝ$ over a closed and convex set $S\subseteq {ℝ}^n.$ A well-known fact is that if $x\in S$ is a local solution to this minimization problem and F is differentiable at x, then x satisfies the variational inequality

$$0\in \nabla F(x)+N_S(x),$$

where $\nabla F:{ℝ}^n\to {ℝ}^n$ is the gradient of the function F. Conversely, if x satisfies the above variational inequality, and F is a convex function, then x is a global minimizer of F over the set S.

Besides the above connection with the original minimization problem, the variational inequality can be equivalently formulated as an equation, using a concept called the normal map. The normal map induced by f and S is a function $f_S:{ℝ}^n\to {ℝ}^n$ given by

$$f_S(z)=f\left({\mathrm{\Pi }}_S(z)\right)+\left(z-{\mathrm{\Pi }}_S(z)\right)\text{for each}z\in {ℝ}^n,$$

where ${\mathrm{\Pi }}_S(z)$ denotes the Euclidean projector of z onto S. For any solution $x\in S$ to the variational inequality (5), the point z = x − f(x) satisfies ${\mathrm{\Pi }}_S(z)=x$ and

$$f_s(z)=0.$$ (6)

Conversely, for any solution z to (6), the point $x={\mathrm{\Pi }}_S(z)$ is a solution to (5) and satisfies z = x − f(x). Equation (6) is called the normal map formulation of (5).

To understand the above relations, we consider an example of minimizing $F(x)=\frac{1}{2}\Vert x-x_0{\Vert }_2^2$ for a fixed point $x_0\in {ℝ}^n$ over the convex set S. We know that the solution is the projection of x₀ onto the convex set S, denoted as ${\mathrm{\Pi }}_S(x_0).$ For the function F(x), the gradient is $\nabla F(x)=x-x_0.$ Thus, the variational inequality formulation (5) of this minimization problem is the problem of finding $x\in S$ such that $x_0-x\in N_S(x).$ That is, we need to find $x\in S$ such that 〈x₀ − x, s − x〉 ≤ 0 for each $s\in S$. We can show that the solution is ${\mathrm{\Pi }}_S(x_0).$. On the other hand, the normal map induced by $\nabla F(x)\text{and}S\text{is}{f}_S(z)=\nabla F\left({\mathrm{\Pi }}_S(z)\right)+z-{\mathrm{\Pi }}_S(z)={\mathrm{\Pi }}_S(z)-x_0+z-{\mathrm{\Pi }}_S(z)=z-x_0.$ Thus, the normal map formulation (6) of this minimization problem is z − x₀ = 0. The solution to this equation is x₀ and the point $x={\mathrm{\Pi }}_S(x_0)$ is the solution to the original minimization problem. More details about the variational inequality and normal map can be found in the supplementary materials. In the following Sections 2.2 and 2.3, we show how to transform the problems (1) and (2) to their corresponding normal map formulations, respectively.

2.2. Transformations of the population penalized regression

In this subsection, we transform the optimization problem (1) into a normal map formulation. Before discussing details about the transformation, we introduce conditions on the penalties P_λi(·). In this subsection, as well as in Sections 2.3 and 3.1-3.3, λ = (λ₁, ⋯, λ_p) > 0 is fixed.

Assumption 1.

• (a)For each i = 1, 2, ⋯,p, P_λi(·) is nonnegative, nondecreasing and continuously differentiable on $[0,+\infty )\mathit{\text{with}}{P}_{{\lambda }_i}^{\prime }(0)>0.$
• (b)For any local solution $({\tilde{\beta }}_0,\tilde{\beta })$ to (1), the second derivative of P_λi (t_i) is Lipchitz continuous on a neighborhood of $t_i=|{\tilde{\beta }}_i|$ for every i = 1, ⋯,p.

Many well-known penalty functions satisfy Assumption 1(a). We list five penalty functions as examples.

• (a)The adaptive LASSO penalty (Zou, 2006) defined as $P_{{\lambda }_i}(|{\beta }_i|)={\lambda }_i|{\beta }_i|,$ where λ_i is the weight for the i^th coordinate.
• (b)The log penalty (Friedman, 2012) defined as $P_{{\lambda }_i}\left(|{\beta }_i|\right)=\frac{{\lambda }_i}{\text{log}(1+a)}\text{log}(a|{\beta }_i|+1),$ where a > 0.
• (c)The transformed ${\mathcal{l}}_1$ penalty (Nikolova, 2000) defined as $P_{{\lambda }_i}\left(|{\beta }_i|\right)={\lambda }_i\frac{(a+1)|{\beta }_i|}{a+|{\beta }_i|},$ where a > 0.
• (d)
  The SCAD penalty (Fan and Li, 2001) defined as P_λ(0) = 0 and

  $$P_{{\lambda }_i}^{\prime }(|{\beta }_i|)={\lambda }_i{\mathbb{1}}_{|{\beta }_i|⩽{\lambda }_i}+\frac{{(a{\lambda }_i-|{\beta }_i|)}_{+}}{a-1}{\mathbb{1}}_{|{\beta }_i|>{\lambda }_i}\text{where}a>2.$$ (7)
• (e)
  The MCP penalty (Zhang, 2010) defined as

  $$P_{{\lambda }_i}\left(|{\beta }_i|\right)={\lambda }_i\left(|{\beta }_i|-\frac{{\beta }_i^2}{2a{\lambda }_i}\right){\mathbb{1}}_{|{\beta }_i|<a{\lambda }_i}+\frac{a{\lambda }_i^2}{2}{\mathbb{1}}_{|{\beta }_i|⩾a{\lambda }_i}\text{where}a>0.$$ (8)

We can check that penalties (a), (b), and (c) satisfy Assumption 1(b). The SCAD and MCP penalties satisfy this assumption almost everywhere. Take the SCAD penalty for example. The function corresponds to a quadratic spline with two knots, at which it is not continuously twice differentiable. Assumption 1(b) requires that no local solution to (1) locates at these two knots for each i. It is a reasonable assumption since the set of points on which twice continuous differentiability fails has measure zero.

In the assumption below, part (a) is to ensure the objective function of (1) to be finite valued, and part (b) will be used in proving convergence results.

Assumption 2.

• (a)The expectations $E[X_1^2],\cdots ,E[X_p^2]\mathit{\text{and}}E[Y^2]$ are finite.
• (b)The expectations $E[X_1^4],\cdots ,E[X_p^4]\mathit{\text{and}}E[Y^4]$ are finite.

Next, we transform the problem (1) into a normal map formulation in three steps. In the first step, we introduce an equivalent problem, in which a new variable $t\in {ℝ}^p$ is added to eliminate the non-smooth term ${\sum }_{i=1}^p{P}_{{\lambda }_i}(|{\beta }_i|)$ from the objective function (1). The new problem is as follows:

$$\begin{array}{l}\underset{{\beta }_0,\beta ,t}{\min }\text{E}{[Y-{\beta }_0-{\sum }_{i=1}^p{\beta }_i{X}_i]}^2+{\sum }_{i=1}^p{P}_{{\lambda }_i}\left(t_i\right)+m\left(\Vert t{\Vert }_2^2-\Vert \beta {\Vert }_2^2\right)\\ \begin{array}{ll}\text{s.t.}\hfill & t_i-{\beta }_i⩾0,i=1,\cdots ,p,\hfill \\ \hfill & t_i+{\beta }_i⩾0,i=1,\cdots ,p,\hfill \end{array}\end{array}$$ (9)

where m is a positive constant. If we define $S_i\subset {ℝ}^2$ as

$$S_i=\{\left({\beta }_i,t_i\right)|t_i-{\beta }_i⩾0,t_i+{\beta }_i⩾0\},i=1,\cdots ,p,$$ (10)

and write

$$({\beta }_0,\beta ,t)=({\beta }_0,{\beta }_1,t_1,{\beta }_2,t_2,\cdots ,{\beta }_p,t_p),$$ (11)

then we can treat the feasible set of (9), denoted by S, as a Cartesian product

$$S=ℝ\times {\mathrm{\Pi }}_{i=1}^p{S}_i.$$ (12)

We will use the two ways of ordering of (β₀, β, t) in (11) interchangeably for notational convenience.

Note that the above transformation is different from the method shown in Lu et al. (2017) for the LASSO penalty. The term $m(\Vert t{\Vert }_2^2-\Vert \beta {\Vert }_2^2)$ is added into the objective function of (9) in order to ensure t_i = |β_i| in any optimal solution to (9), so that there is an one-to-one correspondence between the optimal solutions to (1) and (9). This is necessary and important when the penalty functions are not strictly increasing on [0, $+\infty$). For instance, some non-convex penalties such as SCAD and MCP are flat on some intervals [d_i, $+\infty$).

In the second step, we transform (9) into a variational inequality. To this end, we need to write down the gradient of its objective function. Define a function $F:ℝ\times {ℝ}^p\times {ℝ}^p\times {ℝ}^p\times ℝ\to {ℝ}^{2p+1}$ as

$$F({\beta }_0,\beta ,t,X,Y)=\left[\begin{array}{c}-2(Y-{\beta }_0-\sum _{i=1}^p{\beta }_i{X}_i)\\ -2(Y-{\beta }_0-\sum _{i=1}^p{\beta }_i{X}_i)X-2m\beta \\ ({P_{{\lambda }_i}^{\prime }(t_i)+2m{t}_i)}_{i=1}^p\end{array}\right].$$ (13)

Furthermore, define a function $f_0:ℝ\times {ℝ}^p\times {ℝ}^p\to {ℝ}^{2p+1}$ as

$$f_0\left({\beta }_0,\beta ,t\right)=E[F\left({\beta }_0,\beta ,t,X,Y\right)].$$ (14)

The function f₀ is well defined and finite valued under Assumption 2(a). If P_λi (t_i) is twice differentiable at t_i for every i = 1, ⋯,p, then we can write down the derivative of F with respect to (β₀, β, t) as

$$d_{1}F({\beta }_0,\beta ,t,X,Y)=\left[\begin{array}{ccc}2& 2X^T& 0\\ 2X& 2X{X}^T-2m{I}_p& 0\\ 0& 0& \mathrm{diag}{(P_{{\lambda }_i}^{″}(t_i)+2m)}_{i=1}^p\end{array}\right],$$ (15)

where $\mathrm{diag}{(P_{{\lambda }_i}^{″}(t_i)+2m)}_{i=1}^p$ represents the diagonal matrix whose ith diagonal element is $P_{{\lambda }_i}^{″}\left(t_i\right)+2m\text{and}{I}_p$ is the p × p identity matrix. Moreover, the Jacobian matrix of f₀ is

$$L(t)=E\left[d_{1}F\left({\beta }_0,\beta ,t,X,Y\right)\right]=\left[\begin{array}{ccc}2& 0& 0\\ 0& 2E[X{X}^T]-2m{I}_p& 0\\ 0& 0& \mathrm{diag}{(P_{{\lambda }_i}^{″}(t_i)+2m)}_{i=1}^p\end{array}\right].$$ (16)

The lemma below shows that there is an one-to-one correspondence between the (local or global) optimal solutions to (1) and (9).

Lemma 1.

Suppose Assumptions 1(a) and 2(a) hold. Then the objective function of (9) is finite valued on ${ℝ}^{2p+1},$ and its gradient at each $\left({\beta }_0,\beta ,t\right)\in {ℝ}^{2p+1}\mathit{\text{is}}{f}_0\left({\beta }_0,\beta ,t\right).$ If $({\tilde{\beta }}_0,\tilde{\beta },\tilde{t})$ is a (local) optimal solution to (9), then ${\tilde{t}}_i=|{\tilde{\beta }}_i|\mathit{\text{for all}}i=1,\cdots ,p,$ and $({\tilde{\beta }}_0,\tilde{\beta })$ is a (local) optimal solution to (1). Conversely, if $({\tilde{\beta }}_0,\tilde{\beta })$ is a (local) optimal solution to (1), then $({\tilde{\beta }}_0,\tilde{\beta },\tilde{t})$ is a (local) optimal solution to (9), where ${\tilde{t}}_i=|{\tilde{\beta }}_i|\mathit{\text{for all}}i=1,\cdots ,p.$

If Assumption 1(b) holds additionally, then the Hessian matrix of the objective function of (9) at $({\tilde{\beta }}_0,\tilde{\beta },\tilde{t})$ is L($\tilde{t}$).

In view of Lemma 1, we can transform (9) to the following variational inequality:

$$-f_0\left({\beta }_0,\beta ,t\right)\in N_S\left({\beta }_0,\beta ,t\right).$$ (17)

In the last step, we state the normal map formulation for (17). Let (f₀)_S be the normal map induced by f₀ and S. Then the normal map formulation for (17) is

$${\left(f_0\right)}_S(z)=0,z\in {ℝ}^{2p+1}.$$ (18)

For the rest of the paper, let $({\tilde{\beta }}_0,\tilde{\beta },\tilde{t})$ be a local solution to (9). Then $({\tilde{\beta }}_0,\tilde{\beta },\tilde{t})$ is also a solution to (17). Therefore, the point $z_0\in {ℝ}^{2p+1}$ defined as

$$z_0=({\tilde{\beta }}_0,\tilde{\beta },\tilde{t})-f_0({\tilde{\beta }}_0,\tilde{\beta },\tilde{t})$$ (19)

is a solution to (18) and satisfies ${\mathrm{\Pi }}_S\left(z_0\right)=({\tilde{\beta }}_0,\tilde{\beta },\tilde{t}).$

Let Σ₀ be the covariance matrix of $F({\tilde{\beta }}_0,\tilde{\beta },\tilde{t},X,Y).$ We can check that Σ₀ is well defined if Assumption 2(b) holds. Let ${\mathrm{\Sigma }}_0^1$ be the upper left (p + 1) × (p + 1) submatrix of Σ₀. Since the last p elements of $F({\tilde{\beta }}_0,\tilde{\beta },\tilde{t},X,Y)$ are not random at $({\tilde{\beta }}_0,\tilde{\beta },\tilde{t})$, we have ${\mathrm{\Sigma }}_0=\left[\begin{array}{cc}{\mathrm{\Sigma }}_0^1& 0\\ 0& 0\end{array}\right].$. In our theoretical analysis as shown in Section 3, we found that the B-derivative of the normal map (f₀)_S at z₀ plays an important role in the construction of the confidence intervals. To study the property of the B-derivative of the normal map (f₀)_S at z₀, we need the following assumption.

Assumption 3.

Let $({\tilde{\beta }}_0,\tilde{\beta })$ be a local solution to (1), define $\tilde{t}\in {ℝ}^p\mathit{\text{and}}\tilde{q}\in {ℝ}^p$ by

$${\tilde{t}}_i=|{\tilde{\beta }}_i|\mathit{\text{and}}{\tilde{q}}_i=E[-2(Y-{\tilde{\beta }}_0-\sum _{j=1}^p{\tilde{\beta }}_j{X}_j)X_i]\mathit{\text{for each}}i=1,\cdots ,p.$$

Let $\Im$ be a subset of {1, ⋯,p} defined as

$$\Im =\{i\in \{1,\cdots ,p\}|{\tilde{\beta }}_i\ne 0\mathit{\text{or}}({\tilde{\beta }}_i=0\mathit{\text{and}}|{\tilde{q}}_i|=|P_{{\lambda }_i}^{\prime }({\tilde{t}}_i)|)\},$$

and denote $L(\tilde{t})$ in (16) by L. Let Q₁ be the submatrix of L that consists of intersections of columns and rows of L with indices in $\{1\}\cup \{i+1,i\in \Im \},$ and let Q₂ be the submatrix of L that consists of intersections of columns and rows of L with indices in $\{i+p+1,i\in \Im \}.$ Define matrix Q as

$$Q=Q_1+\left[\begin{array}{cc}0& 0\\ 0& Q_2\end{array}\right].$$ (20)

Assume that Q is nonsingular.

In the above assumption, Q_i is a submatrix of the upper left (p + 1) × (p + 1) submatrix of L, and Q₂ is a submatrix of the lower right p × p submatrix of L. If $({\tilde{\beta }}_0,\tilde{\beta })$ is a solution to the optimization problem (1), then for every $i\in \{1,\dots ,p\},\text{we have}{\tilde{\beta }}_i\ne 0\mathit{\text{and}}|{\tilde{q}}_i|=|P_{{\lambda }_i}^{\prime }({\tilde{t}}_i)|,\text{or}{\tilde{\beta }}_i=0\mathit{\text{and}}|{\tilde{q}}_i|\le |P_{{\lambda }_i}^{\prime }({\tilde{t}}_i)|.$ Since we have ${\tilde{\beta }}_i=0\text{and}|{\tilde{q}}_i|<|P_{{\lambda }_i}^{\prime }\left({\tilde{t}}_i\right)|$ for some coefficients, $|\mathcal{J}|$ is generally not equal to p. Furthermore, in our theoretical analysis, we assume that the dimension p is fixed, the matrix Q can be nonsingular in many cases. The nonsingularity of Q is a standard assumption to guarantee that $({\tilde{\beta }}_0,\tilde{\beta })$ is a locally unique optimal solution.

As shown in Robinson (1995), the B-derivative of the normal map (f₀)_S at z₀ is the same as the normal map L_K induced by the linear function defined by the matrix L and the critical cone K to S associated with z₀, defined as

$$\begin{gathered} K=\{w\in T_S({\mathrm{\Pi }}_S(z_0))|\langle z_0-{\mathrm{\Pi }}_S(z_0),w\rangle =0\} \\ =\{w\in T_S({\tilde{\beta }}_0,\tilde{\beta },\tilde{t})|\langle f_0({\tilde{\beta }}_0,\tilde{\beta },\tilde{t}),w\rangle =0\}, \end{gathered}$$ (21)

where for each $x\in S,$

$T_S(x)=\{w\in {ℝ}^n|\exists \{x_k\}\subset S\text{and}\{{\tau }_k\}\subset ℝ$ such that $x_k\to x,{\tau }_k\to 0,\text{and}(x_k-x)/{\tau }_k\to w\}$ is the tangent cone to S at x. To be specific, the normal map L_K is defined as $L_K(z)=L\cdot {\mathrm{\Pi }}_K(z)+z-{\mathrm{\Pi }}_K(z)\text{for any}z\in {ℝ}^{2p+1}.$ The tangent cone T_S(x) contains all the directions along which x can be approached by a sequence of points in S converging to x. Lemma 2 below shows that L_K is a global homeomorphism from ${ℝ}^{2p+1}\text{to}{ℝ}^{2p+1}$ (a continuous bijective function from ${ℝ}^{2p+1}\text{to}{ℝ}^{2p+1}$ whose inverse function is also continuous). In the proof of Lemma 2, we will give the explicit expression of the critical cone K.

Lemma 2.

Suppose that Assumptions 1, 2(a) and 3 hold. Then the normal map L_K is a global homeomorphism from ${ℝ}^{2p+1}\text{to}{ℝ}^{2p+1}$, and there is a neighborhood of $({\tilde{\beta }}_0,\tilde{\beta },\tilde{t})$ in which it is the unique local solution to (9).

Combining Lemma 1 and 2, we can conclude that the assumptions in Lemma 2 guarantee $({\tilde{\beta }}_0,\tilde{\beta })$ to be the unique local solution to (1) in a neighborhood of it.

2.3. Transformations of the SAA problem

We follow the same steps in Subsection 2.2 to formulate the SAA problem (2) as a normal map equation. First, by introducing the variable $t\in {ℝ}^p,$ we transform the SAA problem (2) to the following equivalent problem:

$$\underset{\left({\beta }_0,\beta ,t\right)\in S}{\min }\frac{1}{N}\sum _{i=1}^N{\left[y_i-{\beta }_0-\sum _{j=1}^p{\beta }_j{x}_{ij}\right]}^2+\sum _{i=1}^p{P}_{{\lambda }_i}(t_i)+m(\Vert t{\Vert }_2^2-\Vert \beta {\Vert }_2^2).$$ (22)

Second, we rewrite (22) as a variational inequality

$$0\in f_N\left({\beta }_0,\beta ,t\right)+N_S\left({\beta }_0,\beta ,t\right),$$ (23)

where $f_N\left({\beta }_0,\beta ,t\right)=N^{-1}{\sum }_{i=1}^{N}F({\beta }_0,\beta ,t,{\mathbf{x}}_i,y_i).\text{If}{P}_{{\lambda }_i}(t_i)$ is twice differentiable at t_i for every i = 1, ⋯,p, then the Jacobian matrix of f_N is given by

$$L_N(t)=d{f}_N\left({\beta }_0,\beta ,t\right)=\left[\begin{array}{ccc}2& 2{\sum }_{i=1}^T{\mathbf{x}}_i^T/N& 0\\ 2\sum _{i=1}^N{\mathbf{x}}_i/N& 2{\sum }_{i=1}^T{\mathbf{x}}_i{\mathbf{x}}_i^T/N-2m{I}_p& 0\\ 0& 0& \mathrm{diag}{\left(P_{{\lambda }_i}^{″}(t_i)+2m\right)}_{i=1}^p\end{array}\right].$$ (24)

Third, denoting the normal map induced by f_N and S by (f_N)_S, we obtain the normal map formulation of (23) as

$${(f_N)}_S(z)=0.$$ (25)

Let $({\widehat{\beta }}_0,\widehat{\beta },\widehat{t})$ be a local solution to (22). Then, $({\widehat{\beta }}_0,\widehat{\beta },\widehat{t})$ is also a solution to (23). So the point $z_N\in {ℝ}^{2p+1}$ defined as

$$z_N=({\widehat{\beta }}_0,\widehat{\beta },\widehat{t})-f_N({\widehat{\beta }}_0,\widehat{\beta },\widehat{t})$$ (26)

is a solution to (25) and satisfies ${\mathrm{\Pi }}_S(z_N)=({\widehat{\beta }}_0,\widehat{\beta },\widehat{t}).$

In fact, under Assumptions 1, 2 and 3, z_N is a locally unique solution to (25) when N is large enough and it converges to a solution z₀ to (18). This result will be shown in Subsection 3.1. Correspondingly, $({\widehat{\beta }}_0,\widehat{\beta },\widehat{t})$ is a locally unique solution to (22) and converges to a local solution $({\tilde{\beta }}_0,\tilde{\beta },\tilde{t})$ to (9). Let Σ_N be the sample covariance matrix of ${\{F({\widehat{\beta }}_0,\widehat{\beta },\widehat{t},{\mathbf{x}}_i,y_i)\}}_{i=1}^N\mathrm{and}{\mathrm{\Sigma }}_N^1$ be the upper left (p + 1) × (p + 1) submatrix of Σ_N, then we have ${\mathrm{\Sigma }}_N=\left[\begin{array}{cc}{\mathrm{\Sigma }}_N^1& 0\\ 0& 0\end{array}\right].$ Lu et al. (2017) (Lemma 3) shows that Σ_N converges to Σ₀ almost surely as $N\to \infty$ for t LASSO penalty. We can similarly prove the same convergence result with a general penalty in this paper under Assumptions 1–4. Assumption 4 is shown as follows.

Assumption 4.

• (a)
  For each $h\in {ℝ}^{2p+1}and\left({\beta }_0,\beta ,t\right)\in {ℝ}^{2p+1},let$

  $$M_{{\beta }_0,\beta ,t}(h)=E[\exp \{\langle h,F({\beta }_0,\beta ,t,X,Y)-f_0({\beta }_0,\beta ,t)\rangle \}]$$

  be the moment generating function of the random variable $F\left({\beta }_0,\beta ,t,X,Y\right)-f_0\left({\beta }_0,\beta ,t\right).$ Let $\mathcal{C}$ be a compact set in ${ℝ}^{2p+1}$ that contains $({\tilde{\beta }}_0,\tilde{\beta },\tilde{t})$ in its interior, and on which the second derivative of P_λi (t_i) is Lipchitz continuous for each i = 1, ㏯, p. Assume the following conditions.

  1. There exists a constant $\zeta >0$ such that $M_{{\beta }_0,\beta ,t}(h)\le \exp \{{\zeta }^2\Vert h{\Vert }^2/2\}$ for each $h\in {ℝ}^{2p+1}\mathit{\text{and}}({\beta }_0,\beta ,t)\in \mathcal{C}.$
  2. There exists a nonnegative random variable κ(X, Y) such that

     $$\Vert F({\beta }_0,\beta ,t,X,Y)-F({\beta }_0^{\prime },{\beta }^{\prime },t^{\prime },X,Y)\Vert \le \kappa (X,Y)\Vert ({\beta }_0,\beta ,t)-({\beta }_0^{\prime },{\beta }^{\prime },t^{\prime })\Vert$$

     for all $({\beta }_0,\beta ,t)\mathit{\text{and}}({\beta }_0^{\prime },{\beta }^{\prime },t^{\prime })$ in $\mathcal{C}$ and almost every (X, Y).
  3. The moment generating function of k is finite valued in a neighborhood of zero.
• (b)The same conditions as in (a) for d₁F(β₀, β, t, X, Y) instead of F(β₀, β, t, X, Y). Accordingly, use E[d₁F(β₀, β, t, X, Y)] to replace f₀(β₀, β, t) in the conditions.
• (c)The same conditions as in (a) for F(β₀, β, t, X, Y)F(β₀, β, t, X, Y)^T. Accordingly, use E[F(β₀, β, t, X, Y)F(β₀, β, t, X, Y)^T] to replace β₀(β₀, β, t) in the conditions.

Assumption 4(a) imposes conditions on the random variable F(β₀, β, t, X, Y) as well as the penalty terms. It will hold if (X, Y) is a bounded random variable and Assumption 1(b) holds. Assumption 4(a) is used to ensure the SAA function f_N to converge to f₀ in probability at an exponential rate. We state the result in the following lemma.

Lemma 3.

Suppose that Assumptions 1, 2 and 4(a) hold. Then there exist positive real numbers δ₁, μ₁, M₁ and σ₁ such that the following holds for each $ϵ>0$ and each sufficiently large N:

$$\mathrm{Prob}\left\{\underset{\left({\beta }_0,\beta ,t\right)\in \mathcal{C}}{\sup }\Vert f_N\left({\beta }_0,\beta ,t\right)-f_0\left({\beta }_0,\beta ,t\right)\Vert ⩾ϵ\right\}⩽{\delta }_1\exp \left\{-N{\mu }_1\right\}+\frac{M_1}{{ϵ}^{2p+1}}\exp \left\{-\frac{N{ϵ}^2}{{\sigma }_1}\right\}.$$ (27)

Parts (b) and (c) of Assumption 4 impose the same type of assumptions on different random variables. Assumption 4(a-b) is needed to construct a reliable estimate for an unknown quantity in the asymptotic distribution in Theorem 1. Assumption 4(c) is only needed when the matrix ${\mathrm{\Sigma }}_0^1$ is singular.

3. Construction of confidence intervals using stochastic variational inequality techniques

In this section, we show the proposed method and some related theoretical results to construct confidence intervals using stochastic variational inequality techniques. We first develop the limiting distribution of SAA solutions in Section 3.1. Then, in Section 3.2, we show how to estimate the unknown quantities in the limiting distribution. The construction of the confidence intervals for the population penalized parameters and the true model parameters in the underlying linear model will be studied in Section 3.3 and Section 3.4, respectively. To present our proposed inference method clearly, we outline the procedures to construct confidence intervals for the true model parameters in Table 1. The extension to the high dimensional case is provided in Section 3.5.

Table 1:

Construction of the (1 − α)% confidence intervals of ${\beta }_0^{true}\text{and}{\beta }^{true}$

Step 1. Find the penalized estimates ${\widehat{\beta }}_0$ and $\widehat{\beta }$ by solving the SAA problem (2). The tuning parameters are chosen by the Generalized Information Criterion (GIC).

Step 2. Calculate the solution of the normal map formulation (25),

$z_N=({\widehat{\beta }}_0,\widehat{\beta },\widehat{t})-f_N({\widehat{\beta }}_0,\widehat{\beta },\widehat{t}),\text{where}{\widehat{t}}_i=|{\widehat{\beta }}_i|,i=1,\cdots ,p.$

Step 3. Calculate $({\hat{\beta }}_0^{true},{\hat{\beta }}^{true})=G^{*}(z_N),$ where G* is the function defined in (59).

Step 4. If for every i ∈ {0, 1, 2,...,p}, ${\widehat{h}}_i$ is very close to 0, we consider Case I to construct individual confidence intervals approximately. Otherwise, we consider Case II.

Case I: the (1 − α)% confidence interval of ${\beta }_i^{true}$ is [${\hat{\beta }}_i^{true}-{\mathrm{\Phi }}^{-1}(1-\alpha /2)·{\tau }_i,{\hat{\beta }}_i^{true}+{\mathrm{\Phi }}^{-1}(1-\alpha /2)·{\tau }_i$], where ${\tau }_i=\sqrt{H_N{\mathrm{\Sigma }}_N{H}_N^T/N}$ and HN is defined in Theorem 5;

Case II: we first use simulation to estimate the (1 − α/2)% percentile of ${\widehat{R}}_{i+1}(Z)$, where ${\widehat{R}}_{i+1}$ is defined by (56) and $Z~\mathcal{N}(0,I_{p+1})\times \overrightarrow{\mathbf{1}}$. The estimated percentile is denoted as ηi. The (1 − α)% confidence interval of ${\beta }_i^{true}$ is [${\hat{\beta }}_i^{true}-{\eta }_i/\sqrt{N},{\hat{\beta }}_i^{true}+{\eta }_i/\sqrt{N}$].

3.1. Convergence and distribution of SAA solutions

Based on Lemma 2 and the relation between (9) and (18), Assumptions 1–3 guarantee z₀ defined in (19) to be a locally unique solution to (18). Furthermore, we show in Theorem 1 below that for sufficiently large N, (25) has a unique solution z_N in a neighborhood of z₀, and that z_N converges almost surely to z₀. This theorem also provides results on asymptotic distributions and convergence rates.

Theorem 1.

Suppose that Assumptions 1, 2 and 3 hold. Then, with probability 1, there exist neighborhoods $\mathcal{Z}\mathit{\text{of}}{z}_0\mathit{\text{and}}{\mathcal{C}}_0\mathit{\text{of}}({\tilde{\beta }}_0,\tilde{\beta },\tilde{t}),$ such that for sufficiently large N, the equation (25) has a unique solution z_N in $\mathcal{Z}$, and the variational inequality (23) has a unique solution in ${\mathcal{c}}_0$ given by $({\widehat{\beta }}_0,\widehat{\beta },\widehat{t})={\mathrm{\Pi }}_S(z_N).$ Moreover,

$$\underset{N\to \infty }{\lim }{z}_N=z_0\mathit{\text{a.s.,}}\underset{N\to \infty }{\lim }({\widehat{\beta }}_0,\widehat{\beta },\widehat{t})=({\tilde{\beta }}_0,\tilde{\beta },\tilde{t})\mathit{\text{a.s.,}}$$ (28)

$$\sqrt{N}(z_N-z_0)\Rightarrow {(L_K)}^{-1}(\mathcal{N}(0,{\mathrm{\Sigma }}_0)),$$ (29)

and

$$\sqrt{N}{L}_K(z_N-z_0)\Rightarrow \mathcal{N}(0,{\mathrm{\Sigma }}_0).$$ (30)

In addition, if Assumption 4(a-b) holds, then there exist positive real numbers ${ϵ}_0,{\delta }_0,{\mu }_0,M_0$ and σ₀, such that for each $ϵ\in (0,{ϵ}_0]$ and each sufficiently large N,

$$\begin{array}{l}\mathrm{Prob}\left\{\Vert ({\widehat{\beta }}_0,\widehat{\beta },\widehat{t})-({\tilde{\beta }}_0,\tilde{\beta },\tilde{t})\Vert <ϵ\right\}\ge \mathrm{Prob}\left\{\Vert z_N-z_0\Vert <ϵ\right\}\\ \ge 1-{\delta }_0\exp \left\{-N{\mu }_0\right\}-\frac{M_0}{{ϵ}^{2p+1}}\exp \left\{-\frac{N{ϵ}^2}{{\sigma }_0}\right\}.\end{array}$$ (31)

In Theorem 1, L_K is the normal map induced by the linear function $L(\tilde{t})$ in (16) and the critical cone K defined in (21). We use $L_K^{-1}$ to denote its inverse function. Functions ${\mathrm{\Pi }}_K$, L_K and $L_K^{-1}$ are linear if K is a subspace, otherwise they are piecewise linear. Compared to (Lu et al., 2017, Theorem 1) which considers the LASSO penalty, Theorem 1 here handles general penalties that satisfy Assumption 1. The results shown in Theorem 1 are used in the construction of confidence intervals of the population penalized parameter as well as the true model parameter as shown in the following sections.

3.2. Estimators of Σ₀ and L_K

In order to use (29) and (30) to obtain computable confidence regions and intervals, we need to find reliable estimators of Σ₀ and L_K, as we discuss in this subsection. One can show that converges to Σ₀ almost Σ₀ surely under Assumptions 1–4. See the remarks below (26). Therefore, we use Σ_N to estimate Σ₀. Our main task in this subsection is to introduce an estimator of the normal map L_K, knowing that L_K is exactly d(f₀)_S(z₀) (Robinson, 1995), the B-derivative of (f₀)_S at z₀. Let dΠ_S(z) be the B-derivative of the Euclidean projector Π_S at z. Since S is a polyhedral convex set in ${ℝ}^{2p+1},$ Π_S coincides with a different affine function on each (2p + 1)-cell in the normal manifold of S (see Table 1 in the supplementary materials for definitions of the normal manifold and cells). The B-derivative $d{\mathrm{\Pi }}_S(z)$ is a linear function for points z in the interior of each such cell, and is piecewise linear for z on the boundary. Moreover, $d{\mathrm{\Pi }}_S(z)$ is not continuous with respect to z at points z on the boundary of any (2p + 1)-cell. Therefore, the function d(f₀)_S(z) is generally not continuous with respect to z at such points, which can be seen from the chain rule of B-differentiability:

$$d{\left(f_0\right)}_S(z)(h)=L(t)d{\mathrm{\Pi }}_S(z)(h)+h-d{\mathrm{\Pi }}_S(z)(h)\text{for each}z\in {ℝ}^{2p+1},h\in {ℝ}^{2p+1}.$$

If d(f₀)_S(·) is not continuous at z₀, then d(f₀)_S(z_N) is not guaranteed to converge to d(f₀)_S(z₀) even though z_N converges to z₀. To introduce the estimators of L_K, we will consider two cases based on the location of z₀.

For each i = 1, ⋯,p, denote the 9 cells in the normal manifold of S_i as $C_i^j,j=0,1,\cdots ,8$ (see Figure 1). According to (10) we derive the constraints defining each $C_i^j$ which are listed in Table 1 in the supplementary materials. That table also lists the critical cones $K_i^j$ to S_i associated with a point in the relative interior of $C_i^j$. Each (2p + 1)-cell in the normal manifold of S can then be written as $ℝ\times {\mathrm{\Pi }}_{i=1}^p{C}_i^{\gamma (i)},$ where γ(i) = 0, ⋯, 8 for each i = 1, ㏯,p. From (19), Assumption 1(a) and Lemma 2, we notice that ((z₀)_2j, (z₀)_2j+1) can only appear in the relative interior of $C_i^3,C_i^4,C_i^6,C_i^7\text{or}{C}_i^8$ for each i. Consequently, $d{\mathrm{\Pi }}_S(z)$ is not continuous at z₀ if and only if ((z₀)_2i, (z₀)_2i+1) is in the relative interior of $C_i^3\text{or}{C}_i^4$ for some index i. The two cases are defined below, where the first case corresponds to the situation in which the random variable ${(L_K)}^{-1}(\mathcal{N}(0,{\mathrm{\Sigma }}_0))$ is normally distributed, and the second case is for situations in which L_K is a piecewise linear function.

Figure 1:

The normal manifold of S_i (left) and $E_i^0,\cdots ,E_i^8$ (right).

• Case I: In this case, ((z₀)_2i, (z₀)_2i+1 is in the relative interior of $C_i^6,C_i^7\text{or}{C}_i^8$ for all $i\in \{1\cdots p\},$ and the normal map L_K and the B-derivative $d{\mathrm{\Pi }}_S(z_0)$ are linear functions. Since d(f₀)_S(z) is continuous at z₀ in this case, we can use $d{\mathrm{\Pi }}_S(z_N)$ and d(f_N)_S(z_N) as the estimators of $d{\mathrm{\Pi }}_S(z_0)$ and L_K respectively.

• Case II: In this case, ((z₀)_2i, (z₀)_2i+1 is in the relative interior of $C_i^3\text{or}{C}_i^4$ for some index $i\in \{1\cdots p\},$ and L_K and $d{\mathrm{\Pi }}_S(z_0)$ are piecewise linear functions. Since d(f₀)_S(z) is generally not continuous at z₀ in this case, we have to derive an estimator of L_K other than d(f_N)_S(z_N).

In both cases, d(f_N)_S(z_N) is an invertible linear map with high probability (Lu, 2014b, Proposition 3.5). While it is reasonable to expect Case I to occur more often than Case II in practice, one cannot identify Case I in advance since z₀ is unknown. To derive an estimator of L_k, first we give the expression of dΠ_S(z), and then construct an asymptotically exact approximation of $d{\mathrm{\Pi }}_S\left(z_0\right)$. According to (12), we have

$$d{\mathrm{\Pi }}_S(z)(h)=({\check{\beta }}_0,d{\mathrm{\Pi }}_{S_1}({\beta }_1,t_1)({\check{\beta }}_1,{\check{t}}_1),\cdots ,d{\mathrm{\Pi }}_{S_p}({\beta }_p,t_p)({\check{\beta }}_p,{\check{t}}_p)),$$ (32)

for each $z=({\beta }_0,\beta ,t)\text{and}h=({\stackrel{⌣}{\beta }}_0,\check{\beta },\check{t}).$ We denote $d{\mathrm{\Pi }}_{S_i}\left({\beta }_i,t_i\right)$ in the relative interior of each $C_i^j$ by a function ${\psi }_j:{ℝ}^2\to {ℝ}^2(j=0,1,\cdots ,8).$ Define four matrices

$$A_1=\left[\begin{array}{ll}1\hfill & 0\hfill \\ 0\hfill & 1\hfill \end{array}\right],A_2=\left[\begin{array}{rr}\hfill 1/2& \hfill -1/2\\ \hfill -1/2& \hfill 1/2\end{array}\right],A_3=\left[\begin{array}{ll}1/2\hfill & 1/2\hfill \\ 1/2\hfill & 1/2\hfill \end{array}\right],A_4=\left[\begin{array}{ll}0\hfill & 0\hfill \\ 0\hfill & 0\hfill \end{array}\right].$$

Table 2 in the supplementary materials shows the expression of each ${\psi }_j$ using these matrices. Denote the relative interior of $ℝ\times {\mathrm{\Pi }}_{i=1}^p{C}_i^{\gamma (i)}\text{by}\mathrm{ri}\left(ℝ\times {\mathrm{\Pi }}_{i=1}^p{C}_i^{\gamma (i)}\right).$ For all $z\in \mathrm{ri}\left(ℝ\times {\mathrm{\Pi }}_{i=1}^p{C}_i^{\gamma (i)}\right),$ we can write $d{\mathrm{\Pi }}_S(z)$ as

$${\mathrm{\Psi }}_{\mathit{\gamma }(z)}(h)=({\stackrel{⌣}{\beta }}_0,{\psi }_{\gamma (1)}({\stackrel{⌣}{\beta }}_1,{\stackrel{⌣}{t}}_1\mathrm{)},\cdots ,{\psi }_{\gamma (p)}({\stackrel{⌣}{\beta }}_p,{\check{t}}_p\mathrm{)}\mathrm{)}\text{for each}h=({\stackrel{⌣}{\beta }}_0,\stackrel{⌣}{\beta },\check{t}\mathrm{)},$$ (33)

where $\mathit{\gamma }(z)=(\gamma (1),\cdots ,\gamma (p))$ such that $z\in \mathrm{ri}\left(ℝ\times {\mathrm{\Pi }}_{i=1}^p{C}_i^{\gamma (i)}\right).$

Table 2:

Coverage rates and average lengths of 95% individual CIs for population penalized parameters $({\tilde{\beta }}_0,\tilde{\beta })$ for different MCP penalties from 500 replications with sample size N = 300 generated in Example 1.

	a = 2									a = 2000
	λ = 0.5			λ = 1			λ = 2			λ = 0.5			λ = 1			λ = 2
	$\tilde{\beta }$	CR	Len	$\tilde{\beta }$	CR	Len	$\tilde{\beta }$	CR	Len	$\tilde{\beta }$	CR	Len	$\tilde{\beta }$	CR	Len	$\tilde{\beta }$	CR	Len
β0	0	0.99	0.26	0	0.99	0.27	0	0.98	0.39	0	0.98	0.28	0	0.98	0.32	0	0.98	0.46
β1	3	0.97	0.30	3.13	0.97	0.36	3.37	0.95	0.88	2.83	0.97	0.32	2.67	0.98	0.38	2.33	0.96	0.56
β2	1.5	0.97	0.30	1.25	0.97	0.49	0.51	0.96	0.84	1.36	0.98	0.33	1.22	0.98	0.38	0.94	0.98	0.56
β3	0	1.00	0.00	0	1.00	0.00	0	1.00	0.00	0	1.00	0.04	0	1.00	0.01	0	1.00	0.00
β4	0	1.00	0.00	0	1.00	0.00	0	1.00	0.00	0	1.00	0.04	0	1.00	0.01	0	1.00	0.00
β5	2	0.98	0.26	2.02	0.98	0.30	1.47	0.98	0.56	1.78	0.99	0.29	1.56	0.98	0.34	1.11	0.98	0.52
β6	0	1.00	0.00	0	1.00	0.00	0	1.00	0.00	0	1.00	0.02	0	1.00	0.00	0	1.00	0.00
β7	0	1.00	0.00	0	1.00	0.00	0	1.00	0.00	0	1.00	0.01	0	1.00	0.00	0	1.00	0.00
β8	0	1.00	0.00	0	1.00	0.00	0	1.00	0.00	0	1.00	0.00	0	1.00	0.00	0	1.00	0.00

Next, we construct an estimator of $d{\mathrm{\Pi }}_S\left(z_0\right).$ We divide the plane (β_i, t_i) into 9 pieces $E_i^0,\cdots ,E_i^8$ (see Figure 1). The constraints that define each of these sets $E_i^0,\cdots ,E_i^8$ are listed in Table 3 in the supplementary materials. The function g(N) in that table can be any combination of finite many terms of the form aN^b with a > 0 and b ∈ (0, 1/2), among other choices. For more details, see Lu and Budhiraja (2013). Each partition $ℝ\times {\mathrm{\Pi }}_{i=1}^p{E}_i^{\gamma (i)}$ is related to the $(2p+1)\text{-cell}ℝ\times {\mathrm{\Pi }}_{i=1}^p{C}_i^{\gamma (i)}.$ Let

$$\widehat{\mathit{\gamma }}(z)=(\gamma (1),\cdots ,\gamma (p))\text{such that}z\in ℝ\times {\mathrm{\Pi }}_{i=1}^p{E}_i^{\gamma (i)}.$$

Given a sample size N and a fixed z, we define a function ${\mathrm{\Lambda }}_N(z):{ℝ}^{2p+1}\to {ℝ}^{2p+1}$ as

$${\mathrm{\Lambda }}_N(z)(h)={\mathrm{\Psi }}_{\hat{\mathit{\gamma }}(z)}(h),\text{for each}h\in {ℝ}^{2p+1}.$$ (34)

According to Theorem 3.1 of Lu (2014a), ${\mathrm{\Lambda }}_N(z_N)$ converges to $d{\mathrm{\Pi }}_S(z_0)$ in probability under Assumptions 1–4.

Table 3:

Coverage rates and average lengths of 95% individual CIs for true model parameters (${\beta }_0^{true},{\beta }^{true}$) for different MCP penalties from 500 replications with sample size N = 300 generated in Example 1.

		a = 2						a = 2000
		λ = 0.5		λ = 1		λ = 2		λ = 0.5		λ = 1		λ = 2
	True	CR	Len	CR	Len	CR	Len	CR	Len	CR	Len	CR	Len
${\beta }_0^{true}$	0	0.96	0.23	0.96	0.23	0.95	0.34	0.96	0.24	0.95	0.28	0.95	0.40
${\beta }_1^{true}$	3	0.95	0.26	0.96	0.27	0.99	0.42	0.96	0.28	0.99	0.33	1.00	0.49
${\beta }_2^{true}$	1.5	0.95	0.29	0.96	0.30	1.00	0.51	0.96	0.31	0.97	0.36	1.00	0.53
${\beta }_3^{true}$	0	0.95	0.28	0.96	0.29	0.97	0.42	0.97	0.30	0.97	0.35	0.98	0.50
${\beta }_4^{true}$	0	0.97	0.28	0.97	0.29	0.97	0.42	0.96	0.30	0.97	0.35	0.98	0.50
${\beta }_5^{true}$	2	0.96	0.28	0.96	0.29	0.99	0.44	0.97	0.30	1.00	0.36	1.00	0.54
${\beta }_6^{true}$	0	0.95	0.28	0.95	0.28	0.98	0.42	0.95	0.30	0.97	0.34	0.98	0.49
${\beta }_7^{true}$	0	0.96	0.28	0.96	0.29	0.98	0.42	0.96	0.30	0.97	0.35	0.99	0.50
${\beta }_8^{true}$	0	0.96	0.25	0.97	0.26	0.99	0.38	0.97	0.27	0.99	0.32	1.00	0.45

Based on (24), (26) and (34), we define a function ${\mathrm{\Phi }}_N(z_N):{ℝ}^{2p+1}\to {ℝ}^{2p+1}$ as

$${\mathrm{\Phi }}_N(z_N)(h)=L_N(\widehat{t}){\mathrm{\Lambda }}_N(z_N)(h)+h-{\mathrm{\Lambda }}_N(z_N)(h)$$ (35)

for each $h\in {ℝ}^{2p+1}.$ The following theorem shows that d(f_N)_S(z_N) is a consistent estimator of L_k for Case I, and ${\mathrm{\Phi }}_N(z_N)$ is a consistent estimator of L_K for both Case I and Case II.

Theorem 2.

• (a)
  Suppose that Assumptions 1, 2 and 3 hold. If z₀ satisfies the conditions for Case I, then $d{\mathrm{\Pi }}_S(z_N)$ defined in (33) converges to $d{\mathrm{\Pi }}_S(z_0)$ almost surely, and

  $$d{(f_N)}_S(z_N)=L_N(\widehat{t})d{\mathrm{\Pi }}_S(z_N)+I-d{\mathrm{\Pi }}_S(z_N)$$ (36)

  converges to L_K almost surely.
• (b)Suppose that Assumptions 1, 2, 3 and 4(a-b) hold. Then ${\mathrm{\Phi }}_N(z_N)$ converges to L_K in probability.

The two functions d(f_N)_S(z_n) and ${\mathrm{\Phi }}_N(z_N)$ are generally different when ((z_N)_2i, (z_N)_2i+1) belongs to $E_i^0,E_i^1,E_i^2,E_i^3\text{or}{E}_i^4$ for some i, in which case ${\mathrm{\Phi }}_N(z_N)$ is a piecewise linear function. In contrast, d(f_N)_S(z_N) is a piecewise linear function only when ((z_N)_2i, (z_N)_2i+1 belongs to $C_i^3\text{or}{C}_i^4$ for some i.

Under Assumptions 1–4, we can show that the weak convergence in (30) still holds after L_K is substituted by ${\mathrm{\Phi }}_N(z_N)$. Consequently, if ${\mathrm{\Sigma }}_0^1$ is nonsingular, then we have

$$\sqrt{N}\left[\begin{array}{cc}{({\mathrm{\Sigma }}_N^1)}^{-1/2}& 0\\ 0& I_p\end{array}\right]({\mathrm{\Phi }}_N(z_N))(z_N-z_0)\Rightarrow \mathcal{N}(0,I_{p+1})\times 0.$$ (37)

If ${\mathrm{\Sigma }}_0^1$ is singular, we decompose ${\mathrm{\Sigma }}_N^1\text{as}{\mathrm{\Sigma }}_N^1=U_N^T{\mathrm{\Delta }}_N{U}_N\text{where}{U}_N$ is an orthogonal (p + 1) × (p + 1) matrix, and Δ_N is a diagonal matrix with monotonically decreasing elements. Let l be the number of positive eigenvalues of ${\mathrm{\Sigma }}_0^1$ counted with regard to their algebraic multiplicities, let D_n be the upper-left submatrix of Δ_N whose diagonal elements are at least 1/g(N), and let l_N be the number of rows in D_N. Furthermore, let (U_N)₁ be the submatrix of U_N that consists of its first l_N rows, and submatrix (U_N)₂ consists of the remaining rows of U_N. We present the weak convergence results in the following theorem, which generalizes Theorem 3 in Lu et al. (2017) to cover all penalties satisfying Assumption 1.

Theorem 3.

Suppose that Assumptions 1, 2, 3 and 4(a-b) hold. Then

$$\sqrt{N}{\mathrm{\Phi }}_N(z_N)(z_N-z_0)\Rightarrow \mathcal{N}(0,{\mathrm{\Sigma }}_0).$$ (38)

If ${\mathrm{\Sigma }}_0^1$ is nonsingular, then

$$N{\left[({\mathrm{\Phi }}_N(z_N))(z_N-z_0)\right]}^T\left[\begin{array}{cc}{({\mathrm{\Sigma }}_N^1)}^{-1}& 0\\ 0& I_p\end{array}\right]\left[({\mathrm{\Phi }}_N(z_N))(z_N-z_0)\right]\Rightarrow {\chi }_{p+1}^2,$$ (39)

and

$$N{\left[({\mathrm{\Phi }}_N(z_N))(z_N-z_0)\right]}^T\left[\begin{array}{cc}0& 0\\ 0& I_p\end{array}\right]\left[({\mathrm{\Phi }}_N(z_N))(z_N-z_0)\right]\Rightarrow 0.$$ (40)

If ${\mathrm{\Sigma }}_0^1$ is singular and Assumption 4(c) holds, then Prob{l_N = l} → 1 as N → ∞,

$$N{\left[({\mathrm{\Phi }}_N(z_N))(z_N-z_0)\right]}^T\left[\begin{array}{cc}{(U_N)}_1^T{D}_N^{-1}{(U_N)}_1& 0\\ 0& 0\end{array}\right]\left[({\mathrm{\Phi }}_N(z_N))(z_N-z_0)\right]\Rightarrow {\chi }_l^2,$$ (41)

and

$$N{\left[({\mathrm{\Phi }}_N(z_N))(z_N-z_0)\right]}^T\left[\begin{array}{cc}0& 0\\ 0& I_p\end{array}\right]\left[({\mathrm{\Phi }}_N(z_N))(z_N-z_0)\right]\Rightarrow 0.$$ (42)

We can treat (39) and (40) as a special case of (41) and (42). In fact, if z₀ satisfies Case I, then Theorem 3 still holds if ${\mathrm{\Phi }}_N(z_N)$ is replaced by d(f_N)_S(z_n).

3.3. Confidence intervals for the population penalized parameters

In this subsection, we describe how to obtain confidence interval for $({\tilde{\beta }}_0,\tilde{\beta })$ from the asymptotic distribution of z_N. First, we investigate the relationship between a solution to the normal map formulation (18) and the corresponding solution to (1). Let $\tilde{q}$ be as defined in Assumption 3 and ${\tilde{q}}_0=E[-2(Y-{\tilde{\beta }}_0-\sum _{i=1}^p{\tilde{\beta }}_i{X}_i)].$ From (13), (14) and (19), we have $z_0=({\tilde{\beta }}_0,\tilde{\beta },\tilde{t})-({\tilde{q}}_0,\tilde{q}-2m\tilde{\beta },{(P_{{\lambda }_i}^{\prime }({\tilde{t}}_i)+2m{\tilde{t}}_i)}_{i=1}^p).$ In the supplementary materials, it is shown in (B.1) that ${\tilde{q}}_0=0,$ which implies ${\tilde{\beta }}_0={\left(z_0\right)}_1.$ Thus, confidence intervals of ${\tilde{\beta }}_0$ are exactly those of ${(z_0)}_1.$ On the other hand, using the fact $({\tilde{\beta }}_i,{\tilde{t}}_i)={\mathrm{\Pi }}_{S_i}({(z_0)}_{2i},{(z_0)}_{2i+1})$ for each i = 1, ⋯,p, we have the following relationship between ${\tilde{\beta }}_i$ and ((z₀)_2i, (z₀)_2i+1):

$${\tilde{\beta }}_i=\mathrm{\Gamma }\left(V_{+},V_{-}\right)=\{\begin{array}{cc}\frac{1}{2}{V}_{+},& \text{if}{V}_{+}>0\text{and}{V}_{-}⩾0,\\ 0,& \text{if}{V}_{+}⩽0\text{and}{V}_{-}⩾0,\\ \frac{1}{2}{V}_{-},& \text{if}{V}_{+}⩽0\text{and}{V}_{-}<0,\end{array}$$ (43)

where V₊ = (z₀)_2i + (z₀)_2i+1 and $V_{-}={(z_0)}_{2i}-{(z_0)}_{2i+1}.$ The above three cases in (43) include all the possible situations for the location of ((z₀)_2i, (z₀)_2i+1. This map $\mathrm{\Gamma }$ can be used to obtain confidence intervals for ${\tilde{\beta }}_i(i=1,\cdots ,p)$ after we calculate confidence intervals for ((z₀)_2i + (z₀)_2i+1) and ((z₀)_2i − (z₀)_2i+1). For a fixed i, we denote the (1 − α/2)% confidence intervals for ((z₀)_2i + (z₀)_2i+1) and ((z₀)_2i − (z₀)_2i+1) as $[L_{+}^i,U_{+}^i]\text{and}[L_{-}^i,U_{-}^i]$ respectively. Then a (1 − α)% (conservative) confidence interval for ${\tilde{\beta }}_i$ is given by

$$[\mathrm{\Gamma }(L_{+}^i,L_{-}^i),\mathrm{\Gamma }(U_{+}^i,U_{-}^i)].$$ (44)

Next, we show how to find confidence intervals for (z₀)₁, ((z₀)_2i + (z₀)_2i+1) and ((z₀)_2i − (z₀)_2i+1). Under Assumptions 1–4, from Theorem 3 we can express the asymptotically exact (1 − α)100% confidence region for z₀ as

$$\left\{z\in {ℝ}^{2p+1}|\begin{array}{c}N{\left[{\mathrm{\Phi }}_N(z_N)(z_N-z)\right]}^T\left[\begin{array}{cc}{(U_N)}_1^T{D}_N^{-1}{(U_N)}_1& 0\\ 0& 0\end{array}\right]\left[{\mathrm{\Phi }}_N(z_N)(z_N-z)\right]⩽{\chi }_{l_N}^2(\alpha )\\ N{\left[{\mathrm{\Phi }}_N(z_N)(z_N-z)\right]}^T\left[\begin{array}{cc}0& 0\\ 0& I_p\end{array}\right]\left[{\mathrm{\Phi }}_N(z_N)(z_N-z)\right]=0\end{array}\right\}$$ (45)

where ${\chi }_{l_N}^2(\alpha )$ is the critical value associated with significant level α of a χ² distribution with l_N degrees of freedom. If ${\mathrm{\Phi }}_N(z_N)$ is a linear map, then the set in (45) is an ellipsoid in a subspace of ${ℝ}^{2p+1}.$ Otherwise it is the union of different ellipsoid fractions. To obtain simultaneous confidence intervals, we find the maximal and minimal values of (z₀)₁, ((z₀)_2i + (z₀)_2i+1) and ((z₀)_2i − (z₀)_2i+1) in the set of (45) by solving optimization problems.

For individual confidence intervals, first we notice that ${\mathrm{\Phi }}_N(z_N)$ is a global homeomorphism with probability 1 as $N\to \infty$ (see the proof of Theorem 2 in the supplementary materials). If ${\mathrm{\Phi }}_N(z_N)$ is a global homeomorphism, we can use

$${({\mathrm{\Phi }}_N(z_N))}^{-1}(\mathcal{N}(0,{\mathrm{\Sigma }}_N))$$ (46)

to approximate the distribution of $\sqrt{N}\left(z_N-z_0\right)$ as in (29). When ${\mathrm{\Phi }}_N\left(z_N\right)$ is a linear map, the distribution in (46) is normal. Therefore (z₀)₁, ((z₀)_2i + (z₀)_2i+1) and ((z₀)_2i − (z₀)_2i+1) also follow normal distributions, from which we can construct individual confidence intervals. When ${\mathrm{\Phi }}_N\left(z_N\right)$ is not a linear map, we simulate data based on the distribution in (46), and find empirical individual confidence intervals for (z₀)₁, ((z₀)_2i + (z₀)_2i+1) and ((z₀)_2i − (z₀)_2i+1) by taking $\frac{\alpha }{2}\%\text{and}\left(1-\frac{\alpha }{2}\right)\%$ percentiles of the data as the lower and upper bounds respectively.

3.4. Confidence intervals for true model parameters in the underlying linear model

In this subsection, we develop a method to compute confidence intervals for $({\beta }_0^{true},{\beta }^{true})$, based on a relation between a population penalized parameter $({\tilde{\beta }}_0,\tilde{\beta })$ and the true model parameter $({\beta }_0^{true},{\beta }^{true})$. Suppose the underlying linear model is

$$Y={\beta }_0^{true}+X^T{\beta }^{true}+\epsilon ,$$ (47)

where ${\beta }_0^{true}\in ℝ\text{and}{\beta }^{true}=\left({\beta }_1^{true},\cdots ,{\beta }_p^{true}\right)\in {ℝ}^p$ are the true model parameters. Let $t^{true}\in {ℝ}^p\text{be defined as}{t}_i^{true}=|{\beta }_i^{true}|.$ Denote the covariance matrix of X as Σ, and assume the random error ε has mean zero and variance σ². Moreover, ε is independent with X_i for all i = 1, ⋯,p. For simplicity we assume E(X_i) = 0 for each i = 1, ⋯,p. Consequently we have $E(Y)={\beta }_0^{true}\text{and}\mathrm{\Sigma }=E(X{X}^T)$. We assume that £ is nonsingular and therefore we do not need Assumption 3 in this subsection.

In developing theoretical results of this subsection, we will let λ = (λ_i, ⋯, λ_p) converge to 0. Due to this change, assumptions stated in Section 2.2 need to be changed accordingly. We will replace Assumptions 1 by Assumption 1’, and keep Assumption 2. We will not need Assumption 4 until the end of this section.

Assumption 1’(a).

For each i = 1, 2, ⋯,p, P₀(t) = 0 for all t ≥ 0. Moreover, for each positive λ_i in a neighborhood of 0, P_λi(·) is nonnegative, nondecreasing and continuously differentiable on $[0,+\infty )\mathit{\text{with}}{P}_{{\lambda }_i}(0)=0\mathit{\text{and}}{P}_{{\lambda }_i}^{\prime }(0)>0.$

Assumption 1’(b).

For each i = 1, ⋯,p, there exist neighborhoods ${\mathcal{T}}_{i}of{t}_i^{true}in{ℝ}_{+}and{\mathrm{\Lambda }}_{i}of0in{ℝ}_{+},suchthat{P}_{{\lambda }_i}^{″}(t_i)$ the second derivative of (·) with respect to t_i, exists for each ${\lambda }_i\in {\mathrm{\Lambda }}_{i}and{t}_i\in {\mathcal{T}}_i.$ Moreover, $P_{{\lambda }_i}^{\prime }(t_i)and{P}_{{\lambda }_i}^{″}(t_i)$ are Lipschitz continuous in $\left({\lambda }_i,t_i\right)on{\mathfrak{T}}_i\times {\mathrm{\Lambda }}_i.$

Assumption 1’(c).

For each i = 1, ⋯,p, there exists a neighborhood ${\mathcal{J}}_{i}of{t}_i^{true}in{ℝ}_{+},$ such that the mixed partial derivatives $\frac{{\partial }^{2}P}{\partial {\lambda }_i\partial t_i}\left(0,t_i\right)and\frac{{\partial }^{3}P}{\partial {\lambda }_i{\partial }^2{t}_i}\left(0,t_i\right)existforeach{t}_i\in {\mathcal{T}}_i,$ with

$$\underset{{\lambda }_i\to 0}{\lim }\underset{t_i\in {\mathcal{T}}_i}{\sup }|\frac{P_{{\lambda }_i}^{\prime }\left(t_i\right)}{{\lambda }_i}-\frac{{\partial }^{2}P}{\partial {\lambda }_i\partial t_i}\left(0,t_i\right)|+|\frac{P_{{\lambda }_i}^{\prime \prime }\left(t_i\right)}{{\lambda }_i}-\frac{{\partial }^{3}P}{\partial {\lambda }_i{\partial }^2{t}_i}\left(0,t_i\right)|=0.$$

Besides the convex LASSO penalty, we can check that many non-convex penalty functions such as SCAD, MCP, the log-penalty, and the transformed ${\mathcal{l}}_1$ penalty satisfy Assumption 1’(a). We can further check that the LASSO penalty, the log-penalty, and the transformed ${\mathcal{l}}_1$ penalty also satisfy Assumptions 1’(b) and 1’(c). For the SCAD and MCP penalties, Assumptions 1’(b) and 1’(c) are satisfied almost everywhere except that $t_i^{true}=0$ for some i. Assumptions 1’(b) and 1’(c) are used to guarantee that the SAA function f_N almost surely converges to the true function f₀ in the space of continuously differentiable functions on a neighborhood of $({\beta }_0^{true},{\beta }^{true},t^{true}),andthat\sqrt{N}(f_N-f_0)$ weakly converges to a random function in that space. This assumption is needed for the techniques based on stochastic variational inequalities to be applicable. It is possible to weaken this assumption by developing techniques for a broader class of problems in which the SAA function f_N (or equivalently, the first order derivative of the penalty function) is not necessarily continuously differentiable, and we will investigate this in future work.

Under Assumption 1’(a), with λ = 0 the problem (1) becomes the least square problem (3), which has a unique solution $({\beta }_0^{true},{\beta }^{true})$ in view of the linear model (47). Let $t^{true}\in {ℝ}^p$ be defined as if $t_i^{true}=|{\beta }_i^{true}|.$ Then (9) with λ = 0 has a unique solution $({\beta }_0^{true},{\beta }^{true},t^{true}).$ By the equivalence between (9) and (17), $({\beta }_0^{true},{\beta }^{true},t^{true})$ is also the unique solution to

$$-f_0\left({\beta }_0,\beta ,t\right)\in N_S\left({\beta }_0,\beta ,t\right)$$

where f₀(β₀, β, t) is as defined in (14) but with λ = 0:

$$f_0\left({\beta }_0,\beta ,t\right)=\left[\begin{array}{c}-2E[Y-{\beta }_0-\sum _{i=1}^p{\beta }_i{X}_i]\\ -2E[(Y-{\beta }_0-\sum _{i=1}^p{\beta }_i{X}_i)X]-2m\beta \\ 2mt\end{array}\right].$$

Let $z_0^{*}$ be as defined in (19) with λ = 0 and $({\beta }_0^{true},{\beta }^{true},t^{true})\text{replacing}({\tilde{\beta }}_0,\tilde{\beta },\tilde{t}).$ The following lemma presents the relation between $({\beta }_0^{true},{\beta }^{true})\text{and}{z}_0^{*}.$

Lemma 4.

Suppose that Assumptions 1’(a) and 2 hold. Then we have

$$({\beta }_0^{true},{\beta }^{true})=G^{*}(z_0^{*})$$ (48)

where $G^{*}\in {ℝ}^{(p+1)\times (2p+1)}$ is defined as

$$G^{*}=\left[\begin{array}{ccc}1& 0& 0\\ 0& {(1+2m)}^{-1}{I}_p& 0\end{array}\right].$$ (49)

Lemma 4 above indicates that an estimator of the true parameter $({\beta }_0^{true},{\beta }^{true})\text{is}{G}^{*}(z_N)$ where z_N is defined in (26). In the following theorem, we show the asymptotic distribution of the estimator $G^{*}(z_N).$ Before stating this theorem, we define a matrix $L^{*}\in {ℝ}^{(2p+1)\times (2p+1)}\text{and a cone}{K}^{*}\subset {ℝ}^{2p+1}$ as follows:

$$L^{*}=d{f}_0({\beta }_0^{true},{\beta }^{true},t^{true})=\left[\begin{array}{ccc}2& 0& 0\\ 0& 2\mathrm{\Sigma }-2m{I}_p& 0\\ 0& 0& 2m{I}_p\end{array}\right],$$ (50)

and

$$K^{*}=\{w\in T_S({\beta }_0^{true},{\beta }^{true},t^{true})|\langle f_0({\beta }_0^{true},{\beta }^{true},t^{true}),w\rangle =0\}.$$ (51)

Note that

$$f_0({\beta }_0^{true},{\beta }^{true},t^{true})=\left[\begin{array}{c}0\\ -2m{\beta }^{true}\\ 2m{t}^{true}\end{array}\right].$$

This implies that

$$K^{*}=ℝ\times {\mathrm{\Pi }}_{i=1}^p{K}_i^{*}$$ (52)

where

$$K_i^{*}=\{\begin{array}{ll}\{\left({\beta }_i,t_i\right)|{\beta }_i=t_i)\}\hfill & \text{if}{\beta }_i^{\mathit{\text{true}}}>0,\hfill \\ \{\left({\beta }_i,t_i\right)|{\beta }_i=-t_i)\}\hfill & \text{if}{\beta }_i^{\mathit{\text{true}}}<0,\hfill \\ S_i\hfill & \text{if}{\beta }_i^{\mathit{\text{true}}}=0.\hfill \end{array}$$ (53)

As the setting considered in this subsection is different from previous sections, L* and K* here are different from L and K defined in Assumption 3 and (21). The previous L and K are associated with $({\tilde{\beta }}_0,\tilde{\beta },\tilde{t}),$ a solution to the population problem with a fixed positive λ, while L* and K* are associated with $({\beta }_0^{true},{\beta }^{true},t^{true})$ and λ = 0.

Theorem 4.

Suppose that Assumptions 1’(a-c) and 2 hold. Let m > 0 be sufficiently small so that $\mathrm{\Sigma }-m{I}_p$ is nonsingular, L* and K* be defined as above, ${\mathrm{\Sigma }}_0^{*}$ be the covariance matrix of the random vector $F({\beta }_0^{true},{\beta }^{true},t^{true},X,Y)$ defined in (13), and ${\mathrm{\Sigma }}_0^{*1}$ be the upper left (p + 1) × (p + 1) submatrix of ${\mathrm{\Sigma }}_0^{*}$. Moreover, let λ_i’s be chosen to satisfy ${\lim }_{N\to \infty }\sqrt{N}{\lambda }_i=c_i$ for some constant c_i ≥ 0, z_N be defined in (26), and define $({\widehat{\beta }}_0^{true},{\widehat{\beta }}^{true})=G^{*}(z_N)andh\in {ℝ}^{p}by{h}_i=c_i\frac{{\partial }^{2}P}{\partial {\lambda }_i\partial t_i}(0,t_i^{true}).Then({\widehat{\beta }}_0^{true},{\widehat{\beta }}^{true})$ is a consistent estimator of $({\beta }_0^{true},{\beta }^{true})$ and

$$\sqrt{N}\left(({\widehat{\beta }}_0^{true},{\widehat{\beta }}^{true})-({\beta }_0^{true},{\beta }^{true})\right)\Rightarrow G^{*}\circ {(L_{K^{*}}^{*})}^{-1}(\mathcal{N}(0,{\mathrm{\Sigma }}_0^{*1}),h).$$ (54)

Note that the distribution of $G^{*}°{(L_{K^{*}}^{*})}^{-1}(\mathcal{N}(0,{\mathrm{\Sigma }}_0^{*1}),h)$ can be normal or non-normal. When the true parameter ${\beta }_i^{true}\ne 0$ for each i, we can show that K* is a subspace of ${ℝ}^{(2p+1)}\mathrm{and}{(L_{K^{*}}^{*})}^{-1}$ is a linear function. Therefore, the limiting distribution of the true parameter estimator G*(z_N) is normal in this case. However, if the true parameter ${\beta }_i^{true}=0$ for some i, the limiting distribution can be normal or non-normal.

Theorem 5.

Suppose that the assumptions in Theorem 4 hold. If h_i = 0 for each i, then $({\widehat{\beta }}_0^{\mathit{\text{true}}},{\widehat{\beta }}^{\mathit{\text{true}}})$ is a consistent estimator of $({\beta }_0^{true},{\beta }^{true})$ and

$$G^{*}\circ {(L_{K^{*}}^{*})}^{-1}(\mathcal{N}(0,{\mathrm{\Sigma }}_0^{*1}),h)=\left[\begin{array}{cc}\frac{1}{2}& 0\\ 0& \frac{1}{2}{\mathrm{\Sigma }}^{-1}\end{array}\right](\mathcal{N}(0,{\mathrm{\Sigma }}_0^{*1}))=\mathcal{N}(0,\left[\begin{array}{cc}{\sigma }^2& 0\\ 0& {\sigma }^2{\mathrm{\Sigma }}^{-1}\end{array}\right]).$$

Furthermore, let $\widehat{\mathrm{\Theta }}$ be a consistent estimate of ${\mathrm{\Sigma }}^{-1}.$ Define $H_N=\left[\begin{array}{cc}\frac{1}{2}& 0\\ 0& \frac{1}{2}\widehat{\mathrm{\Theta }}\end{array}\right].$ Then,

$$\frac{\sqrt{N}({\widehat{\beta }}_i^{true}-{\beta }_i^{true})}{\sqrt{{(H_N{\mathrm{\Sigma }}_N^1{H}_N^T)}_{i+1,i+1}}}\Rightarrow \mathcal{N}(0,1),foralli=0,1,\cdots ,p.$$ (55)

Since $h_i=c_i\frac{{\partial }^{2}P}{\partial {\lambda }_i\partial t_i}(0,t_i^{true}),$ we know that $h_i=0\text{if}{c}_i=0$. Therefore, if λ_i’s are chosen to be $o(1/\sqrt{N})$ in the penalty function, the limiting distribution will be a multivariate normal distribution. In this normal case, the above Theorem 5 provides a method to compute asymptotically exact individual confidence intervals for $({\beta }_0^{true},{\beta }^{true})$.

When $h_i\ne 0$ for some i, the asymptotic distribution in Theorem 4 does not necessarily reduce to a normal distribution. To see this, consider the following example. Let p = 2, ${\beta }_0^{true}=0,{\beta }_1^{true}=0\text{and}{\beta }_2^{true}=a_0$ for some constant $a_0>0.\text{Let}\lambda =\frac{1}{\sqrt{N}},\text{so}{c}_1=c_2=1.\text{Let}\mathrm{\Sigma }=\left[\begin{array}{ll}2\hfill & 0\hfill \\ 0\hfill & 2\hfill \end{array}\right],m=1,\text{and}\frac{{\partial }^{2}P}{\partial {\lambda }_1\partial t_1}(0,t_1^{true})=\frac{{\partial }^{2}P}{\partial {\lambda }_2\partial t_2}(0,t_2^{true})=1$ (which is satisfied by the LASSO penalty function P(λ, t) = λt). It follows that h₁ = h₂ = 1. Let q₀, q₁, q₂ ∈ $\mathbb{R}$. To find (L*_K*)⁻¹(q₀, q₁, q₂, h₁, h₂) we consider the following problem

$$\underset{\left({\beta }_0,{\beta }_1,t_1,{\beta }_2,t_2\right)\in K^{*}}{\min }{\beta }_0^2+{\beta }_1^2+t_1^2+{\beta }_2^2+t_2^2-q_0{\beta }_0-q_1{\beta }_1-h_1{t}_1-q_2{\beta }_2-h_2{t}_2,$$

whose solution satisfies $\left(q_0,q_1,q_2,h_1,h_2\right)\in L^{*}\left({\beta }_0,{\beta }_1,{\beta }_2,t_1,t_2\right)+N_{K^{*}}\left({\beta }_0,{\beta }_1,{\beta }_2,t_1,t_2\right).$ Here $K^{*}=ℝ\times S_1\times \{({\beta }_2,t_2)|{\beta }_2=t_2\}.$ The solution to the above problem is given by

$${\beta }_0=\frac{q_0}{2},{\beta }_1=\{\begin{array}{ll}\frac{q_1}{2}\hfill & \text{if}-1\le q_1\le 1\hfill \\ \frac{q_1+1}{4}\hfill & \text{if}{q}_1\ge 1\hfill \\ \frac{q_1-1}{4}\hfill & \text{if}{q}_1\le -1\hfill \end{array},t_1=|{\beta }_1|,{\beta }_2=t_2=\frac{q_2+1}{4}.$$

As a result, ${(L_{K^{*}}^{*})}^{-1}\left(q_0,q_1,q_2,h_1,h_2\right)=\left({\beta }_0,{\beta }_1,{\beta }_2,t_1,t_2\right)$ is a piecewise affine function of (q₀,q₁,q₂,h₁,h₂) with three pieces. Furthermore, since G*(·) is a linear transformation, we conclude that the asymptotic distribution is non-normal in this case.

Next, we show how to estimate ${(L_{K^{*}}^{*})}^{-1}$ in the situation considered in Theorem 4. We need to make an assumption analogous to Assumption 4.

Assumption 4’(a).

The same conditions as in Assumption 4(a), with $({\beta }_0^{true},{\beta }^{true},t^{true})replacing({\tilde{\beta }}_0,\tilde{\beta },\tilde{t}).$

Assumption 4’(b).

The same conditions as in Assumption 4(b), with $({\beta }_0^{true},{\beta }^{true},t^{true})replacing({\tilde{\beta }}_0,\tilde{\beta },\tilde{t}).$

To estimate ${(L_{K^{*}}^{*})}^{-1},$ we can also use ${\mathrm{\Phi }}_N(z_N).$ Under Assumption 4’, to show that ${\mathrm{\Phi }}_N(z_N)$ is a consistent estimator of ${(L_{K^{*}}^{*})}^{-1},$ the key is to show (31) holds (with $({\beta }_0^{true},{\beta }^{true},t^{true})\mathrm{replacing}({\tilde{\beta }}_0,\tilde{\beta },\tilde{t})).$ To show this, one needs to show that f_N converges to f₀ in probability at an exponential rate in the space of continuous differentiable functions. For the first p + 1 components, this follows from Assumption 4’. For the last p components, note that the norm of the function $P_{{\lambda }_i}^{\prime }/{\lambda }_i$ in the space of continuous differentiable functions is bounded due to Assumption 1’(c). In addition, since ${\lim }_{N\to \infty }\sqrt{N}{\lambda }_i=c_i$ for each i, we have $\sqrt{N}\Vert P_{{\lambda }_i}^{\prime }\Vert \le \rho$ for some constant ρ. As a result, for each $ϵ>0,$ we have $\Vert P_{{\lambda }_i}^{\prime }\Vert \le \rho /\sqrt{N}<ϵ/2$ for sufficiently large N. Therefore, f_N converges to f₀ in probability at an exponential rate in the space of continuous differentiable functions, and ${\mathrm{\Phi }}_N(z_N).$ converges to ${(L_{K^{*}}^{*})}^{-1}$ in probability.

For the normal case, equation (55) justifies using the diagonal elements of $H_N{\mathrm{\Sigma }}_N{H}_N^T,$ divided by N, as the estimated variances of $({\beta }_0^{true},{\beta }^{true}).$. For the non-normal case, first we define two functions R and $\widehat{R}\text{from}{ℝ}^{2p+1}\text{to}{ℝ}^{p+1}$ as

$$R=G^{*}\circ {(L_{K^{*}}^{*})}^{-1}\left[\begin{array}{cc}{({\mathrm{\Sigma }}_0^{*1})}^{\frac{1}{2}}& 0\\ 0& \mathrm{diag}{(h_i)}_{i=1}^p\end{array}\right]\text{and}\widehat{R}=G^{*}\circ {({\mathrm{\Phi }}_N(z_N))}^{-1}\left[\begin{array}{cc}{({\mathrm{\Sigma }}_N^1)}^{\frac{1}{2}}& 0\\ 0& \mathrm{diag}{({\widehat{h}}_i)}_{i=1}^p\end{array}\right],$$ (56)

where ${\widehat{h}}_i=\sqrt{N}{\lambda }_i\frac{{\partial }^{2}P}{\partial {\lambda }_i\partial t_i}(0,|{\widehat{\beta }}_i^{true}|).$ Denote the i^th component function of R and $\widehat{R}\text{as}{R}_i\text{and}{\widehat{R}}_i$ respectively for each i. Let $f:{ℝ}^{2p+1}\to ℝ$ be a continuous function and Z be a (2p + 1)-dimensional random variable with $Z~\mathcal{N}(0,I_{p+1})\times \overrightarrow{\mathbf{1}}.\text{Define}{a}^r(f)\in (0,\infty )$ as

$$a^r(f)=\inf \{c⩾0|\text{Prob}\{-c⩽f(Z)-r⩽c\}⩾1-\alpha \}.$$ (57)

Suppose that Prob {f(Z) = b} = 0 for all $b\in ℝ.$ Then for any given $r\in ℝ\text{and}\alpha \in (0,1),a^r(f)$ as defined in (57) is the smallest value that satisfies

$$\mathrm{Prob}\{-a^r(f)⩽f(Z)-r⩽a^r(f)\}=1-\alpha .$$

Since the map G* has full row rank, $L_{K^{*}}^{*}$ is a global homeomorphism, and ${\mathrm{\Sigma }}_0^{*1}$ is nonsingular. If h_i ≠ 0 for each i, then the matrix representation of each piece of the map R has full row rank as well. Therefore, Prob $\{R_i(Z)=b\}=0\text{for all}b\in ℝ.$ The following theorem provides a way to compute individual confidence intervals for $({\beta }_0^{true},{\beta }^{true})$ in the general case where h_i ≠ 0 for each i.

Theorem 6.

Suppose that assumptions in Theorem 4 and Assumptions 4’(a-b) hold, and h_i ≠ 0 for each i. Let $\alpha \in (0,1)$ and a^r(·) be as in (57). Then for every $r\in ℝ$ and all i = 0, 1, ⋯,p, we have

$$\underset{N\to \infty }{\lim }\mathrm{Prob}\{|\sqrt{N}({\widehat{\beta }}_i^{\mathit{\text{true}}}-{\beta }_i^{\mathit{\text{true}}})-r|⩽a^r({\widehat{R}}_{i+1})\}=1-\alpha ,$$ (58)

where R and $\widehat{R}$ are defined in (56).

From (58), one can compute the empirical (1 − α) percentile confidence intervals for $({\beta }_0^{true},{\beta }^{true})$ by simulating data from $\widehat{R}(Z)$. The constant r can be used to control the centers of the confidence intervals for all ${\beta }_i^{\mathit{\text{true}}}$ simultaneously, which may affect the interval lengths. A reasonable choice of r is 0 if the empirical distribution of $\widehat{R}(Z)$ is approximately symmetric with respect to 0. Results of Theorems 5 and 6 are applicable to a wide range of general penalty functions which covers the LASSO as a special case (Lu et al., 2017). The procedure to construct confidence intervals of the true regression coefficients is summarized in Table 1.

3.5. Extension to the high dimensional case

In our previous theoretical analysis, we assume that the dimension p is fixed. It is interesting to study the extension of our proposed method to the high dimensional case where the dimension p is also allowed to go to infinity.

As shown in Lemma 4, since $({\beta }_0^{true},{\beta }^{true})=G^{*}(z_0^{*}),$ our proposed estimate for the true parameter $({\beta }_0^{true},{\beta }^{true})\text{is}{G}^{*}(z_N).$ In fact, we can also show that $({\beta }_0^{true},{\beta }^{true})=G(z_0),$ where z₀ is defined in (19) with λ > 0 and G is a map from ${ℝ}^{2p+1}\text{to}{ℝ}^{p+1}$ defined as

$$G=\frac{1}{2}\left(\left[\begin{array}{cc}1& 0\\ 0& {\mathrm{\Sigma }}^{-1}\end{array}\right]B+\left[\begin{array}{cc}1& 0\\ 0& 2I-(1+2m){\mathrm{\Sigma }}^{-1}\end{array}\right]B\circ {\mathrm{\Pi }}_K\right),$$ (59)

and the matrix $B\in {ℝ}^{(p+1)\times (2p+1)}\text{is given by}B=[\begin{array}{cc}{I}_{p+1}& 0\end{array}].$ Motivated by this result, we can also estimate the true parameter $({\beta }_0^{true},{\beta }^{true})\text{by}\widehat{G}(z_N),$ where $\widehat{G}$ is a map from ${ℝ}^{2p+1}$ to ${ℝ}^{p+1}$ defined as

$$\widehat{G}=\frac{1}{2}\left(\left[\begin{array}{cc}1& 0\\ 0& \widehat{\mathrm{\Theta }}\end{array}\right]B+\left[\begin{array}{cc}1& 0\\ 0& 2I-(1+2m)\widehat{\mathrm{\Theta }}\end{array}\right]B\circ d{\mathrm{\Pi }}_S\left(z_N\right)\right),$$ (60)

and $\widehat{\mathrm{\Theta }}$ is a consistent estimate of Σ⁻¹. Theoretically, if ${\lim }_{N\to \infty }\sqrt{N}{\lambda }_i=0$ for each i, we can show that G*(z_N) and $G^{*}(z_N)\text{and}\widehat{G}(z_N)$ have the same asymptotic distribution.

According to the definition of $\widehat{G}$ in (60) and the definition of z_N in (26), we can show that $\widehat{G}(z_N)=({\widehat{\beta }}_0+\frac{1}{N}{1}_N^T(\mathbf{y}-{\widehat{\beta }}_0{1}_N-\mathbf{X}\widehat{\beta }),\widehat{\beta }+\frac{1}{N}\widehat{\mathrm{\Theta }}{\mathbf{X}}^T(\mathbf{y}-{\widehat{\beta }}_0{1}_N-\mathbf{X}\widehat{\beta })).$ Therefore, the estimate of β is the sum of an initial estimate $\widehat{\beta }$ (e.g., LASSO, SCAD or MCP estimate) and a bias-correction term. Interestingly, although we have different motivations, $\widehat{G}(z_N)$ turns out to be the same as the estimate proposed by Van de Geer et al. (2014). For the high dimensional case with $p\gg N,$ if we choose $\lambda =O(\sqrt{\log (p)/N})$ converging to 0 as $N\to \infty ,$ and use conditions to guarantee that: (a) ${\Vert \widehat{\beta }-{\beta }^{true}\Vert }_1=O_p(s_0\sqrt{\log (p)/N})$ where s₀ is the number of true nonzero regression coefficients; (b) $s_0=o(\sqrt{N}/\log (p)),$ and some sparisty assumptions about the precision matrix Σ⁻¹, we can show that the asymptotic distribution of $\widehat{G}(z_N)$ is normal (Van de Geer et al. (2014)). However, the theoretical analysis of the asymptotic distribution of G*(z_N) and $\widehat{G}(z_N)$ using stochastic variational inequality techniques for $p\gg N$ case is challenging. Many fundamental results about variational inequality (e.g., some results used in the proof of Theorem 1) need to be generalized to the high dimensional case.

Although we assume that the dimension p is fixed in the theoretical analysis using stochastic variational inequality techniques, our proposed method is applicable to the large p small N data in practice. For the high dimensional data, we can use the results shown in Theorem 6 to construct confidence intervals. In Section 4, we will use Example 3 to study the performance of our method for the high dimensional data.

4. Numerical examples

In this section, we use the MCP methods in (8) to illustrate the performance of the techniques proposed in Section 3. For all examples in this section, we choose $\frac{1}{g(N)}=\frac{0.001}{N^{1/3}}\text{and}m=\frac{1}{2}$ in (9). We use the mixed integer quadratically constrained program (MIQCP) solver in the optimization modeling language GAMS (Brooke et al., 1998) to obtain accurate solutions to (2).

For all simulated examples, we generate the data using the following linear model:

$$Y=X^T{\beta }^{true}+\sigma ϵ,$$ (61)

where ${\beta }^{true}\in {ℝ}^p,X$ is a p-dimensional normal random variable with mean 0 and covariance Σ, is a standard normal random error which is independent of X. We set the noise level σ = 1. Under the model (61), the population penalized regression problem (1) can be written as

$$\underset{{\beta }_0,\beta }{\min }{({\beta }^{\mathit{\text{true}}}-\beta )}^T\mathrm{\Sigma }({\beta }^{\mathit{\text{true}}}-\beta )+{\beta }_0^2+\sum _{j=1}^p{P}_{{\lambda }_j}(|{\beta }_j|).$$ (62)

We compute confidence intervals for the population penalized parameter $({\tilde{\beta }}_0,\tilde{\beta })$ and the true model parameter β^true, which we refer to as the first and second types of confidence intervals respectively. To show their performance in simulation study, we report the following two measures: the empirical coverage rate (the fraction of total replications in which the confidence intervals contain the corresponding population penalized parameters or true model parameters) and the average confidence interval length. For the second type of confidence interval, we compare our proposed method with the LDPE method (Van de Geer et al. (2014); Zhang and Zhang (2014)), the method introduced by Javanmard and Montanari (2014) (dentoed as JM method), and the method proposed by Lu et al. (2017) (denoted as SVI-Lasso). In terms of the tuning parameter λ, we study the performance of our proposed method with some fixed values as well as the value of A chosen by the Generalized Information Criterion (GIC, Konishi and Kitagawa (2008)).

4.1. Example 1: Low dimensional setting with the auto-regressive covariance structure

For this example, we generate a training dataset with 500 replications of sample size N = 300, dimension p = 8, true model parameter β^true = (3, 1.5, 0, 0, 2, 0, 0, 0), and true covariance matrix ${\mathrm{\Sigma }}_{ij}={0.5}^{|i-j|}.$ We consider six MCP penalties with parameters (λ, a) taking the following values: λ = 0.5, 1 or 2, and a = 2 or 2000. When a = 2000, the MCP penalties are very close to the LASSO penalty. In each replication, after solving the SAA problem for every MCP penalty, we compute the two types of individual confidence intervals with the confidence level 0.95 (α = 0.05).

Tables 2 and 3 show the empirical coverage rates (CR) and average interval lengths (Len) for 95% individual confidence intervals of the 500 replications. In Table 2, the $“\tilde{\beta }”$ column contains the population penalized parameters for different MCP penalties, which are expected to be covered by the first type of confidence intervals. In Table 3, the “True” column contains the true model parameters β^true, which are expected to be covered by the second type of confidence intervals. Note that the coverage rate is 100% for the first type of confidence interval when ${\tilde{\beta }}_i=0.$ This is due to the shrinkage effect of the projection Γ (43) from z₀ to $\tilde{\beta }$, which causes the confidence intervals for ${\tilde{\beta }}_i$ to be the singleton {0}. When λ = 0.5 and a = 2, the population penalized parameters coincide with the true model parameters. The second type of confidence intervals are much longer than the first type for the inactive parameters β₃, β₄, β₆, β₇ and β₈ as expected. In practice, which type of confidence intervals to use depends on the type of parameters of interest. The first type of confidence intervals can be used to assess the randomness of the penalized estimates with a fixed penalty. This type of inference is especially useful when the penalty conveys prior information on the parameters. In contrast, the second type of confidence intervals provide inference information for the underlying true parameters directly.

As a remark, the parameter λ controls the level of penalization and the parameter a in the MCP penalty controls the degree of non-convexity. As shown in Tables 2 and 3, when λ increases to 1 and 2, the differences between the population penalized parameters and the true model parameters become larger. On the other hand, as a gets large, such as a = 2000, the MCP penalty becomes close to the LASSO penalty. The lengths of the second type of confidence intervals for a = 2 are generally shorter than those for a = 2000 with similar coverage rates as shown in Table 3. This may be due to the smaller bias imposed by the MCP penalty with a small a. In addition, as shown in Table 4, our proposed method with λ selected by GIC has very good performance. The comparison between our proposed methods and the LASSO-type methods indicate that our methods perform well for the inference of the true parameters in the linear model.

Table 4:

Coverage rates and average lengths of 95% individual CIs for true model parameters (${\beta }_0^{true},{\beta }^{true}$) for different methods from 500 replications with sample size N = 300 generated in Example 1.

		Our method (a = 2)								Lasso type methods
		λ = 0.5		λ = 1		λ = 2		GIC		SVI-Lasso		LDPE		JM
	True	CR	Len	CR	Len	CR	Len	CR	Len	CR	Len	CR	Len	CR	Len
${\beta }_1^{true}$	3	0.95	0.26	0.96	0.27	0.99	0.42	0.95	0.26	0.95	0.26	0.94	0.26	0.88	0.25
${\beta }_2^{true}$	1.5	0.95	0.29	0.96	0.30	1.00	0.51	0.95	0.29	0.95	0.29	0.94	0.28	0.89	0.28
${\beta }_3^{true}$	0	0.95	0.28	0.96	0.29	0.97	0.42	0.95	0.28	0.95	0.28	0.95	0.28	0.99	0.28
${\beta }_4^{true}$	0	0.97	0.28	0.97	0.29	0.97	0.42	0.97	0.28	0.97	0.28	0.96	0.28	0.97	0.28
${\beta }_5^{true}$	2	0.96	0.28	0.96	0.29	0.99	0.44	0.96	0.28	0.97	0.29	0.96	0.28	0.92	0.28
${\beta }_6^{true}$	0	0.95	0.28	0.95	0.28	0.98	0.42	0.95	0.28	0.94	0.28	0.95	0.28	0.98	0.28
${\beta }_7^{true}$	0	0.96	0.28	0.96	0.29	0.98	0.42	0.96	0.28	0.95	0.28	0.95	0.28	0.99	0.28
${\beta }_8^{true}$	0	0.96	0.25	0.97	0.26	0.99	0.38	0.97	0.25	0.96	0.26	0.97	0.26	0.98	0.25

4.2. Example 2: Low dimensional setting with the equi-correlation covariance structure

In this example, we consider the equi-correlation covariance structure where Σ_ij = 0.5 for all i ≠ j and Σ_jj = 1 for all j. The other settings are the same as Example 1.

Table 5 shows the performance of the 95% individual confidence intervals of the population penalized parameters. The results shown in this table are very similar to the results of Example 1 shown in Table 2. As shown in Table 5, for each fixed λ, the proposed method using a = 2 delivers better performance than the method using a = 2000 in most cases, especially when λ is small. Table 6 shows the comparison of the individual confidence intervals of the true model parameters constructed by our method and the LASSO-type methods. Similar to Example 1, our proposed method using GIC performs well. In addition, for this example, ourmethod (GIC), SVI-Lasso and LDPE methods deliver similar performance. All these three methods perform better than the JM method.

Table 5:

Coverage rates and average lengths of 95% individual CIs for population penalized parameters $({\tilde{\beta }}_0,\tilde{\beta })$ for different MCP penalties from 500 replications with sample size N = 300 generated in Example 2.

	a = 2									a = 2000
	λ = 0.5			λ = 1			λ = 2			λ = 0.5			λ = 1			λ = 2
	$\tilde{\beta }$	CR	Len	$\tilde{\beta }$	CR	Len	$\tilde{\beta }$	CR	Len	$\tilde{\beta }$	CR	Len	$\tilde{\beta }$	CR	Len	$\tilde{\beta }$	CR	Len
β0	0	0.98	0.26	0	0.98	0.27	0	0.99	0.38	0	0.98	0.27	0	0.97	0.31	0	0.97	0.41
β1	3	0.99	0.31	3.10	0.98	0.36	3.57	0.91	0.95	2.88	0.98	0.33	2.75	0.97	0.38	2.50	0.97	0.52
β2	1.5	0.98	0.32	1.20	0.98	0.55	0.57	0.95	0.88	1.37	0.97	0.34	1.25	0.97	0.38	1.00	0.96	0.52
β3	0	1.00	0.00	0	1.00	0.00	0	1.00	0.00	0	1.00	0.05	0	1.00	0.02	0	1.00	0.01
β4	0	1.00	0.00	0	1.00	0.00	0	1.00	0.00	0	1.00	0.05	0	1.00	0.02	0	1.00	0.00
β5	2	0.97	0.32	2.10	0.99	0.38	1.57	0.98	0.93	1.88	0.97	0.34	1.75	0.97	0.38	1.50	0.98	0.52
β6	0	1.00	0.00	0	1.00	0.00	0	1.00	0.00	0	1.00	0.05	0	1.00	0.02	0	1.00	0.00
β7	0	1.00	0.00	0	1.00	0.00	0	1.00	0.00	0	1.00	0.05	0	1.00	0.02	0	1.00	0.00
β8	0	1.00	0.00	0	1.00	0.00	0	1.00	0.00	0	1.00	0.05	0	1.00	0.02	0	1.00	0.00

Table 6:

Coverage rates and average lengths of 95% individual CIs for true model parameters $({\beta }_0^{true},{\beta }^{true})$ for different methods from 500 replications with sample size N = 300 generated in Example 2.

		Our method (a = 2)								Lasso type methods
		λ = 0.5		λ = 1		λ = 2		GIC		SVI-Lasso		LDPE		JM
	True	CR	Len	CR	Len	CR	Len	CR	Len	CR	Len	CR	Len	CR	Len
${\beta }_1^{true}$	3	0.97	0.30	0.97	0.31	1.00	0.46	0.97	0.30	0.97	0.30	0.97	0.30	0.91	0.29
${\beta }_2^{true}$	1.5	0.95	0.30	0.97	0.32	1.00	0.50	0.95	0.30	0.95	0.31	0.95	0.30	0.89	0.29
${\beta }_3^{true}$	0	0.95	0.30	0.96	0.31	0.99	0.43	0.95	0.30	0.96	0.30	0.96	0.30	0.98	0.29
${\beta }_4^{true}$	0	0.96	0.30	0.97	0.31	0.99	0.43	0.96	0.30	0.96	0.30	0.96	0.30	0.99	0.29
${\beta }_5^{true}$	2	0.92	0.30	0.94	0.31	0.99	0.46	0.92	0.30	0.92	0.31	0.92	0.30	0.88	0.29
${\beta }_6^{true}$	0	0.95	0.30	0.95	0.31	1.00	0.43	0.95	0.30	0.95	0.30	0.94	0.30	0.98	0.29
${\beta }_7^{true}$	0	0.95	0.30	0.95	0.31	0.99	0.43	0.95	0.30	0.95	0.30	0.95	0.30	0.96	0.29
${\beta }_8^{true}$	0	0.94	0.30	0.95	0.31	1.00	0.43	0.94	0.30	0.94	0.30	0.94	0.30	0.98	0.29

4.3. Example 3: High dimensional example

In this example, we consider a high dimensional case in which the dimension is much larger than the sample size. We choose p = 300 with β^true being a 300-dimensional vector: ${\beta }_1^{true}=3,{\beta }_2^{true}={\beta }_{100}^{true}={\beta }_{200}^{true}={\beta }_{300}^{true}=1.5,{\beta }_5^{true}={\beta }_{95}^{true}=2,{\beta }_{10}^{true}=1,{\beta }_{25}^{true}=0.5$, and all the other components are 0. The true covariance matrix is ${\mathrm{\Sigma }}_{ij}={0.5}^{|i-j|}.$ We generate a training dataset with 500 replications of sample size N = 100. For this high dimensional example, we consider three MCP penalties with parameters λ = 0.5, 1 or 2, and a = 3. In each replication, we use the nodewise LASSO regression introduced by Meinshausen and Bühlmann (2006) to compute the estimate of the precision matrix $\widehat{\mathrm{\Theta }},$ and compute the individual confidence intervals of the true model parameters with the confidence level 0.95. Define the active set as $\mathcal{A}=\{i:{\beta }_i^{\text{true}}\ne 0\}=\{1,2,5,10,25,95,100,200,300\}\text{and}{\mathcal{A}}^c=\{0,1,2,\cdots ,300\}\backslash \mathcal{A}.$ In Table 7, for different methods, we report the average coverage rate, median coverage rate, average length and median length of the individual confidence intervals for true model parameters in $\mathcal{A}\text{and}{\mathcal{A}}^c$ respectively:

$$\mathrm{Avgcov}\mathcal{A}=|\mathcal{A}{|}^{-1}\sum _{i\in \mathcal{A}}{\text{CR}}_i,\mathrm{Avgcov}{\mathcal{A}}^c={|{\mathcal{A}}^c|}^{-1}\sum _{i\in {\mathcal{A}}^c}{\text{CR}}_i,$$

$$\text{Avglen}\mathcal{A}=|\mathcal{A}{|}^{-1}\sum _{i\in \mathcal{A}}{\text{Len}}_i,\text{Avglen}{\mathcal{A}}^c={|{\mathcal{A}}^c|}^{-1}\sum _{i\in {\mathcal{A}}^c}{\text{Len}}_i,$$

$$\text{Medcov}\mathcal{A}=\underset{i\in \mathcal{A}}{\mathrm{median}}\{{\text{CR}}_i\},\text{Medcov}{\mathcal{A}}^c=\underset{i\in {\mathcal{A}}^c}{\mathrm{median}}\{{\text{CR}}_i\},$$

$$\text{Medlen}\mathcal{A}=\underset{i\in \mathcal{A}}{\mathrm{median}}\{{\text{Len}}_i\},\text{Medlen}{\mathcal{A}}^c=\underset{i\in {\mathcal{A}}^c}{\mathrm{median}}\{{\text{Len}}_i\},$$

where CR_i and Len_i denote the empirical coverage rate and average interval length of the confidence interval for the parameter ${\beta }_i^{\mathit{\text{true}}}$ for the 500 replications, respectively.

Table 7:

Average coverage rates and lengths of 95% individual confidence intervals for the true model parameters in the linear model with different methods computed from 500 replications with sample size N = 100 and dimension p = 300 generated in Example 3.

	Our method (λ = 0.5)				Our method (λ = 1)				Our method (λ = 2)
	Avgcov	Medcov	Avglen	Medlen	Avgcov	Medcov	Avglen	Medlen	Avgcov	Medcov	Avglen	Medlen
$\mathcal{A}$	92.82	92.60	0.46	0.45	94.64	95.00	0.66	0.65	95.24	95.40	1.26	1.25
${\mathcal{A}}^c$	93.26	93.40	0.39	0.39	93.51	93.60	0.56	0.56	93.73	94.00	1.08	1.08
	Our method (GIC)				LDPE				JM
	Avgcov	Medcov	Avglen	Medlen	Avgcov	Medcov	Avglen	Medlen	Avgcov	Medcov	Avglen	Medlen
$\mathcal{A}$	92.91	93.00	0.57	0.56	93.84	94.40	1.13	1.14	88.07	87.80	0.55	0.55
${\mathcal{A}}^c$	93.37	93.40	0.47	0.47	95.31	95.60	1.14	1.14	99.38	99.40	0.55	0.55

For our proposed methods, as λ increases to 1 and 2, both the average coverage rates and lengths increase. Compared with LDPE, our proposed method using GIC to choose the tuning parameter has much shorter average lengths while the average coverage rates are only slightly lower. Although the JM method delivers similar average lengths as our proposed method (GIC), the average coverage rates of our proposed method are much closer to the nominal level 95%. Overall, the results shown in Table 7 indicate that our proposed method still delivers comparable performance for the high dimensional case.

4.4. Example 4: ADNI data

In this real data example, we consider the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset (http://www.loni.ucla.edu/ADNI). The main goal of ADNI was to test whether the serial structural magnetic resonance imaging (MRI), fluorodeoxyglucose positron emission tomography (FDG-PET) images and some other biological markers such as cerebrospinal fluid (CSF) could be combined to measure the progression of mild cognitive impairment (MCI) and early Alzheimer’s Disease (AD). To that end, 800 adults with ages between 55 and 90 were recruited from over 50 sites across the US and Canada. In our analysis, we use data from 199 subjects who have complete baseline MRI, FDG-PET, and CSF data. Using the data processing method shown in Thung et al. (2014), we obtained 93 MRI features, 93 PET features, and 5 CSF features for each subject. The response variable is the Mini-Mental State Examination (MMSE) score (Folstein et al. (1975)) which is often used to screen for cognitive impairment.

The data are standardized at the beginning of our analysis. For our proposed method, we use the MCP penalty with the parameter a = 3 and choose the best tuning parameter λ by GIC. Table 8 shows the selected features of different methods, where the selected features are the features whose 95% confidence intervals do not contain 0. The numbers of features selected by our method, SVI-Lasso, LDPE, and JM are 13, 12, 13, and 3, respectively. Among the 13 features selected by our proposed method, 11 features are selected by the SVI-Lasso method, 9 features are selected by the LDPE method and 3 features are selected by the JM method. Table 9 shows the estimates and 95% individual confidence intervals of the 13 features selected by our proposed method. The results of our proposed method and the results of SVI-Lasso and LDPE methods are comparable. As shown in Table 9, for most features among the 13 features, the absolute values of the estimates delivered by the JM method are much smaller than the corresponding values of the other methods. The 95% confidence intervals of the JM method are also very different from the corresponding confidence intervals of the other methods.

Table 8:

Selected features of different methods for the ADNI data.

Method	Selected Features
Our method (GIC)	9, 19, 40, 59, 67, 80, 95, 130, 134, 147, 156, 168, 178
SVI-Lasso	9, 19, 40, 77, 80, 95, 130, 134, 147, 156, 168, 178
LDPE	9, 19, 40, 59, 77, 80, 83, 90, 111, 134, 147, 156, 168
JM	19, 40, 134

Table 9:

Estimates and 95% individual confidence intervals of the 13 features selected by our proposed method for the ADNI data.

	Our method (GIC)		SVI-Lasso		LDPE		JM
	Est	Ind CI	Est	Ind CI	Est	Ind CI	Est	Ind CI
${\beta }_9^{true}$	−0.20	[−0.33, −0.07]	−0.20	[−0.33, −0.07]	−0.19	[−0.34, −0.04]	−0.15	[−0.31, 0.01]
${\beta }_{19}^{true}$	0.24	[0.06, 0.41]	0.23	[0.07, 0.40]	0.24	[0.09, 0.40]	0.25	[0.08, 0.42]
${\beta }_{40}^{true}$	−0.21	[−0.36, −0.06]	−0.21	[−0.35, −0.06]	−0.20	[−0.35, −0.05]	−0.16	[−0.33, 0.00]
${\beta }_{59}^{true}$	0.15	[0.01, 0.29]	0.15	[0.00, 0.30]	0.16	[0.01, 0.30]	0.12	[−0.03, 0.28]
${\beta }_{67}^{true}$	0.13	[0.00, 0.27]	0.13	[0.00, 0.26]	0.12	[−0.02, 0.26]	0.11	[−0.03, 0.26]
${\beta }_{80}^{true}$	0.23	[0.03, 0.43]	0.23	[0.03, 0.42]	0.21	[0.04, 0.38]	0.15	[−0.01, 0.30]
${\beta }_{95}^{true}$	0.20	[0.00, 0.40]	0.21	[0.01, 0.41]	0.19	[−0.01, 0.39]	0.04	[−0.11, 0.20]
${\beta }_{130}^{true}$	0.21	[0.00, 0.42]	0.20	[0.01, 0.40]	0.18	[−0.03, 0.39]	0.04	[−0.12, 0.19]
${\beta }_{134}^{true}$	0.25	[0.08, 0.43]	0.24	[0.02, 0.45]	0.24	[0.06, 0.43]	0.23	[0.08, 0.38]
${\beta }_{147}^{true}$	−0.22	[−0.41, −0.02]	−0.22	[−0.41, −0.03]	−0.21	[−0.40, −0.02]	−0.03	[−0.18, 0.13]
${\beta }_{156}^{true}$	−0.19	[−0.36, −0.02]	−0.19	[−0.36, −0.02]	−0.18	[−0.37, 0.00]	−0.05	[−0.21, 0.11]
${\beta }_{168}^{true}$	−0.24	[−0.43, −0.04]	−0.24	[−0.43, −0.04]	−0.22	[−0.43, −0.02]	0.01	[−0.14, 0.16]
${\beta }_{178}^{true}$	−0.19	[−0.34, −0.03]	−0.19	[−0.35, −0.04]	−0.18	[−0.36, 0.00]	−0.07	[−0.22, 0.08]

5. Discussion

In this paper we propose a unified framework to construct confidence intervals for the population penalized parameters as well as the true model parameters for a large class of penalties. By transforming the population penalized regression problem (1) and its SAA problem (2) to the equivalent problems (9) and (22) respectively, we exclude the non-smoothness in the objectives. Furthermore, we obtain their normal map formulations (18) and (25), and derive the asymptotic distributions and the two types of confidence intervals. Our numerical results show that these methods are effective. When the objective functions in (1) and (2) are non-convex as a result of non-convex penalty functions, most existing algorithms are only guaranteed to obtain a local optimal solution for the SAA problem. Our proposed methods will generate confidence intervals based on that local solution. In practice, we solve for a SAA solution $({\widehat{\beta }}_0,\widehat{\beta })$ and then use (26) to obtain a solution to (25). The first type of confidence intervals we compute are for a local optimal solution of the population penalized regression problem (1). From any local solution of (2), we can always compute confidence intervals for the true model parameters, which are the second type of confidence intervals we compute.