Adaptive Elastic Net for Generalized Methods of Moments

Abstract

Model selection and estimation are crucial parts of econometrics. This paper introduces a new technique that can simultaneously estimate and select the model in generalized method of moments (GMM) context. The GMM is particularly powerful for analyzing complex data sets such as longitudinal and panel data, and it has wide applications in econometrics. This paper extends the least squares based adaptive elastic net estimator of Zou and Zhang (2009) to nonlinear equation systems with endogenous variables. The extension is not trivial and involves a new proof technique due to estimators lack of closed form solutions. Compared to Bridge-GMM of Caner (2009), we allow for the number of parameters to diverge to infinity as well as collinearity among a large number of variables, also the redundant parameters set to zero via a data dependent technique. This method has the oracle property, meaning that we can estimate nonzero parameters with their standard limit and the redundant parameters are dropped from the equations simultaneously. Numerical examples are used to illustrate the performance of the new method.

1 Introduction

One of the most commonly used estimation techniques is Generalized Method of Moments (GMM) estimation. The GMM provides a unified framework for parameter estimation by encompassing many common estimation methods such as ordinary least squares (OLS), maximum likelihood estimator (MLE), and instrumental variables. We can estimate the parameters by two-step efficient Generalized Method of Moments (GMM) of Hansen (1982). The GMM is an important tool in econometrics, finance, accounting, and strategic planning literature as well. In this paper we are concerned about model selection in GMM when the number of parameters diverges. These situations can arise in labor economics, international finance (see Alfaro, Kalemli-Ozcan, Volosovych (2008)), and so on. In linear models when the some of the regressors are correlated with errors and there are a large number of covariates, the model selection tools are essential, since they can improve finite sample performance of the estimators.

Model selection techniques are very useful and widely used in statistics. For example Tibshirani (1996) propose the lasso method, Knight and Fu (2000) derive the asymptotic properties of the lasso, and Fan and Li (2001) propose the SCAD estimator. In econometrics, Knight (2008), and Caner (2009) offer Bridge-least squares and Bridge-GMM estimators respectively. But these procedures all consider finite dimensions and do not take into account the collinearity among variables. Recently model selection with a large number of parameters has been analyzed in least squares by Huang, Horowitz, and Ma (2008), and Zou and Zhang (2009), where the first article analyzes the Bridge estimator, and the second paper is concerned with the adaptive elastic net estimator.

Adaptive elastic net estimator has the oracle property when the number of parameters diverges with the sample size. Furthermore, this method can handle the collinearity arising from a large number of regressors when the system is linear with endogenous regressors. When some of the parameters are redundant (i.e. when the true model has a sparse representation), this estimator can estimate the zero parameters as exactly zero.

In this paper we extend the least squares based adaptive elastic net of Zou and Zhang (2009) to GMM. The following issues are pertinent to model selection in GMM: (i) handling a large number of control variables in the structural equation in a simultaneous equation system, or a large number of parameters in a nonlinear system with endogenous and control variables; (ii) taking into account correlation among variables; (iii) achieving selection consistency and estimation efficiency simultaneously. All of these are successfully addressed in this work. In the least squares case of Zou and Zhang (2009), they do not need an explicit consistency proof since the least squares estimator has a simple and closed form solution. However, in this paper, since the GMM estimator does not have a closed form solution, an explicit consistency proof is needed before deriving the finite sample risk bounds. This is one major contribution of this paper. Furthermore, in order to get a consistency proof, we have substantially extended the technique used in the consistency proof of Bridge least squares estimator of Huang, Horowitz and Ma (2008) to the GMM with adaptive elastic net penalty. To derive the finite sample risk bounds, we use the mean value theorem and benefit from consistency proof, unlike the least squares case of Zou and Zhang (2009). The nonlinear nature of the functions introduces additional difficulties. The GMM case involves partial derivatives of the sample moments which depend on parameter estimates. This is unlike the least squares case where the same quantity does not depend on parameter estimates. This results in the need for consistency proof as mentioned above. Also we extend Zou and Zhang (2009) to conditionally heteroscedastic data, and this results in tuning parameter for l₁ norm to be larger than the one in least squares case. We also pinpoint ways to handle stationary time series cases. The estimator also has the oracle property, and the nonzero coefficients are estimated converging to a normal distribution. This is their standard limit and furthermore, the zero parameters are estimated as zero. Note that the oracle property is a pointwise criterion.

Earlier works on diverging parameters include Huber (1988) and Portnoy (1984), He and Shao (2000). In recent years, there are a few works on penalized methods for standard linear regression with diverging parameters. Fan and Peng (2004) study the nonconcave penalized likelihood with a growing number of nuisance parameters, Lam and Fan (2007) analyze the profile likelihood ratio inference with a growing number of parameters, and Huang et al. (2008) study asymptotic properties of bridge estimators in sparse linear regression models. As far as we know, this is the first paper to estimate and select the model in GMM with a diverging number of parameters. In econometrics, sieve estimation will be a natural application of shrinkage estimators. There are several articles that use sieves, Ai and Chen (2003), Newey and Powell (2003), Chen (2007), Chen and Ludvigson (2009). In these articles, we see that sieve dimension is determined by trying several possibilities or left for future work. Adaptive elastic net GMM can simultaneously determine the sieve dimension and estimate the structural parameters. We also see in unpenalized GMM with many parameters case, there is a paper by Han and Phillips (2006). Liao (2011) considers adaptive lasso with fixed number of invalid moment conditions.

Section 2 presents the model and the new estimator. Then in Section 3 we derive the asymptotic results for the proposed estimator. Section 4 conducts simulations. Section 5 provides an asset pricing example used in Chen and Ludvigson (2009). Section 6 concludes. Appendix includes all the proofs.

2 Model

Let β be a p-dimensional parameter vector, where β ∈ B_p which is a compact subset in R^p. The true value of β is β₀. We allow p to grow with the sample size n, so when n → ∞, we have p → ∞, but p/n → 0 as n → ∞. We do not provide a subscript of n for parameter space not to burden ourselves with the notation. The population orthogonality conditions are

$$E\left[g\right(X_i,{\beta }_0\left)\right]=0,$$

where the data are {X_i : i = 1, 2 ⋯ n}, g(·) is a known function, and the number of orthogonality restrictions is q, q ≥ p. So we also allow q to grow with the sample size n, but q/n → 0 as n → ∞. From now on, we denote g(X_i, β) as g_i(β) for simplicity. Also assume that g_i(β) are independent, and we do not use g_ni(β) just to simplify the notation.

2.1 The Estimators

We first define the estimators that we use. The estimators that we are interested in aim to answer the following questions. If we have a large number of control variables, some of which may be irrelevant (we may have also a large number of endogenous variables and control variables) in the structural equation in a simultaneous equation system or a large number of parameters in a nonlinear system with endogenous and control variables, can we select the relevant ones as well as estimate the selected system simultaneously? If we have a large number of variables among which there may be possibly some correlation among the variables, can this method handle that? Is it also possible for the estimator to achieve the oracle property? The answers to all three questions are affirmative. First of all, the adaptive elastic net estimator simultaneously selects and estimates the model when there are a large number of parameters/regressors. It can also take into account the possible correlation among the variables. By achieving the oracle property, the nonzero parameters are estimated with their standard limits, and the zero ones are estimated as exactly zero. This method is computationally easy and uses data dependent methods to set small coefficient estimates to zero. A subcase of the adaptive elastic net estimator is the adaptive lasso estimator which can handle the first and third questions but does not handle correlation among a large number of variables.

First we introduce the notation: we use the following norms for the vector ${\Vert \beta \Vert }_1={\sum }_{j=1}^p\mid {\beta }_j\mid$, ${\Vert \beta \Vert }_2^2={\sum }_{j=1}^p{\mid {\beta }_j\mid }^2$, also ${\Vert \beta \Vert }_{2+l}^{2+l}={\sum }_{j=1}^p{\mid {\beta }_j\mid }^{2+l}$, where l > 0 is a positive number. For a matrix A, the norm is ${\Vert A\Vert }_2^2=\mathit{tr}\left(A^{\prime }A\right)$. We start by introducing the adaptive elastic net estimator, given the positive and diverging tuning parameters λ₁, ${\lambda }_1^{\ast }$, λ₂ (how to choose them in finite samples and its asymptotic properties will be discussed below in Assumptions and then in Simulation Section)

$${\widehat{\beta }}_{\mathit{aenet}}=(1+{\lambda }_2∕n)\left\{\arg \underset{\beta \in B_p}{\min }\right[{\left(\sum _{i=1}^n{g}_i\right(\beta \left)\right)}^{\prime }{W}_n\left(\sum _{i=1}^n{g}_i\right(\beta \left)\right)+{\lambda }_2{\Vert \beta \Vert }_2^2+{\lambda }_1^{\ast }\sum _{j=1}^p{\widehat{w}}_j\mid {\beta }_j\mid \left]\right\},$$ (1)

where ${\widehat{w}}_j=\frac{1}{{\mid {\widehat{\beta }}_{\mathit{enet}}\mid }^{\gamma }},{\widehat{\beta }}_{\mathit{enet}}$ is a consistent estimator immediately explained below, γ is a positive constant, and p = n^α, 0 < α < 1. The assumption on γ will be explained in detail in Assumption 3(iii). W_n is a q × q weight matrix that will be defined in Assumptions below.

The elastic net estimator, which is used in the weights of the penalty above, is

$${\widehat{\beta }}_{\mathit{enet}}=(1+{\lambda }_2∕n)\left\{\arg \underset{\beta \in B_p}{\min }{S}_n\right(\beta \left)\right\},$$

where

$$S_n\left(\beta \right)={\left[\sum _{i=1}^n{g}_i\right(\beta \left)\right]}^{\prime }{W}_n\left[\sum _{i=1}^n{g}_i\right(\beta \left)\right]+{\lambda }_2{\Vert \beta \Vert }_2^2+{\lambda }_1{\Vert \beta \Vert }_1,$$ (2)

λ₁, λ₂ are positive and diverging sequences that will be defined in Assumption 5.

We now discuss the penalty functions in both estimators and explain why we need ${\widehat{\beta }}_{\mathit{enet}}$. The elastic net estimator has both l₁ and l₂ penalties. The l₁ penalty is used to perform automatic variable selection, and the l₂ penalty is used to improve the prediction and handles the collinearity that may arise with a large number of variables. However, the standard elastic net estimator does not provide the oracle property. It turns out that, by introducing an adaptive weight in the elastic net, we can obtain the oracle property. The adaptive weights play crucial roles, since they provide data dependent penalization.

An important point to remember is when we set λ₂ = 0 in the adaptive elastic net estimator (1), we obtain the adaptive lasso estimator. This is simple and we can also get the oracle property. However, with a large number of parameters/variables which may be highly collinear, an additional ridge-type penalty as in adaptive elastic net offers estimation stability and better selection. Before the assumptions we introduce the following notations. Let the collection of nonzero parameters be the set A = {j : ∣β_j0∣ ≠ 0} and denote the absolute value of the minimum of the nonzero coefficients as η = min_j∈A∣β_j0∣. Also the cardinality of A is p_A (the number of nonzero coefficients). We now provide the main assumptions.

Assumptions

1. Define the following q × p matrix ${\widehat{G}}_n\left(\beta \right)=\frac{{\sum }_{i=1}^n\partial g_i\left(\beta \right)}{\partial {\beta }^{\prime }}$. Assume the following uniform law of large numbers

   $$\underset{\beta \in B_p}{\sup }{\Vert {\widehat{G}}_n\left(\beta \right)∕n-G\left(\beta \right)\Vert }_2^2\stackrel{P}{\to }0,$$

   where G(β) is continuous in β and has a full column rank p. Also β ∈ B_p ⊂ R^p, B_p is compact and all the absolute value of individual components of the vector β, are uniformly bounded by a constant a, ∣β_j0∣, 0 ≤ a < ∞, j = 1, ⋯, p. Note that specifically ${\sup }_{\beta \in B_p}{\Vert \frac{1}{2}{\sum }_{i=1}^{n}E\frac{\partial g_i\left(\beta \right)}{\partial {\beta }^{\prime }}-G\left(\beta \right)\Vert }_2^2\to 0$, defines G(β).

2. W_n is a symmetric, positive definite matrix, and ${\Vert W_n-W\Vert }_2^2\stackrel{P}{\to }0$, where W is finite and positive definite matrix as well.

3. (i). ${\Vert {\left[n^{-1}{\sum }_{i=1}^n{\mathit{Eg}}_i\right({\beta }_0\left)g_i{\left({\beta }_0\right)}^{\prime }\right]}^{-1}-{\Omega }^{-1}\Vert }_2^2\to 0$.

   (ii). Assume p = n^α, 0 < α < 1, p/n → 0, q/n → 0 as n → ∞, and p, q → ∞, q ≥ p, so q = n^ν, 0 < ν < 1, α ≤ ν.

   (iii). The coefficient on the weights: γ satisfies the following bound $\gamma >\frac{2+\alpha }{1-\alpha }$.

4. Assume that

   $$\underset{i}{\max }\frac{E{\Vert g_i\left({\beta }_{A,0}\right)\Vert }_{2+l}^{2+l}}{n^{l∕2}}\to 0,$$

   for l > 0, and where β_A,0 represents the true values of the nonzero parameters and it is of p_A dimension. The dimension also increases with the sample size; p_A → ∞, as n → ∞, and 0 ≤ p_A ≤ p.

5. (i). ${\lambda }_1∕n\to 0,{\lambda }_2∕n\to 0,{\lambda }_1^{\ast }∕n\to 0$

   (ii).

   $$\frac{{\lambda }_1^{\ast }}{n^{3+\alpha }}{n}^{\gamma (1-\alpha )}\to \infty ,$$ (3)
   (iii).

   $$n^{1-\nu }{\eta }^2\to \infty .$$ (4)
   (iv).

   $$n^{1-\alpha }{\eta }^{\gamma }\to \infty .$$ (5)

   (v). Set η = O(n^−m), 0 < m < α/2.

Now we provide some discussions on the above assumptions. Most of them are standard and used in other papers which establish asymptotic results for penalized estimation in the context of diverging parameters. The rest of them are typically used for the GMM to deal with nonlinear equation systems with endogenous variables. Since p → ∞, Assumption 1 can be thought of uniform convergence over sieve spaces B_p. For iid subcase, primitive conditions are available in Condition 3.5M in Chen (2007).

Assumptions 1-2 are standard in the literature of GMM (Newey and Windmeijer, 2009, Chen 2007). They are similar to Assumptions 7 and 3 in Newey and Windmeijer (2009a). It is also important to see that Assumption 2 is needed for the two-step nature of the GMM problem. In the first step we can use any consistent estimator (i.e. elastic net) and substitute in $W_n=n^{-1}{\sum }_{i=1}^n{g}_i\left({\widehat{\beta }}_{\mathit{enet}}\right)g_i{\left({\widehat{\beta }}_{\mathit{enet}}\right)}^{\prime }$ in the adaptive elastic net estimation, where ${\widehat{\beta }}_{\mathit{enet}}$ is the elastic net estimator. Also note that with different estimators, we can define the limit weight W differently. Depending on the estimators W_n, W can change. Assumption 3 provides a definition of variance covariance matrix, and then establishes that the number of diverging parameters cannot exceed the sample size. This is also used in Zou and Zhang (2009). For the penalty exponent in the weight, our condition is more stringent than in the least squares case of Zou and Zhang (2009). This is needed for model selection for local to zero coefficients in GMM format. Assumption 4 is a primitive condition for the triangular array central limit theorem. This is also restraining the number of orthogonality conditions q.

The main issue are the tuning parameter assumptions that reduce bias. We first compare with Bridge estimator of Knight and Fu (2000), there in Theorem 3, they need λ/n^α/2 → λ₀ ≥ 0, where 0 < a < 1, and λ represented the only tuning parameter. In our Assumption 5(i), we need ${\lambda }_1∕n\to 0,{\lambda }_1^{\ast }∕n\to 0,{\lambda }_2∕n\to 0$, so ours can be larger than the Knight and Fu (2000) estimator, we can penalize more in our case. This is due to Bridge type of penalty, which requires less penalization in order to reduce bias and get the oracle property. Theorem 2 of Zou (2006) for the adaptive lasso in least squares assumes λ/n^1/2 → 0. We think the reason that the GMM estimator requires large penalization is due to its complex model nature, since there are more elements that contribute to bias here. Theorem 1 of Gao and Huang (2010) display the same tuning analysis as Zou (2006).

The rates on λ₁, λ₂ are standard, but the rate of ${\lambda }_1^{\ast }$ depends on α and γ. The conditions on ${\lambda }_1,{\lambda }_1^{\ast },{\lambda }_2$ are needed for consistency and the bounds on the moments of the estimators. We also allow for local to zero (nonzero) coefficients, but Assumptions 3 and 5, (4)(5) restrict their magnitude. This is tied to the possible number of nonzero coefficients and seeing that α ≤ ν. If there are too many nonzero coefficients (α near 1), then for model selection purpose, the coefficients should slowly approach zero. If there are few nonzero coefficients, to give an example (α near 0), then the order of η should be slightly larger than n^−1/2. This also confirms and extends Leeb and Pőtscher (2005) finding that local to zero coefficients should be larger than n^−1/2 in order to be differentiated from zero coefficients. This is shown in Proposition A.1(2) in their paper. Our results extend that result to the diverging parameter case. Assumption 5(iii) is needed to get consistency for local to zero coefficients. Assumptions 5(iv)(v) are needed for model selection consistency of local to zero coefficients.

To be specific about implications of Assumptions 5(iii)(iv)(v) on the order of η, since η = O(n^−m), then Assumption 5(iii) implies that

$$\frac{1-\nu }{2}>m,$$

and Assumption 5(iv) implies that

$$\frac{1-\alpha }{\gamma }>m.$$

Combining these two inequalities with Assumption 5(v), we obtain

$$m<\min (\frac{1-\alpha }{\gamma },\frac{1-\nu }{2},\frac{\alpha }{2}).$$

Now we can see that with a large number of moments, or a large number of parameters, m may get small, so the magnitude of η should be large. To give an example take γ = 5, α = 1/3, ν = 2/3, which gives us an upper bound of m < 2/15. So in that scenario, η = O(n^−2/15), to get selected as non zero. It is clear that this is much larger than n^−1/2 which Leeb and Pőtscher (2005) found.

Note also that with $\gamma >\frac{2+\alpha }{1-\alpha }$ (i.e. Assumption 3(iii)), we can assure that the conditions on ${\lambda }_1^{\ast }$ in Assumptions 5(i) and (ii) are compatible with each other.

Using Assumptions 1, 2, 3(i), we can see that

$$0<b\le {\text{Eig}}_{\min }\left[\right(\widehat{G}{\left(\beta \right)}^{\prime }∕n\left)W_n\right(\widehat{G}\left(\beta \right)∕n\left)\right],$$ (6)

and

$${\text{Eig}}_{\max }\left[\right(\widehat{G}{\left(\beta \right)}^{\prime }∕n\left)W_n\right(\widehat{G}\left(\beta \right)∕n\left)\right]\le B<\infty ,$$ (7)

with probability approaching one, where $\beta \in [{\beta }_0,{\widehat{\beta }}_w)$, B > 0. These are obtained by Exercise 8.26b of Abadir and Magnus (2005), and Lemma A0 of Newey and Windmeijer (2009b) result (eigenvalue inequality for increasing number of dimension case). ${\widehat{\beta }}_w$ is related to adaptive elastic net estimator and immediately defined below. Here Eig_min(M) and Eig_max(M) respectively represent the minimal and maximal eigenvalue of a generic matrix M.

3 Asymptotics

We define an estimator which is related to the adaptive elastic net estimator in (1) and also used in the risk bound calculations.

$${\widehat{\beta }}_w=\arg \underset{\beta \in B_p}{\min }\left[{\left(\sum _{i=1}^n{g}_i\right(\beta \left)\right)}^{\prime }{W}_n\right(\sum _{i=1}^n{g}_i\left(\beta \right))+{\lambda }_2{\Vert \beta \Vert }_2^2+{\lambda }_1\sum _{j=1}^p{\widehat{w}}_j\mid {\beta }_j\mid ],$$ (8)

The following theorem provides consistency for both the elastic net estimator and ${\widehat{\beta }}_w$.

Theorem 1

Under Assumptions 1-3, 5

(i).

$${\Vert {\widehat{\beta }}_{\mathit{enet}}-{\beta }_0\Vert }_2^2\stackrel{P}{\to }0.$$

(ii).

$${\Vert {\widehat{\beta }}_w-{\beta }_0\Vert }_2^2\stackrel{P}{\to }0.$$

Remark

It is clear from Theorem 1(ii) that adaptive elastic net estimator in (1) is also consistent. We should note that in Zou and Zhang (2009) where the least squares adaptive elastic net estimator is studied, there is no explicit consistency proof. This is due to using a simple linear model. However, for the GMM adaptive elastic net estimator we have the partial derivative of g(·) which depends on estimators unlike in the linear model case. Specifics are in equations (45)-(50). Therefore we need a new and different consistency proof compared with the least squares case. We need to introduce an estimator that is closely tied to the elastic net estimator above.

$$\widehat{\beta }({\lambda }_2,{\lambda }_1)=\arg \underset{\beta \in B_p}{\min }{S}_n\left(\beta \right),$$ (9)

where S_n(β) is defined in (2). This is also the estimator we get when we set for all j, ${\widehat{w}}_j=1$ in ${\widehat{\beta }}_w$. Next, we provide bounds for our estimators. These are used then in the proofs for oracle property, and the limits of the estimators.

Theorem 2

Under Assumptions 1-3, 5

$$E\left({\Vert {\widehat{\beta }}_w-{\beta }_0\Vert }_2^2\right)\le 4\frac{{\lambda }_2^2{\Vert {\beta }_0\Vert }_2^2+n^3\mathit{pB}+{\lambda }_1^{2}E\sum _{j=1}^p{\widehat{w}}_j^2+o\left(n^2\right)}{{[n^{2}b+{\lambda }_2+o(n^2\left)\right]}^2},$$

and

$$E\left({\Vert \widehat{\beta }({\lambda }_2,{\lambda }_1)-{\beta }_0\Vert }_2^2\right)\le 4\frac{{\lambda }_2^2{\Vert {\beta }_0\Vert }_2^2+n^3\mathit{pB}+{\lambda }_1^{2}p+o\left(n^2\right)}{{[n^{2}b+{\lambda }_2+o(n^2\left)\right]}^2}.$$

Remark

Note that the first bound is related to the estimator in (8). The second bound is related to the estimator in (9). ${\widehat{\beta }}_w$ is related to the adaptive elastic net estimator in (1), and $\widehat{\beta }({\lambda }_1,{\lambda }_2)$ is related to the estimator in (2). Even though ${\Vert {\beta }_0\Vert }_2^2=O\left(p\right)$, and p → ∞, the bound depends on ${\lambda }_2^2{\Vert {\beta }_0\Vert }_2^2∕n^2\to 0$ in large samples by Assumptions 3(ii) and 5. Also ${\lambda }_1^{2}E{\sum }_{j=1}^p{\widehat{w}}_j^2$, is dominated by n⁴ in the denominator in large samples as seen in the proof of Theorem 3(i) below. It is clear from the last result that the elastic net estimator is converging at the rate of $\sqrt{n∕p}$.

Theorem 2 extends the least squares case of Theorem 3(i) in Zou and Zhang (2009) to the nonlinear GMM case. The risk bounds are different from their case due to the nonlinear nature of our problem. The partial derivative of the sample moment depends on parameter estimates in our case which complicates the proofs.

Write ${\beta }_0={({\beta }_{A,0}^{\prime },0_{p-p_A}^{\prime })}^{\prime }$ where β_A,0 represents the vector of nonzero parameters (true values) Its dimension grows with the sample size, and 0_p–A vector of p – p_A elements represents the zero (redundant) parameters. Let β_A represent the nonzero parameters and of dimension p_A × 1.

Then define

$$\tilde{\beta }=\arg \underset{\beta }{\min }\left\{{\left[\sum _{i=1}^n{g}_i\right({\beta }_A\left)\right]}^{\prime }{W}_n\right[\sum _{i=1}^n{g}_i\left({\beta }_A\right)]+{\lambda }_2\sum _{j\in A}{\beta }_j^2+{\lambda }_1^{\ast }\sum _{j\in A}{\widehat{w}}_j\mid {\beta }_j\mid \},$$

where A = {j : β_j0 ≠ 0, j = 1, 2, ⋯, p}. Our next goal is to show that, with probability one, $\left[\right(1+{\lambda }_2∕n)\tilde{\beta },0_{p-p_A}]$ converges to the solution of the adaptive elastic net estimator in (1).

Theorem 3

Given Assumptions 1-3, and 5,

(i). with probability tending to one, $\left(\right(1+\frac{{\lambda }_2}{n})\tilde{\beta },0)$ is the solution to (1).

(ii). (Consistency in Selection) we also have

$$P\left(\right\{j:{\widehat{\beta }}_{\mathit{aenet},j}\ne 0\}=A)\to 1.$$

Remarks

1. Theorem 3(i) shows that ideal estimator $\tilde{\beta }$ becomes the same as the adaptive elastic net estimator in large samples. So the GMM elastic adaptive net estimator has the same solution as $\left(\right(1+{\lambda }_2∕n)\tilde{\beta },0_{p-p_A})$. Theorem 3(ii) shows that the nonzero adaptive elastic net estimates display the oracle property with Theorem 4 below. This is a sharper result than the one in Theorem 3(i). This is an important extension of the least squares case of Theorems 3.2, 3.3 of Zou and Zhang (2009) to the GMM estimation.

2. We allow for local to zero parameters and also provide an assumption when they may be considered as nonzero. This is Assumption 5 (iii)-(iv), n^1–νη² → ∞, n^1–αη^γ → ∞ where q = n^ν, p = n^α, 0 < α ≤ ν 1. The implications of the assumptions on the magnitude of the smallest nonzero coefficient is discussed after Assumptions. The proof of Theorem 3(ii) clearly shows that, as long as Assumption 5 is satisfied, the model selection for local to zero coefficients is possible. However, the local to zero coefficients cannot be arbitrarily close to zero in order to be selected. This is well established in Leeb and Pőtscher (2005). Leeb and Pőtscher (2005) show, in their Proposition A1(2), as long as the order of local to zero coefficients is larger than n^−1/2 in magnitude they can be selected. So this is like a lower bound for nonzeros to be selected as zeros. Our Assumption 5 is the extension of their result to the GMM estimator with a diverging number of parameters. In the diverging parameter case, there is a tradeoff between the number of local to zero coefficients and the requirement on their order of the magnitude.

Now we provide the limit law for the estimates of the nonzero parameter values (true values). Denote the adaptive elastic net estimators that correspond to nonzero true parameter values as the vector ${\widehat{\beta }}_{\mathit{aenet},A}$, which is of dimension p_A × 1. Define a consistent variance estimator for nonzero parameters that can be derived from elastic net estimators as: ${\widehat{\Omega }}_{\ast }$. We also define ${\Omega }_{\ast }^{-1}$ as ${\Vert {\left[n^{-1}{\sum }_{i=1}^n{\mathit{Eg}}_i\right({\beta }_{A,0}\left)g_i{\left({\beta }_{A,0}\right)}^{\prime }\right]}^{-1}-{\Omega }_{\ast }^{-1}\Vert }_2^2\to 0$.

Theorem 4

Under Assumptions 1-5, given $W_n={\widehat{\Omega }}_{\ast }^{-1}$, set $W={\Omega }_{\ast }^{-1}$

$${\delta }^{\prime }{K}_n{\left[\widehat{G}{\left({\widehat{\beta }}_{\mathit{aenet},A}\right)}^{\prime }{\widehat{\Omega }}_{\ast }^{-1}\widehat{G}\right({\widehat{\beta }}_{\mathit{aenet},A}\left)\right]}^{1∕2}{n}^{-1∕2}({\widehat{\beta }}_{\mathit{aenet},A}-{\beta }_{A,0})\stackrel{d}{\to }N(0,1),$$

where $K_n=\left[\frac{I+{\lambda }_2{\left(\widehat{G}{\left({\widehat{\beta }}_{\mathit{aenet},A}\right)}^{\prime }{\widehat{\Omega }}_{\ast }^{-1}\widehat{G}\right({\widehat{\beta }}_{\mathit{aenet},A}\left)\right)}^{-1}}{1+{\lambda }_2∕n}\right]$ is a square matrix of dimension p_A and δ is a vector of Euclidean norm 1.

Remarks

1. First we see that

$${\Vert K_n-I_{p_A}\Vert }_2^2\stackrel{P}{\to }0,$$

due to Assumptions 1, 2, and λ₂ = o(n).

2. This theorem clearly extends Zou and Zhang (2009) from the least squares case to the GMM estimation. This result generalizes theirs to nonlinear functions of endogenous variables which are heavily used in econometrics and finance. The extension is not straightforward, since the new limit result depends on an explicit separate consistency proof unlike the least squares case of Zou and Zhang (2009). This is mainly because the partial derivative of the sample moment function depends on the parameter estimates, which is not shared by the least squares estimator. The limit that we derive also corresponds to the standard GMM limit in Hansen (1982), where the same result was derived for a fixed number of parameters with a well specified model. In this way, Theorem 4 also generalizes the result of Hansen (1982) to the direction of a large number of parameters with model selection.

3. Note that K_n term is a ridge regression like term which helps to handle the collinearity among the variables.

4. Note that if we set λ₂ = 0, we obtain the limit for adaptive Lasso GMM estimator. In that case K_n = I_pA, and

$${\delta }^{\prime }{\left(\widehat{G}{\left({\widehat{\beta }}_{\mathit{alasso},A}\right)}^{\prime }{\widehat{\Omega }}_{\ast }^{-1}\widehat{G}\right({\widehat{\beta }}_{\mathit{alasso},A}\left)\right)}^{1∕2}{n}^{-1∕2}({\widehat{\beta }}_{\mathit{alasso},A}-{\beta }_{A,0})\stackrel{d}{\to }N(0,1),$$

There will be discussions on how to choose the tuning parameters ${\lambda }_1,{\lambda }_2,{\lambda }_1^{\ast }$, and how to set the small parameter estimates to zero in finite samples in the simulation section.

5. Instead of Liapounov Central Limit Theorem, we can use Central Limit Theorem for stationary time series data. These already exist in Davidson (1994). Theorem 4 will proceed as before in the independent data case. When we define the GMM objective function, use sample moments as weighted in time. We conjecture that this result in same proofs for Theorems 1-3. This technique of weighting sample moments by time is used in Otsu (2006) and Guggenberger and Smith (2008).

6. After obtaining the adaptive elastic net GMM results, one can run the unpenalized GMM with nonzero parameters and conduct inference.

7. First from Remark 1, we have ${\Vert K_n-I_{p_A}\Vert }_2^2=o_p\left(1\right)$. Then ${\Vert \delta \Vert }_2={({\delta }_1^2+\cdots +{\delta }_{p_A}^2)}^{1∕2}=1$, and δ is a p_A vector. And then by Assumption 1 and consistency of the adaptive elastic net we have ${\Vert \widehat{G}\left({\widehat{\beta }}_{\mathit{aenet},A}\right)\Vert }_2=O_p\left(n^{1∕2}\right)$. These provide the rate of $\sqrt{n∕p_A}$ for the adaptive elastic net estimators.

4 Simulation

In this section we analyze the finite sample properties of the adaptive elastic net estimator for GMM. Namely, we evaluate its bias, the root mean squared error as well as the correct number of redundant versus relevant parameters. We have the following simultaneous equations for all i = 1, ⋯, n

$$\begin{array}{c}\hfill y_i=x_i^{\prime }{\beta }_0+{∊}_i,\hfill \\ \hfill x_i=z_i^{\prime }\pi +{\eta }_i,\hfill \\ \hfill {∊}_i=\rho {\iota }^{\prime }{\eta }_i+\sqrt{1-{\rho }^2}{\iota }^{\prime }{v}_i,\hfill \end{array}$$

where the number of instruments q is set to be equal to the number of parameters p, x_i is a p × 1 vector, z_i is a p × 1 vector, ρ = 0.5, and π is a square matrix of dimension p. Furthermore, η_i is iid N(0, I_p), v_i is iid with N(0, I_p), and Ι is a p × 1 vector of ones.

The model that is estimated:

$${\mathit{Ez}}_i{∊}_i=0,$$

for all i = 1, ⋯, n.

We have two different designs for the parameter vector β₀. In the first case β₀ = {3, 3, 0, 0, 0} (Design 1), and in the second one β₀ = {3, 3, 3, 3, 0} (Design 2). We have n = 100, and z_i ≅ N(0, Ω_z) for all i = 1, ⋯, n, and

$${\Omega }_z=\left[\begin{array}{ccccc}\hfill 1\hfill & \hfill 0.5\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0.5\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill \end{array}\right].$$

So there is correlation between z_i’s and this affects the correlation between x_i’s since two equations are correlated. In this section, we compare three methods: GMM-BIC of Andrews and Lu (2001), Bridge-GMM of Caner (2009), and the adaptive elastic net GMM. We use four different measures to compare them here. First, we look at the percentage of correct models selected. Then we evaluate the following summary Mean Squared Error (MSE)

$$E\left[{(\widehat{\beta }-{\beta }_0)}^{\prime }{\Sigma }_{∊}\right(\widehat{\beta }-{\beta }_0\left)\right],$$ (10)

where ${\Sigma }_{∊}=E{∊}_i{∊}_i^{\prime }$, and $\widehat{\beta }$ represents the estimated coefficient vector given by three different methods. This measure is commonly used in statistics literature; see Zou and Zhang (2009). The other two measures are concerned about individual coefficients. First, the bias of each individual coefficient estimate is measured. Then the root mean squared error of each coefficient is computed. We use 10,000 iterations.

Truncation of small coefficient estimates is set to zero via $\mid {\widehat{\beta }}_{\mathit{Bridge}}\mid <2∕\lambda$ for Bridge-GMM as suggested in Caner (2009). For the adaptive elastic net, we use the modified shooting algorithm given in Appendix 2 of Zhang and Lu (2007). Least Angle Regression (LAR) is not used because it is not clear whether it is useful in the GMM context.

This modified shooting algorithm amounts to using Kuhn-Tucker conditions for a corner solution. First, the absolute value of the partial derivative of the GMM objective (unpenalized one) with respect to the parameter of interest is evaluated at zero for that parameter, and for the rest at the current adaptive elastic net estimates. If this is less than ${\lambda }_1^{\ast }∕{\mid {\widehat{\beta }}_{\mathit{enet}}\mid }^{4.5}$, then we set that parameter to zero. We have also tried slightly larger exponents than 4.5, and observed that the results are not affected much. Note that the reason for a large γ comes from Assumption 3(iii). This is similar to the adaptive lasso case used in Zhang and Lu (2007).

The choice of λ’s in both Bridge-GMM and the adaptive elastic net GMM is done via BIC. This is suggested by Zou, Hastie, and Tibshirani (2007), and as well as by Wang, Li, and Tsai (2007). Specifically, we use the following BIC from Wang, Li, Leng (2009). For each pair of ${\lambda }_s=({\lambda }_1^{\ast },{\lambda }_2)\in \Lambda$

$$\mathit{BIC}\left({\lambda }_s\right)=\log \left(\mathit{SSE}\right)+\mid A\mid \frac{\log n}{n},$$

where ∣A∣ is the cardinality of the set A. $\mathit{SSE}={\left[n^{-1}{\sum }_{i=1}^n{g}_i\right(\widehat{\beta }\left)\right]}^{\prime }{W}_n\left[n^{-1}{\sum }_{i=1}^n{g}_i\right(\widehat{\beta }\left)\right]$ Basically, given a specific λ_s, we analyze how many nonzero coefficients are in the estimator and use this to calculate the cardinality of A, and for that choice compute SSE. The final λ is chosen as

$${\widehat{\lambda }}_s=\arg \underset{{\lambda }_s\in \Lambda }{\min }\mathit{BIC}\left({\lambda }_s\right),$$

where Λ represents a finite number of possible values of λ_s.

The Bridge-GMM estimator in Caner (2009) is $\widehat{\beta }$ that minimizes U_n(β), where

$$U_n\left(\beta \right)={\left[\sum _{i=1}^n{g}_i\right(\beta \left)\right]}^{\prime }{W}_n\left[\sum _{i=1}^n{g}_i\right(\beta \left)\right]+\lambda \sum _{j=1}^p{\mid {\beta }_j\mid }^{\gamma },$$ (11)

for a given positive regularization parameter λ and 0 < γ < 1.

We now describe the model selection by GMM-BIC proposed in Andrews and Lu (2001). Let b ∈ R^p denote a model selection vector. By definition, each element of b is either zero or one. If the jth element of b is one, the corresponding β_j is to be estimated; if the jth element of b is zero we set β_j to be zero. We set ∣b∣ as the number of parameters to be estimated, or in the equivalent form, $\mid b\mid ={\sum }_{j=1}^p\mid b_j\mid$. We then set β_[b] as the p × 1 vector representing the element by the element (Hadamard) product of β and b. The model selection will be based on the GMM objective function and a penalty term. The objective function in BIC benefits from:

$$J_n\left(b\right)={\left[\sum _{i=1}^n{g}_i\right({\beta }_{\left[b\right]}\left)\right]}^{\prime }{W}_n\left[\sum _{i=1}^n{g}_i\right({\beta }_{\left[b\right]}\left)\right],$$ (12)

where in the simulation

$$g_i\left({\beta }_{\left[b\right]}\right)=z_i(y_i-x_i^{\prime }{\beta }_{\left[b\right]}).$$

The model selection vectors “b” in our case represent 31 different possibilities (excluding the all-zero case). The following are the possibilities for all “b” vectors

$$M=[M_{11},M_{12},M_{13},M_{14},M_{15}],$$

where M₁₁ is the identity matrix of dimension 5, I₅, which represents all the possibilities with only one nonzero coefficient. M₁₂ represents all the possibilities with two nonzero coefficients,

$$M_{12}=\left(\begin{array}{cccccccccc}\hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 1\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 1\hfill \end{array}\right).$$ (13)

In the same way, M₁₃ represents all possibilities with 3 nonzero coefficients, M₁₄ represents all the possibilities with four nonzero coefficients, and M₁₅ is the vector of ones, showing all nonzero coefficients. The true model in Design 1 is the first column vector in M₁₂. For design 2, the true model is in M₁₄ and that is (1, 1, 1, 1, 0)’.

The GMM-BIC selects the model based on minimizing the following criterion among the 31 possibilities

$$J_n\left(b\right)+\mid b\mid \log \left(n\right).$$ (14)

The penalty term penalizes larger models more. Denote the optimal model selection vector by b*. After selecting the optimal model in (14), the vector b*, we then estimate the model parameters by the GMM. Next we present the results on Tables 1-4 for these three techniques that are examined in the simulation section. In Table 1, we provide the correct model selection percentages for Designs 1 and 2. We see that both Bridge and Adaptive Elastic Net are doing very well. The Bridge-GMM selects the correct model 100%, and the Adaptive Elastic Net 91 – 95% of the time, whereas the GMM-BIC selects only 0 – 6.9%. This is due to lots of possibilities in the case of GMM-BIC, and with a large number of parameters the performance of GMM-BIC tends to deteriorate. Table 2 provides a summary of MSE measure results. This clearly shows that the Adaptive Elastic Net estimator is the best among the three, since its MSE figures are the smallest. The GMM-BIC is much worse in terms of MSE, due to its wrong model selection, and after the model selection estimating the zero coefficients with nonzero and large magnitudes. Tables 3 and 4 provide the bias and root mean squared error (MSE) of each coefficient in Designs 1 and 2. Comparing the Bridge with the Adaptive Elastic Net, we observe that the bias of the nonzero coefficients are generally smaller for the Adaptive Elastic Net. The same is true for generally in the case of root mean squared errors, which are smaller for the nonzero coefficients in the Adaptive Elastic Net estimator.

Table 1.

Success Percentages of Selecting the Correct Model

Estimators	Design 1	Design 2
Adaptive Elastic Net	91.2	94.9
Bridge-GMM	100.0	100.0
GMM-BIC	6.9	0.0

Note: The GMM-BIC (Andrews and Lu, 2001) represents the models that are selected according to BIC and subsequently we use GMM. The Bridge-GMM estimator is studied in Caner (2009). The Adaptive Elastic Net estimator is the new procedure proposed in this study.

Table 4.

Bias, RMSE of Design 2

	Adaptive Elastic Net		Bridge-GMM		GMM-BIC
	BIAS	RMSE	BIAS	RMSE	BIAS	RMSE
β 1	−0.181	0.193	−0.112	0.124	−0.805	158.171
β 2	−0.181	0.193	−0.662	0.665	−0.326	112.970
β 3	0.010	0.061	0.157	0.166	−0.314	120.358
β 4	−0.038	0.071	0.337	0.341	−6.759	659.673
β 5	−0.001	0.007	0.000	0.000	7.740	617.509

Note: The GMM-BIC (Andrews and Lu, 2001) represents models that are selected according to BIC and subsequently we use GMM. The Bridge-GMM estimator is studied in Caner (2009). The Adaptive Elastic Net estimator is the new procedure proposed in this study.

Table 2.

Summary Mean Squared Error (MSE)

Estimators	Design 1	Design 2
Adaptive Elastic Net	1.8	1.3
Bridge-GMM	4.2	1.3
GMM-BIC	165848.5	876080.2

Note: The MSE formula is given in (10). Instead of expectations, the average of iterations are used. A small number for summary MSE is desirable for a model. The GMM-BIC (Andrews and Lu, 2001) represents models that are selected according to BIC and subsequently we use GMM. The Bridge-GMM estimator is studied in Caner (2009). The Adaptive Elastic Net is the new procedure proposed in this study.

Table 3.

Bias and RMSE Results of Design 1

	Adaptive Elastic Net		Bridge-GMM		GMM-BIC
	BIAS	RMSE	BIAS	RMSE	BIAS	RMSE
β 1	−0.244	0.272	−0.117	0.126	2.903	159.85
β 2	−0.244	0.272	−0.667	0.669	−4.082	261.32
β 3	0.013	0.042	0.000	0.000	−0.859	158.839
β 4	0.000	0.009	0.000	0.000	0.612	188.510
β 5	0.013	0.041	0.000	0.000	1.162	62.240

Note: The GMM-BIC (Andrews and Lu, 2001) represents models that are selected according to BIC and subsequently we use GMM. The Bridge-GMM estimator is studied in Caner (2009). The Adaptive Elastic Net estimator is the new procedure proposed in this study.

To get confidence intervals for nonzero parameters, one can run the adaptive elastic net first and find the zero and nonzero coefficients. Then for those nonzero estimates, we have the standard GMM standard errors by Theorem 4. By using that we can calculate confidence intervals for nonzero coefficient parameters.

5 Application

In this part we go through a useful application of the new estimator. The following is the external habit specification model considered by Chen and Ludvigson (2009) (also equation (2.7) in Chen (2007)):

$$E\left({\iota }_0{\left(\frac{C_{t+1}}{C_t}\right)}^{-{\varphi }_0}\frac{{(1-h_0(\frac{C_t}{C_{t+1}}\left)\right)}^{-{\varphi }_0}}{{(1-h_0(\frac{C_{t-1}}{C_t}\left)\right)}^{-{\varphi }_0}}{R}_{l,t+1}-1\mid z_t\right)=0,$$

where C_t represents the consumption at time t, and Ι₀ and ϕ₀ are both positive and they represent time discount and curvature of the utility function respectively. R_l,t+1 is the lth asset return at time t + 1, h₀(.) ∈ [0, 1) is an unknown habit formation function, and z_t is the information set and this will be linked to valid instruments. We took only one lag in consumption ratio, rather than several of them. The possibility of this specific model is mentioned in p.1069 of Chen and Ludvigson(2009). Chen and Ludvigson (2009) use the sieve estimation to estimate the unknown h₀ function. They setup the dimension of the sieve as a given number. In this paper we use the adaptive elastic net GMM to automatically select the dimension of sieve and estimate the structural parameters at the same time. The parameters and the unknown habit function that we try to estimate are δ₀, γ₀, h₀. Now denote

$$\rho (C_t,R_{l,t+1},{\iota }_0,{\varphi }_0,h_0)={\iota }_0{\left(\frac{C_{t+1}}{C_t}\right)}^{-{\varphi }_0}\frac{{(1-h_0(\frac{C_t}{C_{t+1}}\left)\right)}^{-{\varphi }_0}}{{(1-h_0(\frac{C_{t-1}}{C_t}\left)\right)}^{-{\varphi }_0}}{R}_{l,t+1}-1.$$

Before setting up the orthogonality restrictions, set s_0j(z_t) is a sequence of known basis functions that can approximate any square integrable function. Then for each l = 1, ⋯, N, j = 1, ⋯, J_T, the restrictions are

$$E\left[\rho \right(C_t,R_{l,t+1},{\iota }_0,{\varphi }_0,h_0\left)s_{0j}\right(z_t\left)\right]=0.$$

In total we have NJ_T restrictions, and N is fixed, but J_T → ∞, as T → ∞, and NJ_T/T → 0, as T → ∞. The main issue is the approximation of the unknown function h₀. Chen and Ludvigson (2009) use sieves to approximate that function. In theory the dimension of sieve K_T → ∞, but K_T/T → 0, as T → ∞. As Chen and Ludvigson (2009) we use an artificial neural network sieve approximation,

$$h\left(\frac{C_{t-1}}{C_t}\right)={\zeta }_0+\sum _{j=1}^{K_T}{\zeta }_j\Psi ({\tau }_j\frac{C_{t-1}}{C_t}+{\kappa }_j),$$

where Ψ(.) is an activation function, and this is chosen as a logistic function Ψ(x) = (1 + e^−x)⁻¹. This implies that in order to estimate the habit function, we need 3K_T + 1 parameters. The parameters are ζ₀, ζ_j, τ_j, κ_j, j = 1, ⋯, K_T. Chen and Ludvigson (2009) use K_T = 3. In our paper, along with parameters δ₀, γ₀, estimation of h₀ through selection of correct sieve dimension will be done. So if true dimension of sieve is K_T0, with 0 ≤ K_T0 ≤ K_T, then the Adaptive Elastic Net GMM aims to estimate that dimension (through estimation of parameters in the habit function). The total number of parameters to be estimated is p = 3(K_T + 1), since we also estimate Ι₀, ϕ₀ in addition to the habit function parameters. The number of orthogonality restrictions is q = NJ_T, and we assume q = NJ_T ≥ 3(K_T + 1) = p. Chen and Ludvigson (2009) use sieve minimum distance estimator, and Chen (2007) uses sieve GMM estimator to estimate the parameters. Specifically, equation (2.16) in Chen (2007) use unpenalized sieve GMM estimation for this problem. Instead we will assume that true dimension of the sieve is unknown and will estimate the parameters along with habit function (parameters in that function) with the adaptive elastic net GMM estimator. Set β = (Ι, τ, h) and the compact sieve space is B_p = B_δ×B_γ×H_T. The compactness assumption is discussed in p.1067 of Chen and Ludvigson (2009), which is mainly needed so that sieve parameters do not generate tail observations on Ψ(.). Also set the approximating known basis functions as s(z_t) = (s_0,1(z_t, ⋯, s_0,JT(z_t))’, which is a J_T × 1 vector. So J_T = 3, there are three instruments. These are a constant, lagged consumption growth, and its square¹. There are seven asset returns that are used in the study so N = 7. The detailed explanations can be found in Chen and Ludvigson (2009).

Implementation Details

1. First, we run the elastic net GMM to obtain the adaptive weights w_j’s. The elastic net GMM has the same objective function as the adaptive version but with w_j = 1 for all j. The enet-GMM estimator is obtained by setting the weights as 1 in the estimator in the third step given as follows.

2.Then for the weights, since a priori it is known that nonzero coefficients cannot be large positive, we use γ = 2.5 in the exponent. For specifying the weights, $w_j=1∕{\mid {\widehat{\beta }}_{\mathit{enet},j}\mid }^{2.5}$ is chosen for all j. We have also experimented with γ = 4.5 as in the simulations, and the results were mildly different but qualitatively very similar, so those are not reported.

3. Our adaptive elastic net GMM estimator is:

$$\begin{array}{cc}\hfill \widehat{\beta }& =(1+{\lambda }_2∕T)\arg \underset{\beta \in B_p}{\min }\left\{{\left[\sum _{t=1}^T\rho \right(C_t,R_{t+1},\beta \left)\otimes s\right(w_t\left)\right]}^{\prime }\widehat{W}\right[\sum _{t=1}^T\rho (C_t,R_{t+1},\beta )\otimes s\left(w_t\right)]\hfill \\ \hfill +& {\lambda }_1^{\ast }\sum _{j=1}^{3(K_T+1)}{\widehat{w}}_j\mid {\beta }_j\mid +{\lambda }_2\sum _{j=1}^{3(K_T+1)}{\beta }_j^2\},\hfill \end{array}$$

where β₁ = ι, β₂ = ϕ, and the remaining 3K_T + 1 parameters correspond to the habit function estimation by sieves. We use the following weight to make the comparison with Chen and Ludvigson (2009) in a fair way:

$$\widehat{W}=I\otimes {\left(S^{\prime }S\right)}^{-},$$

where S is (s(z₁), ⋯, s(z_t), ⋯, s(z_T))’, which is T × 3 matrix, where we use Moore-Penrose inverse as described in (2.16) of Chen (2007). Note that ρ(C_t, R_l,t+1, β) is an N × 1 vector, and

$$\rho (C_t,R_{t+1},\beta )={\left(\rho \right(C_t,R_{1,t+1},\beta ),\cdots ,\rho (C_t,R_{l,t+1},\beta ),\cdots ,\rho (C_t,R_{N,t+1},\beta \left)\right)}^{\prime }.$$

After the implementation steps we explain the data here. The data points start from the second quarter of 1953 and end at the second quarter of 2001. This is a slightly shorter span than Chen and Ludvigson (2009) since we did not want to use missing data cells for certain variables. So NJ_T = 21 (number of orthogonality restrictions). At first we try K_T = 3 like Chen and Ludvigson (2009), but this is the maximal number of sieve dimension, and we estimate sieve parameter with structural parameters and select the model unlike Chen and Ludvigson (2009). We also try K_T = 5. When K_T = 3, the total number of parameters to be estimated is 12, and if K_T = 5, then this number is 18. We also use BIC to choose from three possible tuning parameter choices, ${\lambda }_1={\lambda }_1^{\ast }={\lambda }_2=\{0.000001,1,10\}$. The tuning parameters are taking the same value for ease of computation. So here we will compare our results with unpenalized sieve GMM of Chen (2007). As discussed above we apply a certain subset of the instruments, since the remainder are unavailable, and do not use missing data in Chen and Ludvigson (2009). So our results corresponding to unpenalized sieve GMM will be slightly different than Chen and Ludvigson (2009).

We provide the estimates for our adaptive elastic net GMM method. This is for the case of K_T = 5. The time discount estimate $\widehat{\iota }=0.88$ and the curvature of the utility curve parameter is $\widehat{\varphi }=0.66$. The sieve parameter estimates are ${\widehat{\zeta }}_0=0$, ${\widehat{\zeta }}_j=0,0,0,0,0$, ${\widehat{\tau }}_j=0.086,0.078,0.084,0.073,0.083$, ${\widehat{\kappa }}_j=0.082,0.072,0.087,0.086,0.080$, for j = 1, 2, 3, 4, 5 respectively. To compare if we use sieve GMM with imposing K_T = 5 as the true dimension of the sieve, we get ${\widehat{\iota }}_{\mathit{sg}}=0.86$, ${\widehat{\varphi }}_{\mathit{sg}}=0.73$ for time discount, and curvature parameter, where subscript sg denotes Chen and Ludvigson (2009), Chen (2007) with ${\lambda }_1={\lambda }_1^{\ast }={\lambda }_2=0$. So the results are basically the same as ours for these two parameters. However, the estimates of sieve parameters in unpenalized sieve GMM case is ${\widehat{\zeta }}_{\mathit{sg},0}=0$, ${\widehat{\zeta }}_{\mathit{sg},j}=0$ for all j = 1, ⋯, 5, and ${\widehat{\tau }}_{\mathit{sg},j}=0.083,0.079,0.079,0.079,0.076$, ${\widehat{\kappa }}_{\mathit{sg},j}=0.089,0.086,0.086,0.084,0.082$ respectively for j = 1, ⋯, 5. So the habit function in the sieve GMM with K_T = 5 is estimated as zero (on the boundary), our method gives the same result.

In Chen and Ludvigson (2009), they fit KT = 3 for the sieve. We provide the estimates for our adaptive elastic net GMM method in this case as well. We also reestimate Chen and Ludvigson (2009). In the adaptive elastic net GMM, the time discount estimate is $\widehat{\iota }=0.93$ and the curvature of the utility curve parameter is $\widehat{\varphi }=0.64$. The sieve parameter estimates are ${\widehat{\zeta }}_0=0$, ${\widehat{\zeta }}_j=0$ for j = 1, 2, 3 ${\widehat{\tau }}_j=0.057,0.054,0.064$, ${\widehat{\kappa }}_j=0.067,0.066,0.058$, for j = 1, 2, 3 respectively. To compare with our method we use the sieve GMM of Chen and Ludvigson (2009) with imposing K_T = 3 as the true dimension of the sieve, we get ${\widehat{\iota }}_{\mathit{sg}}=0.94$, ${\widehat{\varphi }}_{\mathit{sg}}=0.71$ for time discount, and curvature parameter, where subscript sg denotes Chen and Ludvigson (2009), Chen (2007) with ${\lambda }_1={\lambda }_1^{\ast }={\lambda }_2=0$. So the results are again basically the same as ours for these two parameters. However, the estimates of sieve parameters in unpenalized sieve GMM case is ${\widehat{\zeta }}_{\mathit{sg},0}=0$, ${\widehat{\zeta }}_{\mathit{sg},j}=0.022,0.025,0.019$ for all j = 1, ⋯, 3, and ${\widehat{\tau }}_{\mathit{sg},j}=0.051,0.055,0.056$, ${\widehat{\kappa }}_{\mathit{sg},j}=0.076,0.075,0.075$, respectively for j = 1, ⋯, 3. So the habit function in the adaptive elastic net GMM with K_T = 3 is estimated as zero (on the boundary), but with the sieve GMM , the estimate of the habit function is positive. In Chen and Ludvigson article (2009), with a larger instrument set than we used for their case, they find the time parameter estimate to be 0.99, and curvature to be 0.76, and the habit function was positive at K_T = 3. Note that we use time series data, and as we suggest after our theorems, this is plausible given our technique and the structure of the proofs.

We now discuss how our assumptions fit this application. First, all of our parameters are uniformly bounded in this application, which is discussed in p.1067 of Chen and Ludvigson (2009). Then the second issue is whether the uniform convergence of partial derivative is plausible. This will be satisfied in iid case through Condition 3.5M in Chen (2007). This amounts to uniformly bounded partial derivatives, Lipschitz continuity of partial derivatives, and log of covering numbers to be growing less than rate T.

Assumption 2 is related to convergence of the weight matrix, which is not restrictive, and it shows a relation between q and T, so q cannot grow fast. In our case q = NJ_T, where N is fixed, so this restricts the growth of J_T. Assumption 3(i) is also similar to Assumption 2. For Assumption 3(ii), we have p = 3(K_T + 1), since q = NJ_T ≥ 3(K_T + 1) = p, given that NJ_T/T → 0 provides us the assumption. Assumption 3(iii) is satisfied here by imposing a value between 0 and 1 (including one) for the exponent in the elastic net based weights. Assumption 4 is concerned about the sample moment functions, since all of our variables are stationary, in terms of ratios, and bounded variables such as returns, we do not expect its 2 + l moment to grow larger than T^1/2. Assumption 5 is a penalty function, and this is satisfied with using T in place of n.

6 Conclusion

In the paper here we analyze the adaptive elastic net GMM estimator. It can simultaneously select the model and estimate that. The new estimator also allows for a diverging number of parameters. The estimator is shown to have the oracle property, so we can estimate nonzero parameters with the standard GMM limit and the redundant parameters are set to zero by a data-dependent technique. Commonly used AIC, BIC methods as well as our estimator face some non uniform consistency issues of the estimators. If we have to select the model with AIC or BIC and then use GMM , this also has the non uniform consistency issues and also does much worse than the adaptive elastic net estimator considered here. The issue with model selection (i.e. including AIC, BIC) is that all of them are uniformly inconsistent (from model selection perspective). So any arbitrarily local to zero coefficients cannot be selected as nonzero. Leeb and Pőtscher (2005) establish that, in order to get selected, the order of the magnitude of local to zero coefficients should be larger than n^−1/2. Between 0 and the magnitude of n^−1/2, the model selection is indeterminate. We study the selection issue of local to zero coefficients in the GMM and extend the results of Leeb and Pőtscher (2005) to the GMM with a diverging number of parameters.

Appendix.

Proof of Theorem 1(i)

Huang, Horowitz and Ma (2008) analyzed the least squares with Bridge penalty with a diverging number of parameters. Here we extend that to a diverging number of moment conditions with parameters and the Adaptive Elastic Net penalty in nonlinear GMM.

Starting with ${\widehat{\beta }}_{\mathit{enet}}$ definition

$$(1+{\lambda }_2∕n)S_n\left({\widehat{\beta }}_{\mathit{enet}}\right)\le (1+{\lambda }_2∕n)S_n\left({\beta }_0\right)$$

which is

$${\left[\sum _{i=1}^n{g}_i\right({\widehat{\beta }}_{\mathit{enet}}\left)\right]}^{\prime }{W}_n\left[\sum _{i=1}^n{g}_i\right({\widehat{\beta }}_{\mathit{enet}}\left)\right]+{\lambda }_1\sum _{j=1}^p\mid {\widehat{\beta }}_{j,\mathit{enet}}\mid +{\lambda }_2{\Vert {\widehat{\beta }}_{\mathit{enet}}\Vert }_2^2\le {\left[\sum _{i=1}^n{g}_i\right({\beta }_0\left)\right]}^{\prime }{W}_n\left[\sum _{i=1}^n{g}_i\right({\beta }_0\left)\right]+{\lambda }_1\sum _{j=1}^p\mid {\beta }_{j0}\mid +{\lambda }_2{\Vert {\beta }_0\Vert }_2^2.$$ (15)

Then setting ${\iota }_n={\lambda }_1{\sum }_{j=1}^p\mid {\beta }_{j0}\mid +{\lambda }_2{\Vert {\beta }_{j0}\Vert }_2^2={\lambda }_1{\sum }_{j\in A}\mid {\beta }_{j0}\mid +{\lambda }_2{\sum }_{j\in A}{\beta }_{j0}^2$, and by (15)

$${\iota }_n\ge {\left[\sum _{i=1}^n{g}_i\right({\widehat{\beta }}_{\mathit{enet}}\left)\right]}^{\prime }{W}_n\left[\sum _{i=1}^n{g}_i\right({\widehat{\beta }}_{\mathit{enet}}\left)\right]-{\left[\sum _{i=1}^n{g}_i\right({\beta }_0\left)\right]}^{\prime }{W}_n\left[\sum _{i=1}^n{g}_i\right({\beta }_0\left)\right]$$ (16)

$$=-2{\left[{\widehat{G}}_n\right(\stackrel{‒}{\beta }\left)\right({\beta }_0-{\widehat{\beta }}_{\mathit{enet}}\left)\right]}^{\prime }{W}_n\left[\sum _{i=1}^n{g}_i\right({\beta }_0\left)\right]+{\left[{\widehat{G}}_n\right(\stackrel{‒}{\beta }\left)\right({\beta }_0-{\widehat{\beta }}_{\mathit{enet}}\left)\right]}^{\prime }{W}_n\left[{\widehat{G}}_n\right(\stackrel{‒}{\beta }\left)\right({\beta }_0-{\widehat{\beta }}_{\mathit{enet}}\left)\right],$$ (17)

via the mean value theorem and $\stackrel{‒}{\beta }\in ({\beta }_0,{\widehat{\beta }}_{\mathit{enet}})$. We now try to simplify (17). In this way, set ${\Delta }_n={\mathit{nW}}_n^{1∕2}\left[\frac{{\sum }_{i=1}^n{G}_i\left(\stackrel{‒}{\beta }\right)}{n}\right]({\beta }_0-{\widehat{\beta }}_{\mathit{enet}})$ and $D_n=n^{1∕2}{W}_n^{1∕2}\left[\frac{{\sum }_{i=1}^n{g}_i\left({\beta }_0\right)}{n^{1∕2}}\right]$ and $G_i\left(\stackrel{‒}{\beta }\right)=\partial g_i\left(\stackrel{‒}{\beta }\right)∕\partial {\beta }^{\prime }$ which is a q × p matrix. The next two equations (18)-(20) are from P.603 Huang, Horowitz and Ma (2008). We use them to illustrate our point. Then it is clear that (17) can be written as

$${\Delta }_n^{\prime }{\Delta }_n-2D_n^{\prime }{\Delta }_n-{\iota }_n\le 0,$$ (18)

which provides us

$${\Vert {\Delta }_n-D_n\Vert }_2^2-{\Vert D_n\Vert }_2^2-{\iota }_n\le 0.$$

Using the last inequality, we deduct

$${\Vert {\Delta }_n-D_n\Vert }_2\le {\Vert D_n\Vert }_2+{\iota }_n^{1∕2},$$

and by triangle inequality

$${\Vert {\Delta }_n\Vert }_2\le {\Vert {\Delta }_n-D_n\Vert }_2+{\Vert D_n\Vert }_2\le 2{\Vert D_n\Vert }_2+{\iota }_n^{1∕2}.$$ (19)

By using (18)(19) with simple algebra, we have

$${\Vert {\Delta }_n\Vert }_2^2\le 6{\Vert D_n\Vert }_2^2+3{\iota }_n.$$ (20)

Note

$$\mathit{nE}\mid {\left(\frac{\sum _{i=1}^n{g}_i\left({\beta }_0\right)}{n^{1∕2}}\right)}^{\prime }{W}_n\left(\frac{\sum _{i=1}^n{g}_i\left({\beta }_0\right)}{n^{1∕2}}\right)\mid -\mathit{nE}\mid {\left(\frac{\sum _{i=1}^n{g}_i\left({\beta }_0\right)}{n^{1∕2}}\right)}^{\prime }{\Omega }^{-1}\left(\frac{\sum _{i=1}^n{g}_i\left({\beta }_0\right)}{n^{1∕2}}\right)\mid =o\left(1\right),$$ (21)

with probability approaching one, by substituting W = Ω⁻¹ and by Assumptions 2-3. Then by the definition of D_n and (21)

$$E{\Vert D_n\Vert }_2^2-\mathit{nE}\mid {\left(\frac{\sum _{i=1}^n{g}_i\left({\beta }_0\right)}{n^{1∕2}}\right)}^{\prime }{\Omega }^{-1}\left(\frac{\sum _{i=1}^n{g}_i\left({\beta }_0\right)}{n^{1∕2}}\right)\mid =o\left(1\right).$$

Next,

$$\begin{array}{cc}\hfill E\mid {\left(\frac{\sum _{i=1}^n{g}_i\left({\beta }_0\right)}{n^{1∕2}}\right)}^{\prime }{\Omega }^{-1}\left(\frac{\sum _{i=1}^n{g}_i\left({\beta }_0\right)}{n^{1∕2}}\right)\mid & =\mathit{tr}\left(E\left[\left(\frac{\sum _{i=1}^n{g}_i\left({\beta }_0\right)}{n^{1∕2}}\right){\left(\frac{\sum _{i=1}^n{g}_i\left({\beta }_0\right)}{n^{1∕2}}\right)}^{\prime }\right]{\Omega }^{-1}\right)\hfill \\ \hfill & =\mathit{tr}\left(I_q\right)=q,\hfill \end{array}$$ (22)

Using (22) in (21) with the definition of D_n, with probability approaching one

$$E{\Vert D_n^2\Vert }_2^2=O\left(\mathit{nq}\right).$$ (23)

Next by the definition of ∥Δ_n∥ and Assumption 2, we have

$$\begin{array}{cc}\hfill {\Vert {\Delta }_n\Vert }_2^2& =n^2\left[{({\beta }_0-{\widehat{\beta }}_{\mathit{enet}})}^{\prime }\left[{\left(\frac{\sum _{i=1}^n{G}_i\left(\stackrel{‒}{\beta }\right)}{n}\right)}^{\prime }{\Omega }^{-1}\left(\frac{\sum _{i=1}^n{G}_i\left(\stackrel{‒}{\beta }\right)}{n}\right)\right]({\beta }_0-{\widehat{\beta }}_{\mathit{enet}})\right]\hfill \\ \hfill & \ge n^2{\text{Eig}}_{\min }\left[{\left(\frac{\sum _{i=1}^n{G}_i\left(\stackrel{‒}{\beta }\right)}{n}\right)}^{\prime }{\Omega }^{-1}\left(\frac{\sum _{i=1}^n{G}_i\left(\stackrel{‒}{\beta }\right)}{n}\right)\right]{\Vert {\widehat{\beta }}_{\mathit{enet}}-{\beta }_0\Vert }_2^2\hfill \\ \hfill & \ge n^{2}b{\Vert {\widehat{\beta }}_{\mathit{enet}}-{\beta }_0\Vert }_2^2,\hfill \end{array}$$ (24)

with probability approaching one and seeing (6) with remembering that elastic net is a subcase of ${\widehat{\beta }}_w$. Next use (23)(24) in (20) to have

$$n^2\mathit{bE}{\Vert {\widehat{\beta }}_{\mathit{enet}}-{\beta }_0\Vert }_2^2\le O\left(\mathit{nq}\right)+O({\lambda }_1{p}_A+{\lambda }_2{p}_A),$$ (25)

by seeing that ${\lambda }_1{\sum }_{j\in A}\mid {\beta }_{j0}\mid +{\lambda }_2{\sum }_{j\in A}{\beta }_{j0}^2=O({\lambda }_1{p}_A+{\lambda }_2{p}_A)$, given that nonzero parameters can be all constants at most. So by Assumptions 3 and 5, since λ₁/n → 0, λ₂/n → 0, p_A/n → 0, q/n → 0, we have

$${\Vert {\widehat{\beta }}_{\mathit{enet}}-{\beta }_0\Vert }_2^2\stackrel{P}{\to }0.$$

Q.E.D.

Proof of Theorem 1(ii)

The proof for the consistency of the estimator ${\widehat{\beta }}_w$ is the same as in the elastic net case in Theorem 1(i). The only difference is the definition of ${\iota }_n={\lambda }_1{\sum }_{j=1}^p{\widehat{w}}_j\mid {\beta }_{j0}\mid +{\lambda }_2{\sum }_{j=1}^p{\beta }_{j0}^2$. Note that when β_j0 = 0, we can write ${\iota }_n={\lambda }_1{\sum }_{j\in A}{\widehat{w}}_j\mid {\beta }_{j0}\mid +{\lambda }_2{\sum }_{j\in A}{\beta }_{j0}^2$. In other words, only nonzero coefficient weights play a role in term Ι_n.

As in (25)

$$n^2\mathit{bE}{\Vert {\widehat{\beta }}_w-{\beta }_0\Vert }_2^2=O\left(\mathit{nq}\right)+O({\lambda }_1\sum _{j\in A}{\widehat{w}}_j\mid {\beta }_{j0}\mid +{\lambda }_2\sum _{j\in A}{\beta }_{j0}^2).$$ (26)

The key issue is the order of the penalty terms. We are only interested in nonzero parameter weights. Defining $\widehat{\eta }={\min }_{j\in A}\mid {\widehat{\beta }}_{j,\mathit{enet}}\mid$, we have for j ∈ A,

$${\widehat{w}}_j=\frac{1}{{\mid {\widehat{\beta }}_{j,\mathit{enet}}\mid }^{\gamma }}\le \frac{1}{{\widehat{\eta }}^{\gamma }}.$$

Then

$$\frac{{\lambda }_1\sum _{j\in A}{\widehat{w}}_j\mid {\beta }_{j0}\mid }{n^2}\le \frac{{\lambda }_1{\widehat{\eta }}^{-\gamma }{p}_{A}C}{n^2},$$ (27)

where we use max_j∈A∣β_j0 ≤ C, where C is a generic positive constant. So we can write (27)

$$\frac{{\lambda }_1{\widehat{\eta }}^{-\gamma }{p}_{A}C}{n^2}=\left[\frac{{\lambda }_1{\eta }^{-\gamma }{p}_{A}C}{n^2}\right]\left[{\left(\frac{\widehat{\eta }}{\eta }\right)}^{-\gamma }\right].$$ (28)

In the above equation we first show

$$E{\left(\frac{\widehat{\eta }}{\eta }\right)}^2=O\left(1\right).$$ (29)

First see that by simple algebraic inequality as in (6.13) of Zou and Zhang (2009),

$$\begin{array}{cc}\hfill E{\left(\frac{\widehat{\eta }}{\eta }\right)}^2& \le 2+\frac{2}{{\eta }^2}E{(\widehat{\eta }-\eta )}^2\hfill \\ \hfill & \le 2+\frac{2}{{\eta }^2}E{\Vert {\widehat{\beta }}_{\mathit{enet}}-{\beta }_0\Vert }_2^2.\hfill \end{array}$$ (30)

Next by Theorem 1(i) (equation (25))

$$\frac{1}{{\eta }^2}E{\Vert {\widehat{\beta }}_{\mathit{enet}}-{\beta }_0\Vert }_2^2=O\left(\frac{\mathit{qn}+{\lambda }_1{p}_A+{\lambda }_2{p}_A}{n^2{\eta }^2}\right),$$

since λ₁/n → 0, λ₂/n → 0 and p_A ≤ q, the largest stochastic order is qn/n²η² = q/nη². By q = n^ν, and 0 < ν < 1 with n^1–νη² → ∞ by Assumption 5(iii) clearly

$$\frac{1}{{\eta }^2}E{\Vert {\widehat{\beta }}_{\mathit{enet}}-{\beta }_0\Vert }_2^2=o\left(1\right),$$ (31)

Next by (30)(31) we have

$$E{\left(\frac{\widehat{\eta }}{\eta }\right)}^2\le 2+o\left(1\right).$$

So (29) is shown, and by Markov’s inequality

$${\left(\frac{\widehat{\eta }}{\eta }\right)}^2=O_p\left(1\right),$$

clearly ${\left(\frac{\widehat{\eta }}{\eta }\right)}^2\ne o_p\left(1\right)$. Note that

$${\left(\frac{\widehat{\eta }}{\eta }\right)}^{-\gamma }={\left[{\left(\frac{\widehat{\eta }}{\eta }\right)}^2\right]}^{-\gamma ∕2}.$$ (32)

Then by (32) we have

$${\left(\frac{\widehat{\eta }}{\eta }\right)}^{-\gamma }=O_p\left(1\right).$$

Next on the right hand side of (28)

$$\frac{{\lambda }_1}{n}\frac{p_{A}C}{n{\eta }^{\gamma }}\to 0,$$

since λ₁/n → 0 by Assumption 5(i), and by Assumption 5(iv) n^1–αη^γ → ∞. The last two equations illustrate that (28)

$$\frac{{\lambda }_1{\widehat{\eta }}^{-\gamma }{p}_{A}C}{n^2}=\left[\frac{{\lambda }_1{\eta }^{-\gamma }{p}_{A}C}{n^2}\right]\left[{\left(\frac{\widehat{\eta }}{\eta }\right)}^{-\gamma }\right]\stackrel{P}{\to }0.$$ (33)

Next in (26) we have

$$\frac{{\lambda }_2\sum _{j\in A}{\beta }_{j0}^2}{n^2}=O\left(\frac{{\lambda }_2{p}_A}{n^2}\right)=o\left(1\right),$$ (34)

by λ₂/n → 0, p_A/n → 0. Also note that by q/n → 0 through Assumption 3 combining (33)(34) in (26) above provides us

$$E{\Vert {\widehat{\beta }}_w-{\beta }_0\Vert }_2^2=O\left(\frac{\mathit{nq}}{n^2}\right)+O\left(\frac{{\lambda }_1\sum _{j\in A}{\widehat{w}}_j\mid {\beta }_{j0}\mid +{\lambda }_2\sum _{j\in A}{\beta }_{j0}^2}{n^2}\right)=o\left(1\right).$$ (35)

Q.E.D.

Proof of Theorem 2

In this proof we start by analyzing the GMM-Ridge estimator that is defined as follows:

$${\widehat{\beta }}_R+\arg \underset{\beta }{\min }{\left[\sum _{i=1}^n{g}_i\right(\beta \left)\right]}^{\prime }{W}_n\left[\sum _{i=1}^n{g}_i\right(\beta \left)\right]+{\lambda }_2{\Vert \beta \Vert }_2^2.$$

Note that this estimator is similar to the elastic net estimator, if we set λ₁ = 0, in elastic net estimator, then we obtain the GMM-Ridge estimator. So since the elastic net estimator is consistent, GMM-Ridge will be consistent as well. Define also the following q×p matrix ${\widehat{G}}_n\left({\widehat{\beta }}_R\right)=\frac{{\sum }_{i=1}^n\partial g_i\left({\widehat{\beta }}_R\right)}{\partial {\beta }^{\prime }}$. Then set $\stackrel{‒}{\beta }\in ({\beta }_0,{\widehat{\beta }}_R)$. Note that ${\widehat{G}}_n\left(\stackrel{‒}{\beta }\right)$ is the value of ${\widehat{G}}_n(.)$ evaluated at $\stackrel{‒}{\beta }$. A mean value theorem around β₀ applied to first order conditions provides, with $g_n\left({\beta }_0\right)={\sum }_{i=1}^n{g}_i\left({\beta }_0\right)$,

$${\widehat{\beta }}_R=-{\left[{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta })+{\lambda }_2{I}_p]}^{-1}\left[{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{g}_n\right({\beta }_0)-{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{\widehat{G}}_n(\stackrel{‒}{\beta }\left){\beta }_0\right].$$ (36)

Also using the mean value theorem with first order conditions, adding and subtracting λ₂ β₀ from first order conditions yields

$${\widehat{\beta }}_R-{\beta }_0=-{\left[{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta })+{\lambda }_2{I}_p]}^{-1}\left[{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{g}_n\right({\beta }_0)+{\lambda }_2{\beta }_0].$$ (37)

We need the following expressions by using (36)

$$\begin{array}{cc}\hfill {\widehat{\beta }}_R^{\prime }\left[{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{g}_n\right({\beta }_0)-{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{\widehat{G}}_n(\stackrel{‒}{\beta }\left){\beta }_0\right]& =-{\left[{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{g}_n\right({\beta }_0)-{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{\widehat{G}}_n(\stackrel{‒}{\beta }\left){\beta }_0\right]}^{\prime }\hfill \\ \hfill & \times {\left[{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta })+{\lambda }_2{I}_p]}^{-1}\hfill \\ \hfill & \times \left[{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{g}_n\right({\beta }_0)-{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{\widehat{G}}_n(\stackrel{‒}{\beta }\left){\beta }_0\right].\hfill \end{array}$$ (38)

$$\begin{array}{cc}\hfill {\widehat{\beta }}_R^{\prime }\left[{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta })+{\lambda }_2{I}_p]{\widehat{\beta }}_R& ={\left[{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{g}_n\right({\beta }_0)-{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{\widehat{G}}_n(\stackrel{‒}{\beta }\left){\beta }_0\right]}^{\prime }\hfill \\ \hfill & \times {\left[{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta })+{\lambda }_2{I}_p]}^{-1}\hfill \\ \hfill & \times \left[{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{g}_n\right({\beta }_0)-{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{\widehat{G}}_n(\stackrel{‒}{\beta }\left){\beta }_0\right].\hfill \end{array}$$ (39)

Next the aim is to rewrite the following GMM-Ridge objective function via a mean value expansion

$$\begin{array}{cc}\hfill {\left[\sum _{i=1}^n{g}_i\right({\widehat{\beta }}_R\left)\right]}^{\prime }{W}_n\left[\sum _{i=1}^n{g}_i\right({\widehat{\beta }}_R\left)\right]& +{\lambda }_2{\Vert {\widehat{\beta }}_R\Vert }_2^2=g_n{\left({\beta }_0\right)}^{\prime }{W}_n{g}_n\left({\beta }_0\right)\hfill \\ \hfill & +g_n{\left({\beta }_0\right)}^{\prime }{W}_n{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right)({\widehat{\beta }}_R-{\beta }_0)+{({\widehat{\beta }}_R-{\beta }_0)}^{\prime }{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{g}_n\left({\beta }_0\right)\hfill \\ \hfill & +{({\widehat{\beta }}_R-{\beta }_0)}^{\prime }{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right)({\widehat{\beta }}_R-{\beta }_0)+{\lambda }_2{\Vert {\widehat{\beta }}_R\Vert }_2^2.\hfill \end{array}$$ (40)

When λ₁ = 0, using Theorem 1 we see that ${\Vert {\widehat{\beta }}_R-\stackrel{‒}{\beta }\Vert }_2^2\stackrel{P}{\to }0$. Then use Assumption 1 to have

$$\begin{array}{cc}\hfill {\widehat{\beta }}_R^{\prime }\left[{\left(\frac{{\widehat{G}}_n\left({\widehat{\beta }}_R\right)}{n}\right)}^{\prime }{W}_n\left(\frac{{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right)}{n}\right)+{\lambda }_2{I}_p\right]{\widehat{\beta }}_R& ={\widehat{\beta }}_R^{\prime }\left[{\left(\frac{{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right)}{n}\right)}^{\prime }{W}_n\left(\frac{{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right)}{n}\right)+{\lambda }_2{I}_p\right]{\widehat{\beta }}_R\hfill \\ \hfill & +{\widehat{\beta }}_R^{\prime }\left[o_P\left(1\right)W_n\left(\frac{{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right)}{n}\right)+{\lambda }_2{I}_p\right]{\widehat{\beta }}_R,\hfill \end{array}$$ (41)

where the o_P(1) term comes from the uniform law of large numbers. Clearly the stochastic order of the second term is smaller than the first one on the right hand side of (41). By using the same argument to get (41), we have

$$\begin{array}{cc}\hfill {\widehat{\beta }}_R^{\prime }\left[{\left(\frac{{\widehat{G}}_n\left({\widehat{\beta }}_R\right)}{n}\right)}^{\prime }{W}_n{g}_n\left({\beta }_0\right)-{\left(\frac{{\widehat{G}}_n\left({\widehat{\beta }}_R\right)}{n}\right)}^{\prime }{W}_n\left(\frac{{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right)}{n}\right){\beta }_0\right]& ={\widehat{\beta }}_R^{\prime }\left[{\left(\frac{{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right)}{n}\right)}^{\prime }{W}_n{g}_n\left({\beta }_0\right)-{\left(\frac{{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right)}{n}\right)}^{\prime }{W}_n\left(\frac{{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right)}{n}\right){\beta }_0\right]\hfill \\ \hfill & +{\widehat{\beta }}_R^{\prime }\left[o_P\left(1\right)W_n{g}_n\left({\beta }_0\right)-o_P\left(1\right)W_n\left(\frac{{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right)}{n}\right){\beta }_0\right].\hfill \end{array}$$ (42)

Again the second term’s stochastic order is smaller than the first one in (42).

Furthermore we can rewrite the right hand side of (40)

$$\begin{array}{cc}\hfill g_n{\left({\beta }_0\right)}^{\prime }{W}_n{g}_n\left({\beta }_0\right)& +{\widehat{\beta }}_R^{\prime }\left[{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{g}_n\right({\beta }_0)-{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n(\stackrel{‒}{\beta }\left){\beta }_0\right]\hfill \\ \hfill & +{\left[{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{g}_n\right({\beta }_0)-{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n(\stackrel{‒}{\beta }\left){\beta }_0\right]}^{\prime }{\widehat{\beta }}_R\hfill \\ \hfill & +{\widehat{\beta }}_R^{\prime }\left[{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta })+{\lambda }_2{I}_p]{\widehat{\beta }}_R\hfill \\ \hfill & -g_n{\left({\beta }_0\right)}^{\prime }{W}_n{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right){\beta }_0-{\beta }_0^{\prime }{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{g}_n\left({\beta }_0\right)+{\beta }_0^{\prime }{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right){\beta }_0\hfill \\ \hfill & =g_n{\left({\beta }_0\right)}^{\prime }{W}_n{g}_n\left({\beta }_0\right)-{\widehat{\beta }}_R^{\prime }\left[{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta })+{\lambda }_2{I}_p]{\widehat{\beta }}_R\hfill \\ \hfill & -g_n{\left({\beta }_0\right)}^{\prime }{W}_n{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right){\beta }_0-{\beta }_0^{\prime }{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{g}_n\left({\beta }_0\right)+{\beta }_0^{\prime }{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right){\beta }_0\hfill \\ \hfill & -{\widehat{\beta }}_R^{\prime }{\mathit{so}}_1{\widehat{\beta }}_R,\hfill \end{array}$$ (43)

where so₁ represents the small order terms mentioned in (41)(42). The equality is obtained through (38)(39) via Assumption 1 and the consistency of GMM-Ridge since ridge is a subcase of elastic net with λ₁ = 0 imposed on the elastic net. Note that the right hand side expression in (38) is just the negative of the right hand side of the expression in (39). As in (43), for the estimator ${\widehat{\beta }}_w$ we have the following where ${\stackrel{‒}{\beta }}_w\in ({\beta }_0,{\widehat{\beta }}_w)$,

$$\begin{array}{cc}\hfill {\left[\sum _{i=1}^n{g}_i\right({\widehat{\beta }}_w\left)\right]}^{\prime }{W}_n\left[\sum _{i=1}^n{g}_i\right({\widehat{\beta }}_w\left)\right]& +{\lambda }_2{\Vert {\widehat{\beta }}_w\Vert }_2^2=g_n{\left({\beta }_0\right)}^{\prime }{W}_n{g}_n\left({\beta }_0\right)\hfill \\ \hfill & +{\widehat{\beta }}_w^{\prime }\left[{\widehat{G}}_n{\left({\stackrel{‒}{\beta }}_w\right)}^{\prime }{W}_n{g}_n\right({\beta }_0)-{\widehat{G}}_n{\left({\stackrel{‒}{\beta }}_w\right)}^{\prime }{W}_n\widehat{G}({\stackrel{‒}{\beta }}_w\left){\beta }_0\right]\hfill \\ \hfill & +{\left[{\widehat{G}}_n{\left({\stackrel{‒}{\beta }}_w\right)}^{\prime }{W}_n{g}_n\right({\beta }_0)-{\widehat{G}}_n{\left({\stackrel{‒}{\beta }}_w\right)}^{\prime }{W}_n{\widehat{G}}_n({\stackrel{‒}{\beta }}_w\left){\beta }_0\right]}^{\prime }{\widehat{\beta }}_w\hfill \\ \hfill & +{\widehat{\beta }}_w^{\prime }\left[{\widehat{G}}_n{\left({\stackrel{‒}{\beta }}_w\right)}^{\prime }{W}_n\widehat{G}\right({\stackrel{‒}{\beta }}_w)+{\lambda }_2{I}_p]{\widehat{\beta }}_w\hfill \\ \hfill & -g_n{\left({\beta }_0\right)}^{\prime }{W}_n{\widehat{G}}_n\left({\stackrel{‒}{\beta }}_w\right){\beta }_0-{\beta }_0^{\prime }{\widehat{G}}_n{\left({\stackrel{‒}{\beta }}_w\right)}^{\prime }{W}_n{g}_n\left({\beta }_0\right)\hfill \\ \hfill & +{\beta }_0^{\prime }{\widehat{G}}_n{\left({\stackrel{‒}{\beta }}_w\right)}^{\prime }{W}_n{\widehat{G}}_n\left({\stackrel{‒}{\beta }}_w\right){\beta }_0.\hfill \end{array}$$ (44)

Then see that by Theorem 1 ${\Vert \widehat{\beta }-{\stackrel{‒}{\beta }}_w\Vert }_2^2\stackrel{P}{\to }0$ and using (36)

$$\begin{array}{cc}\hfill {\widehat{\beta }}_w^{\prime }\left[{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{g}_n\right({\beta }_0)-{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n\widehat{G}(\stackrel{‒}{\beta }\left){\beta }_0\right]& ={\widehat{\beta }}_w^{\prime }\left[{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta })+{\lambda }_2{I}_p]{\left[{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta })+{\lambda }_2{I}_p]}^{-1}\hfill \\ \hfill & \times \left[{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{g}_n\right({\beta }_0)-{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n(\stackrel{‒}{\beta }\left){\beta }_0\right]\hfill \\ \hfill & =-{\widehat{\beta }}_w^{\prime }\left[{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta })+{\lambda }_2{I}_p]{\widehat{\beta }}_R-{\widehat{\beta }}_w^{\prime }{\mathit{so}}_2{\widehat{\beta }}_R.\hfill \end{array}$$ (45)

The term so₂ above comes from the same type of analysis done for the second and small stochastic order terms in (41)(42). Next substitute (45) into (44) to have

$$\begin{array}{cc}\hfill {\left[\sum _{i=1}^n{g}_i\right({\widehat{\beta }}_w\left)\right]}^{\prime }{W}_n\left[\sum _{i=1}^n{g}_i\right({\widehat{\beta }}_w\left)\right]& +{\lambda }_2{\Vert {\widehat{\beta }}_w\Vert }_2^2=g_n{\left({\beta }_0\right)}^{\prime }{W}_n{g}_n\left({\beta }_0\right)-{\widehat{\beta }}_w^{\prime }\left[{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta })+{\lambda }_2{I}_p]{\widehat{\beta }}_R\hfill \\ \hfill & -{\widehat{\beta }}_R^{\prime }\left[{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta })+{\lambda }_2{I}_p]{\widehat{\beta }}_w+{\widehat{\beta }}_w^{\prime }\left[{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta })+{\lambda }_2{I}_p]{\widehat{\beta }}_w\hfill \\ \hfill & -g_n{\left({\beta }_0\right)}^{\prime }{W}_n{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right){\beta }_0\hfill \\ \hfill & -{\beta }_0^{\prime }{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{g}_n\left({\beta }_0\right)+{\beta }_0^{\prime }{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right){\beta }_0\hfill \\ \hfill & -{\widehat{\beta }}_w^{\prime }{\mathit{so}}_2{\widehat{\beta }}_R-{\widehat{\beta }}_R^{\prime }{\mathit{so}}_3{\widehat{\beta }}_w+{\widehat{\beta }}_w^{\prime }{\mathit{so}}_4{\widehat{\beta }}_w.\hfill \end{array}$$ (46)

Term so₃ is the transpose of so₂, and term so₄ comes from approximation error between $\stackrel{‒}{\beta }$ and ${\stackrel{‒}{\beta }}_w$. Specifically note that ${\widehat{\beta }}_w^{\prime }{\mathit{so}}_2{\widehat{\beta }}_R{\widehat{\beta }}_R^{\prime }{\mathit{so}}_3{\widehat{\beta }}_w{\widehat{\beta }}_w^{\prime }{\mathit{so}}_4{\widehat{\beta }}_w$ are smaller order terms than the second, third, and fourth terms on the right hand side of (46) respectively. Denote so₅ = min(so₁, so₂, so₃, so₄). Now subtract (43) from (46) to have

$$\begin{array}{cc}\hfill {\left[\sum _{i=1}^n{g}_i\right({\widehat{\beta }}_w\left)\right]}^{\prime }{W}_n\left[\sum _{i=1}^n{g}_i\right({\widehat{\beta }}_w\left)\right]& +{\lambda }_2{\Vert {\widehat{\beta }}_w\Vert }_2^2-\left({\left[\sum _{i=1}^n{g}_i\right({\widehat{\beta }}_R\left)\right]}^{\prime }{W}_n\right[\sum _{i=1}^n{g}_i\left({\widehat{\beta }}_R\right)]+{\lambda }_2{\Vert {\widehat{\beta }}_R\Vert }_2^2)\hfill \\ \hfill & \ge {({\widehat{\beta }}_w-{\widehat{\beta }}_R)}^{\prime }\left[{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta })+{\lambda }_2{I}_p]({\widehat{\beta }}_w-{\widehat{\beta }}_R)\hfill \\ \hfill & +{({\widehat{\beta }}_w-{\widehat{\beta }}_R)}^{\prime }{\mathit{so}}_5({\widehat{\beta }}_w-{\widehat{\beta }}_R).\hfill \end{array}$$ (47)

The next analysis is very similar to equations (6.1)-(6.6) of Zou and Zhang (2009). After this important result see that by the definitions of ${\widehat{\beta }}_w$ and ${\widehat{\beta }}_R$

$$\begin{array}{cc}\hfill {\lambda }_1\sum _{j=1}^p{\widehat{w}}_j(\mid {\widehat{\beta }}_{j,R}\mid -\mid {\widehat{\beta }}_{j,w}\mid )& \ge {\left(\sum _{i=1}^n{g}_i\right({\widehat{\beta }}_w\left)\right)}^{\prime }{W}_n\left(\sum _{i=1}^n{g}_i\right({\widehat{\beta }}_w\left)\right)+{\lambda }_2{\Vert {\widehat{\beta }}_w\Vert }_2^2\hfill \\ \hfill & -\left[{\left(\sum _{i=1}^n{g}_i\right({\widehat{\beta }}_R\left)\right)}^{\prime }{W}_n\right(\sum _{i=1}^n{g}_i\left({\widehat{\beta }}_R\right))+{\lambda }_2{\Vert {\widehat{\beta }}_R\Vert }_2^2].\hfill \end{array}$$ (48)

Then also see that

$$\sum _{j=1}^p{\widehat{w}}_j(\mid {\widehat{\beta }}_{j,R}\mid -\mid {\widehat{\beta }}_{j,w}\mid )\le \sqrt{\sum _{j=1}^p{\left({\widehat{w}}_j\right)}^2}{\Vert {\widehat{\beta }}_R-{\widehat{\beta }}_w\Vert }_2.$$ (49)

Next benefit from (48), with (47)(49) to have

$$\begin{array}{cc}\hfill {({\widehat{\beta }}_w-{\widehat{\beta }}_R)}^{\prime }\left[{\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta })+{\lambda }_2{I}_p]({\widehat{\beta }}_w-{\widehat{\beta }}_R)& +{({\widehat{\beta }}_w-{\widehat{\beta }}_R)}^{\prime }{\mathit{so}}_5({\widehat{\beta }}_w-{\widehat{\beta }}_R)\hfill \\ \hfill & \le {\lambda }_1\sqrt{\sum _{j=1}^p{\left({\widehat{w}}_j\right)}^2}{\Vert {\widehat{\beta }}_R-{\widehat{\beta }}_w\Vert }_2.\hfill \end{array}$$ (50)

See that ${\text{Eig}}_{\min }\left({\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta })+{\lambda }_2{I}_p)={\text{Eig}}_{\min }\left({\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta }\left)\right)+{\lambda }_2$. Use this in (50) to have

$${\Vert {\widehat{\beta }}_w-{\widehat{\beta }}_R\Vert }_2\le \frac{{\lambda }_1\sqrt{\sum _{j=1}^p{\widehat{w}}_j^2}}{{\text{Eig}}_{\min }\left({\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta }\left)\right)+{\lambda }_2+{\mathit{so}}_5},$$ (51)

where ${\text{Eig}}_{\min }\left({\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta }\left)\right)$ is a term of larger stochastic order than so₅ which is explained in (46) (43). We also want to modify the last inequality. By the consistency of ${\widehat{\beta }}_w,{\widehat{\beta }}_R,\stackrel{‒}{\beta }\stackrel{P}{\to }{\beta }_0$. Then with the uniform law of large numbers on the partial derivative we have by Assumptions 1-3

$${\Vert {\left[\frac{{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right)}{n}\right]}^{\prime }{W}_n\left[\frac{{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right)}{n}\right]-G{\left({\beta }_0\right)}^{\prime }\mathit{WG}\left({\beta }_0\right)\Vert }_2^2\stackrel{P}{\to }0.$$

The last equation is true also for ${\widehat{\beta }}_w$, ${\widehat{\beta }}_R$ replacing $\stackrel{‒}{\beta }$. Then

$${\Vert {\widehat{G}}_n{\left(\stackrel{‒}{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right)-n^2\left[G{\left({\beta }_0\right)}^{\prime }\mathit{WG}\right({\beta }_0\left)\right]-o\left(n^2\right)\Vert }_2^2\stackrel{P}{\to }0.$$ (52)

Using Lemma A0 of Newey and Windmeijer (2009b), modify (51) in the following way given the last equality, set W = Ω⁻¹ (since this is the efficient limit weight as shown by Hansen (1982))

$${\Vert {\widehat{\beta }}_w-{\widehat{\beta }}_R\Vert }_2\le \frac{{\lambda }_1\sqrt{\sum _{j=1}^p{\widehat{w}}_j^2}}{n^2{\text{Eig}}_{\min }{\left(G\right({\beta }_0)}^{\prime }{\Omega }^{-1}G\left({\beta }_0\right))+{\lambda }_2+o_P(n^2)}.$$ (53)

Now we consider the second part of the proof of this theorem. We use the GMM ridge formula. Note that from (37)

$$\begin{array}{cc}\hfill {\widehat{\beta }}_R-{\beta }_0& =-{\lambda }_2{\left[{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta })+{\lambda }_2{I}_p]}^{-1}{\beta }_0\hfill \\ \hfill & -{\left[{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{\widehat{G}}_n\right(\stackrel{‒}{\beta })+{\lambda }_2{I}_p]}^{-1}\left[{\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{g}_n\right({\beta }_0\left)\right].\hfill \end{array}$$ (54)

We try to modify the equation above a little.

In the same way we obtain (52)

$${\Vert {\widehat{G}}_n{\left({\widehat{\beta }}_R\right)}^{\prime }{W}_n{g}_n\left({\beta }_0\right)-n\left[G{\left({\beta }_0\right)}^{\prime }W{g}_n\right({\beta }_0\left)\right]-o_P\left(n\right)\Vert }_2^2\stackrel{P}{\to }0.$$ (55)

Second, see that by gi(β) being independent E_gn(β₀)g_n(β₀)’ – nΩ → 0.

$$\begin{array}{cc}\hfill E\left[g_n{\left({\beta }_0\right)}^{\prime }{\Omega }^{-1}G\right({\beta }_0\left)G{\left({\beta }_0\right)}^{\prime }{\Omega }^{-1}{g}_n\right({\beta }_0\left)\right]& =\mathit{tr}\left\{G{\left({\beta }_0\right)}^{\prime }{\Omega }^{-1}E\right[g_n\left({\beta }_0\right)g_n{\left({\beta }_0\right)}^{\prime }\left]{\Omega }^{-1}G\right({\beta }_0\left)\right\}\hfill \\ \hfill & =\mathit{ntr}\left\{G{\left({\beta }_0\right)}^{\prime }{\Omega }^{-1}G\right({\beta }_0\left)\right\}+o\left(1\right)\hfill \\ \hfill & \le \mathit{np}{\text{Eig}}_{\max }\left(\Sigma \right)+o\left(1\right),\hfill \end{array}$$ (56)

where we use Σ = G(β₀)’Ω⁻¹G(β₀). Now we modify (54) using (55) (52), W = Ω⁻¹.

$$\begin{array}{cc}\hfill {\widehat{\beta }}_R-{\beta }_0& =-{\lambda }_2{\left[n^{2}G{\left({\beta }_0\right)}^{\prime }{\Omega }^{-1}G\right({\beta }_0)+{\lambda }_2{I}_p+o_P(n^2\left)\right]}^{-1}{\beta }_0\hfill \\ \hfill & -{\left[n^{2}G{\left({\beta }_0\right)}^{\prime }{\Omega }^{-1}G\right({\beta }_0)+{\lambda }_2{I}_p+o_P(n^2\left)\right]}^{-1}\left[\mathit{nG}{\left({\beta }_0\right)}^{\prime }{\Omega }^{-1}{g}_n\right({\beta }_0)+o_P(n\left)\right].\hfill \end{array}$$ (57)

Then see that

$$\begin{array}{cc}\hfill E\left({\Vert {\widehat{\beta }}_R-{\beta }_0\Vert }_2^2\right)& \le 2{\lambda }_2^2{\left[n^2{\text{Eig}}_{\min }\right(G{\left({\beta }_0\right)}^{\prime }{\Omega }^{-1}G\left({\beta }_0\right))+{\lambda }_2+o(n^2\left)\right]}^{-2}{\Vert {\beta }_0\Vert }_2^2\hfill \\ \hfill & +2{\left[n^2{\text{Eig}}_{\min }\right(G{\left({\beta }_0\right)}^{\prime }{\Omega }^{-1}G\left({\beta }_0\right))+{\lambda }_2+o(n^2\left)\right]}^{-2}\hfill \\ \hfill & \times n^{2}E\left[g_n{\left({\beta }_0\right)}^{\prime }{\Omega }^{-1}G\right({\beta }_0\left)G{\left({\beta }_0\right)}^{\prime }{\Omega }^{-1}{g}_n\right({\beta }_0\left)\right]+o\left(n^2\right)\hfill \\ \hfill & \le 2{\left[n^2{\text{Eig}}_{\min }\right(G{\left({\beta }_0\right)}^{\prime }{\Omega }^{-1}G\left({\beta }_0\right)+{\lambda }_2+o\left(n^2\right)]}^{-2}\hfill \\ \hfill & \times [{\lambda }_2^2{\Vert {\beta }_0\Vert }^2+n^{3}p{\text{Eig}}_{\max }(\Sigma )+o(n^2\left)\right],\hfill \end{array}$$ (58)

where the last inequality is by (56). Note that we do not use orders smaller than o(n²) in (58) since this will not make any difference in the proofs of the theorems below. Now use (53) and (58) to have

$$\begin{array}{cc}\hfill E\left({\Vert {\widehat{\beta }}_w-{\beta }_0\Vert }_2^2\right)& \le 2E\left({\Vert {\widehat{\beta }}_R-{\beta }_0\Vert }_2^2\right)+2E\left({\Vert {\widehat{\beta }}_w-{\widehat{\beta }}_R\Vert }_2^2\right)\hfill \\ \hfill & \le \frac{4{\lambda }_2^2{\Vert {\beta }_0\Vert }_2^2+4n^3\mathit{pB}+o\left(n^2\right)+2{\lambda }_1^{2}E\sum _{j=1}^p{\widehat{w}}_j^2}{{[n^{2}b+{\lambda }_2+o(n^2\left)\right]}^2}.\hfill \end{array}$$ (59)

See that b ≤ Eig_min(G(β₀)’Ω⁻¹G(β₀)) = Eig_min(Σ), B ≥ Eig_max(Σ).

Q.E.D

Proof of Theorem 3(i)

The proof is similar to proof of Theorem 3.2 in Zou and Zhang (2009). The differences are due to nonlinear nature of the problem. Our upper bounds in Theorem 2 converge to zero at a different rate than Zou and Zhang (2009). To prove the theorem, we need to show the following (Note that by Kuhn-Tucker conditions of (1)),

$$P[\forall j\in A^c,\mid 2{\widehat{G}}_{n,j}{\left(\tilde{\beta }\right)}^{\prime }{W}_n(\sum _{i=1}^n{g}_i\left(\tilde{\beta }\right))\mid \le {\lambda }_1^{\ast }{\widehat{w}}_j]\to 1,$$

where ${\widehat{G}}_n\left(\tilde{\beta }\right)=\frac{{\sum }_{i=1}^n\partial g_i\left(\tilde{\beta }\right)}{\partial {\beta }^{\prime }}$, and A^c = {j : β_j0 = 0, j = 1,2, ⋯, p}. ${\widehat{G}}_{n,j}\left(\tilde{\beta }\right)$ denotes the j th column of the partial derivative matrix which corresponds to irrelevant parameters, evaluated at $\tilde{\beta }$. Or we need to show

$$P[\exists j\in A^c,\mid 2{\widehat{G}}_{n,j}{\left(\tilde{\beta }\right)}^{\prime }{W}_n(\sum _{i=1}^n{g}_i\left(\tilde{\beta }\right))\mid >{\lambda }_1^{\ast }{\widehat{w}}_j]\to 0,$$

Now set η = min_j∈A∣β_j0, $\widehat{\eta }={\min }_{j\in A}\mid {\widehat{\beta }}_{j,\mathit{enet}}\mid$, and A = {j : β_j0 ≠ 0, j = 1,2, ⋯, p}. So

$$\begin{array}{cc}\hfill P[\exists j\in A^c,\mid 2{\widehat{G}}_{n,j}{\left(\tilde{\beta }\right)}^{\prime }{W}_n(\sum _{i=1}^n{g}_i\left(\tilde{\beta }\right))\mid >{\lambda }_1^{\ast }{\widehat{w}}_j]& \le \sum _{j\in A^c}P[\mid 2{\widehat{G}}_{n,j}{\left(\tilde{\beta }\right)}^{\prime }{W}_n(\sum _{i=1}^n{g}_i\left(\tilde{\beta }\right))\mid >{\lambda }_1^{\ast }{\widehat{w}}_j,\widehat{\eta }>\eta ∕2]\hfill \\ \hfill & +P[\widehat{\eta }\le \eta ∕2].\hfill \end{array}$$ (60)

Then as in equation (6.7) of Zou and Zhang (2009), we can show that

$$\begin{array}{cc}\hfill P[\widehat{\eta }\le \eta ∕2]& \le \frac{E{\Vert {\widehat{\beta }}_{\mathit{enet}}-{\beta }_0\Vert }_2^2}{{\eta }^2∕4}\hfill \\ \hfill & \le 16\frac{{\lambda }_2^2{\Vert {\beta }_0\Vert }_2^2+n^3\mathit{pB}+{\lambda }_1^{2}p+o\left(n^2\right)}{{[n^{2}b+{\lambda }_2+o(n^2\left)\right]}^2{\eta }^2},\hfill \end{array}$$ (61)

where the second inequality is due to Theorem 2. Then we can also have

$$\begin{array}{cc}\hfill \sum _{j\in A^c}P[\mid 2{\widehat{G}}_{n,j}{\left(\tilde{\beta }\right)}^{\prime }{W}_n(\sum _{i=1}^n{g}_i\left(\tilde{\beta }\right))\mid & >{\lambda }_1^{\ast }{\widehat{w}}_j,\widehat{\eta }>\eta ∕2]\hfill \\ \hfill & \le \sum _{j\in A^c}P[\mid 2{\widehat{G}}_{n,j}{\left(\tilde{\beta }\right)}^{\prime }{W}_n(\sum _{i=1}^n{g}_i\left(\tilde{\beta }\right))\mid >{\lambda }_1^{\ast }{\widehat{w}}_j,\widehat{\eta }>\eta ∕2,\mid {\widehat{\beta }}_{j,\mathit{enet}}\mid \le M]\hfill \\ \hfill & +\sum _{j\in A^c}P[\mid {\widehat{\beta }}_{j,\mathit{enet}}\mid >M]\hfill \\ \hfill & \le \sum _{j\in A^c}P[\mid 2{\widehat{G}}_{n,j}{\left(\tilde{\beta }\right)}^{\prime }{W}_n(\sum _{i=1}^n{g}_i\left(\tilde{\beta }\right))\mid >{\lambda }_1^{\ast }{M}^{-\gamma },\widehat{\eta }>\eta ∕2]\hfill \\ \hfill & +\sum _{j\in A^c}P[\mid {\widehat{\beta }}_{j,\mathit{enet}}\mid >M],\hfill \end{array}$$ (62)

where $M={\left(\frac{{\lambda }_1^{\ast }}{n^{3+\alpha }}\right)}^{\frac{1}{2\gamma }}$. Compared to Zou and Zhang (2009), M converges to zero faster.

In (62) we consider the second term on the right hand side. Via inequality (6.8) of Zou and Zhang (2009) and Theorem 2, we have

$$\begin{array}{cc}\hfill \sum _{j\in A^c}P[\mid {\widehat{\beta }}_{j,\mathit{enet}}\mid >M]& \le \frac{E{\Vert {\widehat{\beta }}_{\mathit{enet}}-{\beta }_0\Vert }_2^2}{M^2}\hfill \\ \hfill & \le 4\frac{{\lambda }_2^2{\Vert {\beta }_0\Vert }_2^2+4n^3\mathit{pB}+{\lambda }_1^{2}p+o\left(n^2\right)}{{[n^{2}b+{\lambda }_2+o(n^2\left)\right]}^2{M}^2}.\hfill \end{array}$$ (63)

Next we can consider the first term on the right hand side of (62)

$$\begin{array}{cc}\hfill \sum _{j\in A^c}P[\mid 2{\widehat{G}}_{n,j}{\left(\tilde{\beta }\right)}^{\prime }{W}_n(\sum _{i=1}^n{g}_i\left(\tilde{\beta }\right))\mid & >{\lambda }_1^{\ast }{M}^{-\gamma },\widehat{\eta }>\eta ∕2]\hfill \\ \hfill & \le \frac{4M^{2\gamma }}{{\lambda }_1^{\ast 2}}E\left[\sum _{j\in A^c}{\mid {\widehat{G}}_{n,j}{\left(\tilde{\beta }\right)}^{\prime }{W}_n\left(\sum _{i=1}^n{g}_i\right(\tilde{\beta }\left)\right)\mid }^2{I}_{\{\widehat{\eta }\ge \eta ∕2\}}\right].\hfill \end{array}$$ (64)

So we try to simplify the term on the right hand side of (64). Now we evaluate

$$\begin{array}{cc}\hfill \sum _{j\in A^c}{\mid {\widehat{G}}_{n,j}{\left(\tilde{\beta }\right)}^{\prime }{W}_n\left(\sum _{i=1}^n{g}_i\right(\tilde{\beta })\mid }^2& \le 2\sum _{j\in A^c}{\mid {\widehat{G}}_{n,j}{\left(\tilde{\beta }\right)}^{\prime }{W}_n\left(\sum _{i=1}^n{g}_i\right({\beta }_{A,0})\mid }^2\hfill \\ \hfill & +2\sum _{j\in A^c}{\mid {\widehat{G}}_{n,j}{\left(\tilde{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right)(\tilde{\beta }-{\beta }_{A,0})\mid }^2,\hfill \end{array}$$ (65)

where we have $\stackrel{‒}{\beta }\in ({\beta }_{A,0},\tilde{\beta })$, and

$$g_i\left(\tilde{\beta }\right)=g_i\left({\beta }_{A,0}\right)+\left[\frac{\partial g_i\left(\stackrel{‒}{\beta }\right)}{\partial {\beta }^{\prime }}\right](\tilde{\beta }-{\beta }_{A,0}).$$

Analyze each term in (65). Note that $\tilde{\beta }$ is consistent if we go through the same steps as in Theorem 1 by Assumption 5 on ${\lambda }_1^{\ast }$. Then applying Assumption 1 with Assumption 2 (Uniform Law of Large Numbers) in the same way as in equation (6.9) of Zou and Zhang (2009), we have

$$2\sum _{j\in A^c}{\mid {\widehat{G}}_{n,j}{\left(\tilde{\beta }\right)}^{\prime }{W}_n\left(\sum _{i=1}^n{g}_i\right({\beta }_{A,0})\mid }^2\le 2n^2{\Vert G{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}\sum _{i=1}^n{g}_i\left({\beta }_{A,0}\right)\Vert }_2^2+o_P\left(n^2\right),$$ (66)

where we put $W={\Omega }_{\ast }^{-1}$ and use Assumption 3(i), ${\Vert {\left[n^{-1}{\Sigma }_{i=1}^{n}E{g}_i\right({\beta }_{A,0}\left)g_i{\left({\beta }_{A,0}\right)}^{\prime }\right]}^{-1}-{\Omega }_{\ast }^{-1}\Vert }_2^2\to 0$ as the definition of the efficient limit weight matrix for the case of nonzero parameters to have

$$E{[\sum _{j\in A^c}\mid {\widehat{G}}_{n,j}{\left(\tilde{\beta }\right)}^{\prime }{W}_n(\sum _{i=1}^n{g}_i\left({\beta }_{A,0}\right)\mid ]}^2\le n^{3}B+o\left(n^3\right),$$ (67)

where we use

$$B\ge {\text{Eig}}_{\max }\left(\Sigma \right)\ge {\text{Eig}}_{\max }\left(G{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}G\right({\beta }_{A,0}\left)\right).$$ (68)

See that ${\Sigma }_{i=1}^n{g}_i\left({\beta }_{\mathcal{A},0}\right)=O_P\left(n^{1∕2}\right)$.

In the same manner as in equation (6.9) of Zou and Zhang (2009), we have

$$\sum _{j\in A^c}{\mid {\widehat{G}}_{n,j}{\left(\tilde{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right)(\tilde{\beta }-{\beta }_{A,0})\mid }^2\le n^4{\mid G{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}G\left({\beta }_{A,0}\right)(\tilde{\beta }-{\beta }_{A,0})\mid }^2+o_P\left(n^4\right).$$ (69)

Then by (68) and taking into account (69), we have

$$\sum _{j\in A^c}{\mid {\widehat{G}}_{n,j}{\left(\tilde{\beta }\right)}^{\prime }{W}_n{\widehat{G}}_n\left(\stackrel{‒}{\beta }\right)(\tilde{\beta }-{\beta }_{A,0})\mid }^2\le n^4{B}^2{\Vert \tilde{\beta }-{\beta }_{A,0}\Vert }_2^2+o_P\left(n^4\right).$$ (70)

Now substitute (67)-(70) into the term on the right hand side of (64), we get

$$\begin{array}{cc}\hfill E\left[\sum _{j\in A^c}{\mid {\widehat{G}}_{n,j}{\left(\tilde{\beta }\right)}^{\prime }{W}_n\left(\sum _{i=1}^n{g}_i\right(\tilde{\beta }\left)\right)\mid }^2{I}_{\{\widehat{\eta }>\eta ∕2\}}\right]& \le 2B^2{n}^{4}E\left({\Vert \tilde{\beta }-{\beta }_{A,0}\Vert }_2^2{I}_{\{\widehat{\eta }>\eta ∕2\}}\right)\hfill \\ \hfill & +2{\mathit{Bn}}^3+o\left(n^4\right).\hfill \end{array}$$ (71)

Define the ridge based version of $\tilde{\beta }$ with imposing ${\lambda }_1^{\ast }=0$

$$\tilde{\beta }({\lambda }_2,0)=\arg \underset{\beta }{\min }\left\{\right(\sum _{i=1}^n{g}_i{\left({\beta }_A\right)}^{\prime }{W}_n\left(\sum _{i=1}^n{g}_i\right({\beta }_A\left)\right)+{\lambda }_2\sum _{j\in A}{\beta }_j^2\}.$$ (72)

Then use the arguments in the proof of Theorem 2 (equation (53)), we have

$$\begin{array}{cc}\hfill {\Vert \tilde{\beta }-\tilde{\beta }({\lambda }_2,0)\Vert }_2& \le \frac{{\lambda }_1^{\ast }{\max }_{j\in A}{\widehat{w}}_j\sqrt{p}}{n^2{\text{Eig}}_{\min }\left(G{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}G\right({\beta }_{A,0}\left)\right)+{\lambda }_2+o_P\left(n^2\right)}\hfill \\ \hfill & \le \frac{{\lambda }_1^{\ast }{\widehat{\eta }}^{-\gamma }\sqrt{p}}{n^{2}b+{\lambda }_2+o_P\left(n^2\right)},\hfill \end{array}$$ (73)

where ${\text{Eig}}_{\min }\left(G{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}G\right({\beta }_{A,0}\left)\right)\ge {\mathrm{Eig}}_{\min }\left(G{\left({\beta }_0\right)}^{\prime }{\Omega }_{\ast }^{-1}G\right({\beta }_0\left)\right)\ge b$.

Then follow the proof of Theorem 2 (i.e. equation (59)), for the right hand side term in (71),

$$E\left({\Vert \tilde{\beta }-{\beta }_{A,0}\Vert }_2^2{I}_{\{\widehat{\eta }>\eta ∕2\}}\right)\le 4\frac{{\lambda }_2^2{\Vert {\beta }_0\Vert }_2^2+n^3\mathit{pB}+{\lambda }_1^{\ast 2}{(\eta ∕2)}^{-2\gamma }p+o\left(n^2\right)}{{({\mathit{bn}}^2+{\lambda }^2+o(n^2\left)\right)}^2}.$$ (74)

Now combine (61) (62)(63)(64)(71)(74) into (60),

$$\begin{array}{cc}\hfill P[\exists j\in A^c,\mid 2{\widehat{G}}_{n,j}{\left(\tilde{\beta }\right)}^{\prime }{W}_n(\sum _{i=1}^n{g}_i\left(\tilde{\beta }\right))\mid >{\lambda }_1^{\ast }{\widehat{w}}_j]& \le \frac{16}{{\eta }^2}\left[\frac{{\lambda }_2^2{{\beta }_0\Vert }_2^2+{\mathit{Bn}}^{3}p+{\lambda }_1^{2}p+o\left(n^2\right)}{{[n^{2}b+{\lambda }_2+o(n^2\left)\right]}^2}\right]\hfill \\ \hfill & +\frac{4}{M^2}\left[\frac{{\lambda }_2^2{{\beta }_0\Vert }_2^2+{\mathit{Bn}}^{3}p+{\lambda }_1^{2}p+o\left(n^2\right)}{{[n^{2}b+{\lambda }_2+o(n^2\left)\right]}^2}\right]\hfill \\ \hfill & +\frac{4M^{2\gamma }}{{\lambda }_1^{\ast 2}}\left[\right(2n^{3}B+o\left(n^3\right))+(2B^2{n}^4+o\left(n^4\right))\hfill \\ \hfill & \times \left(\frac{{\lambda }_2^2{\Vert {\beta }_0\Vert }_2^2+n^3\mathit{pB}+{\lambda }_1^{\ast 2}{(\eta ∕2)}^{-2\gamma }p+o\left(n^2\right)}{{({\mathit{bn}}^2+{\lambda }_2+o(n^2\left)\right)}^2}\right)].\hfill \end{array}$$ (75)

Now we need to show that each square bracketed term on the right hand side of equation (75) converges in probability to zero. We consider each of the right hand side elements in (75). We start with the first square bracketed term. The orders of the expressions are

$$\begin{array}{c}\hfill \frac{p}{n{\eta }^2}\frac{{\lambda }_2^2}{n^3}\to 0,\hfill \\ \hfill \frac{p}{n{\eta }^2}\to 0,\hfill \\ \hfill \frac{p}{n{\eta }^2}\frac{{\lambda }_1^2}{n^3}\to 0,\hfill \end{array}$$

by ${\lambda }_1^2∕n^3\to 0,{\lambda }_2^2∕n^3\to 0$ via Assumption 5, and

$$\frac{p}{n{\eta }^2}=\frac{1}{n^{1-\alpha }{\eta }^2}\to 0,$$ (76)

by Assumption 5(iv) and 3 with n^1–νη² → ∞ with α ≤ ν. Note that ${\Vert {\beta }_0\Vert }_2^2=O\left(p\right)$ by Assumption 1.

Next we consider the second square bracketed term on the right hand side of (75). See that the dominating term in that expression is stochastic order of

$$O\left(\frac{p}{n}\frac{1}{M^2}\right)\to 0.$$

The above is true since by Assumption 5, $\frac{{\lambda }_1^{\ast }}{n^{3+\alpha }}{n}^{\gamma (1-\alpha )}\to \infty$, and $M={\left[\frac{{\lambda }_1^{\ast }}{n^{3+\alpha }}\right]}^{1∕2\gamma }$ since γ > 0

$$\frac{M^{2}n}{p}={\left(\frac{{\lambda }_1^{\ast }}{n^{3+\alpha }}\right)}^{1∕\gamma }{n}^{1-\alpha }\to \infty .$$

The other terms in the second term on the right hand side of (75) are

$$\frac{{\lambda }_2^2}{n^3}\frac{{\Vert {\beta }_0\Vert }_2^2}{{\mathit{nM}}^2}\to 0,$$

by ${\lambda }_2^2∕n^3\to 0,{\Vert {\beta }_0\Vert }_2^2=O\left(p\right),\frac{p}{n{M}^2}\to 0$ by the analysis of the dominating term above. Then also in the same way

$$\frac{{\lambda }_1^{2}p}{M^2{n}^4}=\frac{{\lambda }_1^2}{n^3}\frac{p}{{\mathit{nM}}^2}\to 0,$$

where we use ${\lambda }_1^2∕n^3\to 0$, and the analysis of the dominating term $\frac{p}{n{M}^2}\to 0$ above.

Now we consider the last square bracketed term in (75). The orders of the expression are $\frac{M^{2\gamma }{n}^3}{{\lambda }_1^{\ast 2}},\frac{M^{2\gamma }{\lambda }_2^{2}p}{{\lambda }_1^{\ast 2}},\frac{M^{2\gamma }{n}^{3}p}{{\lambda }_1^{\ast 2}},\frac{M^{2\gamma }{\lambda }_1^{\ast 2}p}{{\lambda }_1^{\ast 2}{\eta }^{2\gamma }}$ respectively. Clearly the order of the third expression dominates that of the first and second expressions, given ${\lambda }_2^2∕n^3\to 0$. Now the order of the third term is

$$\frac{M^{2\gamma }{n}^{3}p}{{\lambda }_1^{\ast 2}}=\frac{1}{{\lambda }_1^{\ast }}\to 0,$$

given $M={\left(\frac{{\lambda }_1^{\ast }}{n^{3+\alpha }}\right)}^{2\gamma }$ definition. For the order of the last expression

$$\frac{M^{2\gamma }}{{\lambda }_1^{\ast 2}}\frac{{\lambda }_1^{\ast 2}}{{\eta }^{2\gamma }}p=M^{2\gamma }\frac{p}{{\eta }^{2\gamma }}=\frac{{\lambda }_1^{\ast }}{n}\frac{1}{n^2{\eta }^{2\gamma }}\to 0,$$

since ${\lambda }_1^{\ast }∕n\to 0,p=n^{\alpha }$, p = n^α, M definition, and by Assumption 5(iv).

Q.E.D.

Proof of Theorem 3(ii)

The proof technique is very similar to the proof of Theorem 3.3 in Zou and Zhang (2009), given Theorems 2 and 3(i). After Theorem 3(i) result, it suffices to prove

$$P({\mathit{min}}_{j\in A}\mid {\tilde{\beta }}_j\mid >0)\to 1.$$

Then we can write the following with $\tilde{\beta }({\lambda }_2,0)$ defined in (72)

$$\underset{j\in A}{\min }\mid {\tilde{\beta }}_j\mid >\underset{j\in A}{\min }\mid \tilde{\beta }{({\lambda }_2,0)}_j\mid -{\Vert \tilde{\beta }-\tilde{\beta }({\lambda }_2,0)\Vert }_2.$$ (77)

Also see that

$$\underset{j\in A}{\min }\mid \tilde{\beta }{({\lambda }_2,0)}_j\mid >\underset{j\in A}{\min }\mid {\beta }_{j0}\mid -{\Vert \tilde{\beta }({\lambda }_2,0)-{\beta }_{A,0}\Vert }_2.$$ (78)

Combine (77)(78) to have

$$\underset{j\in A}{\min }\mid {\tilde{\beta }}_j\mid >\underset{j\in A}{\min }\mid {\beta }_{j0}\mid -{\Vert \tilde{\beta }({\lambda }_2,0)-{\beta }_{A,0}\Vert }_2-{\Vert \tilde{\beta }-\tilde{\beta }({\lambda }_2,0)\Vert }_2.$$ (79)

Now we consider the last two terms on the right hand side of (79). Similar to derivation of (57)(58) we have

$$\begin{array}{cc}\hfill E{\Vert \tilde{\beta }({\lambda }_2,0)-{\beta }_{A,0}\Vert }_2^2& \le \frac{4{\lambda }_2^2{\Vert {\beta }_{A,0}\Vert }_2^2+4n^3\mathit{pB}+o\left(n^2\right)}{{[n^{2}b+{\lambda }_2+o(n^2\left)\right]}^2}\hfill \\ \hfill & =O(p∕n)=o\left(1\right),\hfill \end{array}$$ (80)

by ${\lambda }_2^2{\Vert {\beta }_{A,0}\Vert }_2^2∕n^4=({\lambda }_2^2∕n^2)({\Vert {\beta }_{A,0}\Vert }_2^2∕n^2)\to 0$, and ${\Vert {\beta }_{A,0}\Vert }_2^2=O\left(p\right)$. Then by (73)

$${\Vert \tilde{\beta }-\tilde{\beta }({\lambda }_2,0)\Vert }_2\le \frac{{\lambda }_1^{\ast }{\widehat{\eta }}^{-\gamma }\sqrt{p}}{n^{2}b+{\lambda }_2+o_P\left(n^2\right)}.$$ (81)

See that

$$\frac{{\lambda }_1^{\ast }{\widehat{\eta }}^{-\gamma }\sqrt{p}}{n^{2}b+{\lambda }_2+o_P\left(n^2\right)}=O\left({(\widehat{\eta }∕\eta )}^{-\gamma }\frac{{\lambda }_1^{\ast }\sqrt{p}}{n^2}{\eta }^{-\gamma }\right).$$ (82)

$$\frac{{\lambda }_1^{\ast }\sqrt{p}}{n^2}{\eta }^{-\gamma }=\frac{1}{p^{1∕2}}\frac{{\lambda }_1^{\ast }}{n}\left(\frac{p}{n}\frac{1}{{\eta }^{\gamma }}\right)=\frac{1}{n^{\alpha ∕2}}o\left(1\right),$$ (83)

by (76) (p = n^α), Assumptions 5(i),(iv). Then by Theorem 2

$$\begin{array}{cc}\hfill E\left[{(\widehat{\eta }∕\eta )}^2\right]& \le 2+\frac{1}{{\eta }^2}E{[\widehat{\eta }-\eta ]}^2\hfill \\ \hfill & \le 2+\frac{2}{{\eta }^2}E{\Vert \widehat{\beta }({\lambda }_2,{\lambda }_1)-{\beta }_0\Vert }_2^2\hfill \\ \hfill & \le 2+\frac{8}{{\eta }^2}\left[\frac{{\lambda }_2^2{\Vert {\beta }_0\Vert }_2^2+n^3\mathit{pB}+{\lambda }_1^{2}p+o\left(n^2\right)}{{[n^{2}b+{\lambda }_2+o(n^2\left)\right]}^2}\right]\hfill \\ \hfill & =O\left(1\right),\hfill \end{array}$$ (84)

by λ₁/n → 0, λ₂/n → 0, p/n → 0, ${\lambda }_1∕n\to 0,{\lambda }_2∕n\to 0,p∕n\to 0{\Vert {\beta }_0\Vert }_2^2=O\left(p\right)$, by (76). Using the same technique in the proof of Theorem 1(ii), substitute (83)(84) into (82) to have

$$\frac{{\lambda }_1^{\ast }{\widehat{\eta }}^{-\gamma }\sqrt{p}}{n^{2}b+{\lambda }_2+o_P\left(n^2\right)}\stackrel{P}{\to }0,$$ (85)

Now use (80) and (83) with η = min_j∈A∣β_j0∣ to have

$${\mathit{min}}_{j\in A}\mid {\tilde{\beta }}_j\mid >\eta -\sqrt{\frac{p}{n}}{O}_P\left(1\right)-\sqrt{\frac{1}{n^{\alpha }}}{o}_P\left(1\right).$$ (86)

Then we obtain the desired result since by Assumption 5(v) and 5(iv) the last two terms on right hand side of (86) converges to zero faster than η.

Q.E.D

Proof of Theorem 4

We now prove the limit result. First, define

$$z_n={\delta }^{\prime }\left[\frac{I+{\lambda }_2{\left(\widehat{G}{\left({\widehat{\beta }}_{\mathit{aenet},A}\right)}^{\prime }{W}_n\widehat{G}\right({\widehat{\beta }}_{\mathit{aenet},A}\left)\right)}^{-1}}{1+{\lambda }_2∕n}\right]{\left(\widehat{G}{\left({\widehat{\beta }}_{\mathit{aenet},A}\right)}^{\prime }{W}_n\widehat{G}\right({\widehat{\beta }}_{\mathit{aenet},A}\left)\right)}^{1∕2}{n}^{-1∕2}({\widehat{\beta }}_{\mathit{aenet},A}-{\beta }_{A,0}).$$

Then we need the following result. Following the proof of Theorem 1 and using Theorem 3,

$${\Vert {\widehat{\beta }}_{\mathit{aenet},A}-{\beta }_{A,0}\Vert }_2^2\stackrel{P}{\to }0,$$

and by (80)

$${\Vert \tilde{\beta }({\lambda }_2,0)-{\beta }_{A,0}\Vert }_2^2\stackrel{P}{\to }0.$$

Next by Assumption 1 and Assumption 2, and considering the above results about consistency, we have

$${\Vert \frac{\widehat{G}{\left({\widehat{\beta }}_{\mathit{aenet},A}\right)}^{\prime }}{n}{W}_n\frac{\widehat{G}\left({\widehat{\beta }}_{\mathit{aenet},A}\right)}{n}-\frac{\widehat{G}{\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }}{n}{W}_n\frac{\widehat{G}\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}{n}\Vert }_2^2\stackrel{P}{\to }0.$$ (87)

Next note that as in p.18 of Zou and Zhang (2009)

$$\begin{array}{cc}\hfill {\delta }^{\prime }[I+{\lambda }_2{\left[\widehat{G}{\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }{W}_n\widehat{G}\right(\tilde{\beta }({\lambda }_2,0)\left)\right]}^{-1}]& \times {\left[\widehat{G}{\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }{W}_n\widehat{G}\right(\tilde{\beta }({\lambda }_2,0)\left)\right]}^{1∕2}{n}^{1∕2}(\tilde{\beta }-\frac{{\beta }_{A,0}}{1+{\lambda }_2∕n})\hfill \\ \hfill & ={\left[\delta {(I+{\lambda }_2[\widehat{G}\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }{W}_n\widehat{G}\right(\tilde{\beta }({\lambda }_2,0)\left)\right]}^{-1}){\left(\widehat{G}{\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }{W}_n\widehat{G}\right(\tilde{\beta }({\lambda }_2,0)\left)\right)}^{1∕2}\hfill \\ \hfill & \times n^{-1∕2}\frac{{\lambda }_2{\beta }_{A,0}}{n+{\lambda }_2}]\hfill \\ \hfill & +{\left[\delta {(I+{\lambda }_2[\widehat{G}\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }{W}_n\widehat{G}\right(\tilde{\beta }({\lambda }_2,0)\left)\right]}^{-1}){\left(\widehat{G}{\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }{W}_n\widehat{G}\right(\tilde{\beta }({\lambda }_2,0)\left)\right)}^{1∕2}\hfill \\ \hfill & \times n^{-1∕2}(\tilde{\beta }-\tilde{\beta }({\lambda }_2,0\left)\right)\hfill \\ \hfill & +{\left[\delta {(I+{\lambda }_2[\widehat{G}\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }{W}_n\widehat{G}\right(\tilde{\beta }({\lambda }_2,0)\left)\right]}^{-1}){\left(\widehat{G}{\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }{W}_n\widehat{G}\right(\tilde{\beta }({\lambda }_2,0)\left)\right)}^{1∕2}\hfill \\ \hfill & \times n^{-1∕2}\left(\tilde{\beta }\right({\lambda }_2,0)-{\beta }_{A,0})].\hfill \end{array}$$ (88)

The last term on the right hand side of (88) can be rewritten as (by (37))

$$\begin{array}{cc}\hfill \{{\lambda }_2{\left[\widehat{G}{\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }{W}_n\widehat{G}\right(\tilde{\beta }({\lambda }_2,0)\left)\right]}^{-1}+I\}& \times {\left[\widehat{G}{\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }{W}_n\widehat{G}\right(\tilde{\beta }({\lambda }_2,0)\left)\right]}^{1∕2}{n}^{-1∕2}\left[\tilde{\beta }\right({\lambda }_2,0)-{\beta }_{A,0}]\hfill \\ \hfill & =-{\lambda }_2{n}^{-1∕2}{\left[\widehat{G}{\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }{W}_n\widehat{G}\right(\tilde{\beta }({\lambda }_2,0)\left)\right]}^{-1∕2}{\beta }_{A,0}\hfill \\ \hfill & -n^{-1∕2}{\left[\widehat{G}{\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }{W}_n\widehat{G}\right(\tilde{\beta }({\lambda }_2,0)\left)\right]}^{-1∕2}\hfill \\ \hfill & \times \left[\widehat{G}{\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }{W}_n{g}_n\right({\beta }_{A,0}\left)\right]+o_P\left(1\right),\hfill \end{array}$$ (89)

where $g_n\left({\beta }_{A,0}\right)={\Sigma }_{i=1}^n{g}_i\left({\beta }_{A,0}\right)$. Note that in the above equation we have an asymptotically negligible term due to using (37) where we have $\stackrel{‒}{\beta }$ instead of $\tilde{\beta }({\lambda }_2,0)$.

Via Theorem 3, and also using (87), with probability tending to one, z_n = T₁ + T₂ + T₃, where

$$\begin{array}{cc}\hfill T_1& =\left[{\delta }^{\prime }\right[(I+{\lambda }_2{\left[\widehat{G}{\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }{W}_n\widehat{G}\right(\tilde{\beta }({\lambda }_2,0)\left)\right]}^{-1})]{\left[\widehat{G}{\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }{W}_n\widehat{G}\right(\tilde{\beta }({\lambda }_2,0)\left)\right]}^{1∕2}\hfill \\ \hfill & \times n^{-1∕2}{\lambda }_2\frac{{\beta }_{A,0}}{n+{\lambda }_2}]\hfill \\ \hfill & -\left[{\delta }^{\prime }{\lambda }_2{n}^{-1∕2}{\left[\widehat{G}{\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }{W}_n\widehat{G}\right(\tilde{\beta }({\lambda }_2,0)\left)\right]}^{-1∕2}{\beta }_{A,0}\right].\hfill \\ \hfill T_2& ={\delta }^{\prime }[I+{\lambda }_2{\left[\widehat{G}{\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }{W}_n\widehat{G}\right(\tilde{\beta }({\lambda }_2,0)\left)\right]}^{-1}]{\left[\widehat{G}{\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }{W}_n\widehat{G}\right(\tilde{\beta }({\lambda }_2,0)\left)\right]}^{1∕2}\hfill \\ \hfill & \times n^{-1∕2}(\tilde{\beta }-\tilde{\beta }({\lambda }_2,0\left)\right).\hfill \\ \hfill & T_3={\delta }^{\prime }{\left[\widehat{G}{\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }{W}_n\widehat{G}\right(\tilde{\beta }({\lambda }_2,0)\left)\right]}^{-1∕2}\left[\widehat{G}{\left(\tilde{\beta }\right({\lambda }_2,0\left)\right)}^{\prime }{W}_n\sum _{i=1}^n\frac{g_i\left({\beta }_{A,0}\right)}{n^{1∕2}}\right].\hfill \end{array}$$

Consider T₁, use Assumptions 1, 2, 5 with $W_n={\widehat{\Omega }}_{\ast }^{-1},W={\Omega }_{\ast }^{-1}$

$$\begin{array}{cc}\hfill T_1^2& \le \frac{2}{n}{\Vert (I+{\lambda }_2{\left(G{\left({\beta }_{A,0}\right)}^{\prime }\mathit{WG}\right({\beta }_{A,0}\left)\right)}^{-1}){\left(G{\left({\beta }_{A,0}\right)}^{\prime }\mathit{WG}\right({\beta }_{A,0}\left)\right)}^{1∕2}\frac{{\lambda }_2{\beta }_{A,0}}{n+{\lambda }_2}\Vert }_2^2\hfill \\ \hfill & +\frac{2}{n}{\Vert {\lambda }_2{\left(G{\left({\beta }_{A,0}\right)}^{\prime }\mathit{WG}\right({\beta }_A,0\left)\right)}^{-1∕2}{\beta }_{A,0}\Vert }_2^2+o_P\left(1\right)\hfill \\ \hfill & \le \frac{2}{n}\frac{{\lambda }_2^2\mathit{Bn}}{{(n+{\lambda }_2)}^2}{(1+\frac{{\lambda }_2}{\mathit{bn}})}^2{\Vert {\beta }_{A,0}\Vert }_2^2+\frac{2}{n}{\lambda }_2^2{\Vert {\beta }_{A,0}\Vert }_2^2\frac{1}{{\mathit{bn}}^2}+o_P\left(1\right)\hfill \\ \hfill & =o_P\left(1\right),\hfill \end{array}$$

via ${\lambda }_2=o\left(n\right),{\Vert {\beta }_{A,0}\Vert }_2^{2}O\left(p\right)$, and ${\lambda }_2^2{\Vert {\beta }_{A,0}\Vert }_2^2∕n^2\to 0$. Consider T₂ similar to the above analysis and (73)

$$\begin{array}{cc}\hfill T_2^2& \le \frac{1}{n}{(1+\frac{{\lambda }_2}{\mathit{bn}})}^2\left(\mathit{Bn}\right){\Vert \tilde{\beta }-\tilde{\beta }({\lambda }_2,0)\Vert }_2^2\hfill \\ \hfill & \le B{(1+\frac{{\lambda }_2}{\mathit{bn}})}^2{\left[\frac{{\lambda }_1^{\ast }{\widehat{\eta }}^{-\gamma }\sqrt{p}}{[n^{2}b+{\lambda }_2+o_P(n^2\left)\right]}\right]}^2\hfill \\ \hfill & =o_P\left(1\right),\hfill \end{array}$$

by (85) and λ₂ = o(n).

For the term T₃ we benefit from Liapunov Central Limit Theorem. By Assumptions 1 and 2

$$\begin{array}{cc}\hfill T_3& =\frac{\sum _{i=1}^n{\delta }^{\prime }{\left[\widehat{G}{\left(\tilde{\beta }\right({\lambda }_{2,0}\left)\right)}^{\prime }{\widehat{\Omega }}_{\ast }^{-1}\widehat{G}\right(\tilde{\beta }({\lambda }_2,0)\left)\right]}^{-1∕2}\left[\widehat{G}{\left(\tilde{\beta }\right({\lambda }_{2,0}\left)\right)}^{\prime }{\widehat{\Omega }}_{\ast }^{-1}\right]g_i\left({\beta }_{A,0}\right)}{n^{1∕2}}\hfill \\ \hfill & =\frac{\sum _{i=1}^n{\delta }^{\prime }{\left[G{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}G\right({\beta }_{A,0}\left)\right]}^{-1∕2}\left[G{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}\right]g_i\left({\beta }_{A,0}\right)}{n^{1∕2}}+o_P\left(1\right).\hfill \end{array}$$

Next set $R_i={\delta }^{\prime }{\left[G{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}G\right({\beta }_{A,0}\left)\right]}^{-1∕2}\left[G{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}\right]\frac{g_i\left({\beta }_{A,0}\right)}{n^{1∕2}}$. So

$$\begin{array}{cc}\hfill \sum _{i=1}^n{\mathit{ER}}_i^2& =n^{-1}\sum _{i=1}^{n}E\left[{\delta }^{\prime }{\left[G{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}G\right({\beta }_{A,0}\left)\right]}^{-1∕2}G{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}{g}_i\right({\beta }_{A,0}\left)g_i{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}G\right({\beta }_{A,0}\left){\left[G{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}G\right({\beta }_{A,0}\left)\right]}^{-1∕2}\delta \right]\hfill \\ \hfill & ={\delta }^{\prime }{\left[G{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}G\right({\beta }_{A,0}\left)\right]}^{-1∕2}G{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}\left[n^{-1}\sum _{i=1}^n{\mathit{Eg}}_i\right({\beta }_{A,0}\left)g_i{\left({\beta }_{A,0}\right)}^{\prime }\right]{\Omega }_{\ast }^{-1}G\left({\beta }_{A,0}\right){\left[G{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}G\right({\beta }_{A,0}\left)\right]}^{-1∕2}\delta .\hfill \end{array}$$

Then use ${\Vert n^{-1}{\Sigma }_{i=1}^{n}E{g}_i\left({\beta }_{A,0}\right)g_i{\left({\beta }_{A,0}\right)}^{\prime }-{\Omega }_{\ast }\Vert }_2^2\to 0$. Take the limit of the term above

$$\begin{array}{cc}\hfill \underset{n\to \infty }{\lim }\sum _{i=1}^n{\mathit{ER}}_i^2& ={\delta }^{\prime }{\left[G{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}G\right({\beta }_{A,0}\left)\right]}^{-1∕2}G{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}{\Omega }_{\ast }{\Omega }_{\ast }^{-1}G({\beta }_A,0){\left[G{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}G\right({\beta }_{A,0}\left)\right]}^{-1∕2}\delta .\hfill \\ \hfill & ={\delta }^{\prime }\delta =1.\hfill \end{array}$$

Next we show ${\Sigma }_{i=1}^{n}E{\mid R_i\mid }^{2+l}\to 0$. See that by (6)(7) and 4

$$\begin{array}{cc}\hfill \sum _{i=1}^n\frac{1}{n^{1+l∕2}}E{\left[{\delta }^{\prime }{\left(G{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}G\right({\beta }_{A,0}\left)\right)}^{-1∕2}G{\left({\beta }_{A,0}\right)}^{\prime }{\Omega }_{\ast }^{-1}{g}_i\right({\beta }_{A,0}\left)\right]}^{2+l}& \le {\left[\frac{B}{b}\right]}^{1+l∕2}\frac{1}{n^{1+l∕2}}\sum _{i=1}^{n}E{\Vert {\Omega }_{\ast }^{-1∕2}{g}_i\left({\beta }_{A,0}\right)\Vert }_{2+l}^{2+l}\hfill \\ \hfill & \to 0.\hfill \end{array}$$

So $T_3\stackrel{d}{\to }N(0,1)$. The desired result then follows from z_n = T₁ + T₂ + T₃ with probability approaching one.

Q.E.D.