LINEAR HYPOTHESIS TESTING FOR HIGH DIMENSIONAL GENERALIZED LINEAR MODELS

Abstract

This paper is concerned with testing linear hypotheses in high-dimensional generalized linear models. To deal with linear hypotheses, we first propose constrained partial regularization method and study its statistical properties. We further introduce an algorithm for solving regularization problems with folded-concave penalty functions and linear constraints. To test linear hypotheses, we propose a partial penalized likelihood ratio test, a partial penalized score test and a partial penalized Wald test. We show that the limiting null distributions of these three test statistics are χ² distribution with the same degrees of freedom, and under local alternatives, they asymptotically follow non-central χ² distributions with the same degrees of freedom and noncentral parameter, provided the number of parameters involved in the test hypothesis grows to ∞ at a certain rate. Simulation studies are conducted to examine the finite sample performance of the proposed tests. Empirical analysis of a real data example is used to illustrate the proposed testing procedures.

1. Introduction.

During the last three decades, there are many works devoted to developing variable selection techniques for high dimensional regression models. Fan and Lv (2010) presents a selective overview on this topic. There are some recent works for hypothesis testing on Lasso estimate (Tibshirani, 1996) in high-dimensional linear models. Lockhart et al. (2014) proposed the covariance test which produces a sequence of p-values as the tuning parameter, λ_n, decreases, and features become non-zero in the Lasso. This approach does not give confidence intervals or p-values for an individual variable’s coefficient. Taylor et al. (2014) and Lee et al. (2016) extended the covariance testing framework to test hypotheses about individual features, after conditioning on a model selected by the Lasso. However, their framework permits inference only about features which have non-zero coefficients in a Lasso regression; this set of features likely varies across samples, making the interpretation difficult. Moreover, these work focused on high dimensional linear regression models, and it remains unknown whether their results can be extended to a more general setting.

This paper will focus on generalized linear models (GLM, McCullagh and Nelder, 1989). Let Y be the response, and X be its associate fixed-design covariate vector. The GLM assumes that the distribution of Y belongs to the exponential family. The exponential family with canonical link has the following probability density function

$$\exp \left(\frac{Y{\mathit{\beta }}_0^T\mathit{X}-b\left({\mathit{\beta }}_0^T\mathit{X}\right)}{{\varphi }_0}\right)c(Y),$$ (1.1)

where β₀ is a p-dimensional vector of regression coefficients, and ϕ₀ is some positive nuisance parameter. In this paper, we assume that b(·) is thrice continuously differentiable with b′′(·) > 0.

We study testing linear hypothesis $H_0:\mathit{C}{\mathit{\beta }}_{0,\mathcal{M}}=\mathit{t}$ in GLM, where ${\mathit{\beta }}_{0,\mathcal{M}}$ is a subvector of β₀, the true regression coefficients. The number of covariates p can be much larger than the sample size n, while the number of parameters in ${\mathit{\beta }}_{0,\mathcal{M}}$ is assumed to be much smaller than n. Such type of hypotheses is of particular interests when the goal is to explore the group structure of β₀. Moreover, it also includes a very important class of hypotheses ${\mathit{\beta }}_{0,\mathcal{M}}=\mathbf{0}$ by setting C to be the identity matrix and t = 0. In the literature, Fan and Peng (2004) proposed penalized likelihood ratio test for H_0a : Cβ_0,S = 0 in GLM, where β_0,S is the vector consisting of all nonzero elements of β₀ when p = o(n^1/5) where n stands for the sample size. Wang and Cui (2013) extended Fan and Peng (2004)’s proposal and considered a penalized likelihood ratio statistic for testing $H_{0b}:{\mathit{\beta }}_{0,\mathcal{M}}=\mathbf{0}$, requiring p = o(n^1/5). Ning and Liu (2017) proposed a decorrelated score test for $H_{0c}:{\mathit{\beta }}_{0,\mathcal{M}}=\mathbf{0}$ under the setting of high dimensional penalized M-estimators with nonconvex penalties. Recently, Fang, Ning and Liu (2017) extended the proposal of Ning and Liu (2017) and developed a class of decorrelated Wald, score and partial likelihood ratio tests for Cox’s model with high dimensional survival data. Zhang and Cheng (2017) proposed a maximal type statistic based on the desparsified Lasso estimator (van de Geer et al., 2014) and a bootstrap-assisted testing procedure for $H_{0d}:{\mathit{\beta }}_{0,\mathcal{M}}=\mathbf{0}$, allowing the cardinality of $\mathcal{M}$ to be an arbitrary subset of [1,...,p]. In this paper, we aim to develop theory of Wald test, score test and likelihood ratio test for $H_0:\mathit{C}{\mathit{\beta }}_{0,\mathcal{M}}=\mathit{t}$ in GLM under ultrahigh dimensional setting (i.e., p grows exponentially with n).

It is well known that the Wald, score and likelihood ratio tests are equivalent in the fixed p case. However, it can be challenging to generalize these statistics to the setting with ultrahigh dimensionality. To better understand this point, we take the Wald statistic for illustration. Consider the null hypothesis $H_0:{\mathit{\beta }}_{0,\mathcal{M}}=\mathbf{0}$. Analogous to the classical Wald statistic, in the high dimensional setting, one might consider the statistic ${\widehat{\mathit{\beta }}}_{\mathcal{M}}^T{\{\widehat{\mathrm{cov}}({\widehat{\mathit{\beta }}}_{\mathcal{M}})\}}^{-1}{\widehat{\mathit{\beta }}}_{\mathcal{M}}$ for some penalized regression estimator $\widehat{\mathit{\beta }}$ and its variance estimator $\widehat{\mathrm{cov}}(\widehat{\mathit{\beta }})$. The choice of the estimators is essential here: some penalized regression estimator such as the Lasso, or the Dantzig estimator (Candes and Tao, 2007) cannot be used due to their large biases when p ≫ n. The non-concave penalized estimator does not have this bias issue, but the minimal signal conditions imposed in Fan and Peng (2004) and Fan and Lv (2011) implies that the associated Wald statistic does not have any power for local alternatives of the type $H_a:{\mathit{\beta }}_{0,\mathcal{M}}={\mathit{h}}_n$ for some sequence h_n such that ∥h_n∥2 ≪ λ_n where ∥ · ∥2 is the Euclidean norm. Moreover, to implement the score and the likelihood ratio statistics, we need to estimate the regression parameter under the null, which involves penalized likelihood under linear constraints. This is a very challenging task and has rarely been studied: (a) the associated estimation and variable selection property is not standard from a theoretical perspective, and (b) there is a lack of constrained optimization algorithms that can produce sparse estimators from a computational perspective.

We briefly summarize our contributions as follows. First, we consider a more general form of hypothesis. In contrast, existing literature mainly focuses on testing ${\mathit{\beta }}_{0,\mathcal{M}}=\mathbf{0}$. Besides, we also allow the number of linear constraints to diverge with n. Our tests are therefore applicable to a wider range of real applications for testing a growing set of linear hypotheses. Second, we propose a partial penalized Wald, a partial penalized score and a partial penalized likelihood-ratio statistic based on the class of folded-concave penalty functions, and show their equivalence in the high dimensional setting. We derive the asymptotic distributions of our test statistics under the null hypothesis and the local alternatives. Third, we systematically study the partial penalized estimator with linear constraints. We derive its rate of convergence and limiting distribution. These results are significant in their own rights. The unconstrained and constrained estimators share similar forms, but the constrained estimator is more efficient due to the additional information contained in the constraints under the null hypothesis. Fourth, we introduce an algorithm for solving regularization problems with folded-concave penalty functions and equality constraints, based on the alternating direction method of multipliers (ADMM, cf. Boyd et al., 2011).

The rest of the paper is organized as follows. We study the statistical properties of the constrained partial penalized estimator with folded concave penalty functions in Section 2. We formally define our partial penalized Wald, score and likelihood-ratio statistics, establish their limiting distributions, and show their equivalence in Section 3. Detailed implementations of our testing procedures are given in Section 3.3, where we introduce our algorithm for solving the constrained partial penalized regression problems. Simulation studies are presented in Section 4. The proof of Theorem 3.1 is presented in Section 5. Other proofs and addition numerical results are presented in the supplementary material (Shi et al., 2018).

2. Constrained partial penalized regression.

2.1. Model setup.

Suppose that {X_i,Y_i}, i = 1,··· ,n is a sample from model (1.1). Denote by Y = (Y₁,...,Y_n) the n-dimensional response vector and X = (X₁,···,X_n)^T is the n×p design matrix. We assume the covariates X_i are fixed design. Let X^j denote the jth column of X. To simplify the presentation, for any r×q matrix Φ and any set J ⊂ [1,2,...,q], we denote by Φ_J the submatrix of Φ formed by columns in J. Similarly, for any q-dimensional vector ϕ, ϕ_J stands for the subvector of ϕ formed by elements in J. We further denote Φ_J1,J2 as the submatrix of Φ formed by rows in J₁ and columns in J₂ for any J₁ ⊆ [1,...,r] and J₂ ⊆ [1,...,q]. Let |J| be the number of elements in J. Define J^c = [1,...,q] − J to be the complement of J.

In this paper, we assume log p = O(n^a) for some 0 < a < 1 and focus on the following testing problem:

$$H_0:\mathit{C}{\mathit{\beta }}_{0,\mathcal{M}}=\mathit{t},$$ (2.1)

for a given $\mathcal{M}\subseteq [1,\dots ,p]$, an $r\times \left|\mathcal{M}\right|$ matrix C and an r-dimensional vector t. We assume that the matrix C is of full row rank. This implies there are no redundant or contradictory constraints in (2.1). Let $m=\left|\mathcal{M}\right|$, we have r ≤ m.

Define the partial penalized likelihood function

$$Q_n(\mathit{\beta },\lambda )=\frac{1}{n}\sum _{i=1}^n\left\{Y_i{\mathit{\beta }}^T{\mathit{X}}_i-b\left({\mathit{\beta }}^T{\mathit{X}}_i\right)\right\}-\sum _{j\notin \mathcal{M}}{p}_{\lambda }\left(\left|{\mathit{\beta }}_j\right|\right).$$

for some penalty function p_λ(·) with a tuning parameter λ. Further define

$${\widehat{\mathit{\beta }}}_0=\arg \underset{{}_{\mathit{\beta }}}{\max }{Q}_n\left(\mathit{\beta },{\lambda }_{n,0}\right)\text{subject}\text{to}\mathit{C}{\mathit{\beta }}_{\mathcal{M}}=\mathit{t},$$ (2.2)

$${\widehat{\mathit{\beta }}}_a=\arg \underset{{}_{\mathit{\beta }}}{\max }{Q}_n\left(\mathit{\beta },{\lambda }_{n,a}\right).$$ (2.3)

Note that in (2.2) and (2.3), we do not add penalties on parameters involved in the constraints. This enables to avoid imposing minimal signal condition on elements of ${\mathit{\beta }}_{0,\mathcal{M}}$. Thus, the corresponding likelihood ratio test, Wald test and score test have power at local alternatives.

We present a lemma to characterize the constrained local maximizer ${\widehat{\mathit{\beta }}}_0$ in the supplementary material (see Lemma S.1). In Section 3, we show that these partial penalized estimators help us to obtain valid statistical inference about the null hypothesis.

2.2. Partial penalized regression with linear constraint.

In this section, we study the statistical properties of ${\widehat{\mathit{\beta }}}_0$ and ${\widehat{\mathit{\beta }}}_a$ by restricting p_λ to the class of folded concave penalty functions. Popular penalty functions such as SCAD (Fan and Li, 2001) and MCP (Zhang, 2010) belong to this class. Let ρ(t₀,λ) = p_λ(t₀)/λ for λ > 0. We assume that ρ(t₀,λ) is increasing and concave in t₀ ∈ [0,∞), and has a continuous derivative ρ′(t₀,λ) with ρ′(0+,λ) > 0. In addition, assume ρ′(t₀,λ) is increasing in λ ∈ (0,∞) and ρ′(0+,λ) is independent of λ. For any vector v = (v₁,...,v_q)^T, define

$$\begin{gathered} \overline{\mathit{\rho }}(\mathit{v},\lambda )={\left\{\mathrm{sgn}\left(v_1\right){\rho }^{\prime }\left(\left|v_1\right|,\lambda \right),\cdots ,\mathrm{sgn}\left(v_q\right){\rho }^{\prime }\left(\left|v_q\right|,\lambda \right)\right\}}^T, \\ \mathit{\mu }(\mathit{v})=\left\{b^{\prime }\left(v_1\right),\dots ,b^{\prime }\left(v_q\right)\right\},\mathbf{\Sigma }(\mathit{v})=\mathrm{diag}\left\{b^{\prime \prime }\left(v_1\right),\dots ,b^{\prime \prime }\left(v_q\right)\right\}, \end{gathered}$$

where sgn(·) denotes the sign function. We further define the local concavity of the penalty function ρ at v with ∥v∥0 = q as

$$\kappa (\rho ,\mathit{v},\lambda )=\underset{ϵ\to 0^{+}}{\lim }\underset{1\le j\le q}{\max }\underset{t_1<t_2\in (|v_j|-ϵ,|v_j|+ϵ)}{\sup }-\frac{{\rho }^{\prime }\left(t_2,\lambda \right)-{\rho }^{\prime }\left(t_1,\lambda \right)}{t_2-t_1}.$$

We assume that the true regression coefficient β₀ is sparse and satisfies $\mathit{C}{\mathit{\beta }}_{0,\mathcal{M}}-\mathit{t}={\mathit{h}}_n$ for some sequence of vectors h_n → 0. When h_n = 0, the null holds. Otherwise, the alternative holds. Let $S=\{j\in {\mathcal{M}}^c:{\beta }_{0,j}\ne 0\}$ and s = |S|. Let d_n be the half minimum signal of β₀,s, i.e, d_n = min_j∈S |β_j|/2. Define ${\mathcal{N}}_0=\{\mathit{\beta }\in {ℝ}^p:{\Vert {\mathit{\beta }}_{S\cup \mathcal{M}}-{\mathit{\beta }}_{0,S\cup \mathcal{M}}\Vert }_2\le \sqrt{(s+m)\log (n)/n},{\mathit{\beta }}_{{(S\cup \mathcal{M})}^c}=0\}$. We impose the following conditions.

(A1) Assume that

$$\begin{gathered} \underset{{}_{1\le j\le p}}{\max }{\Vert {\mathit{X}}^j\Vert }_{\infty }=O\left(\sqrt{n/\log (p)}\right),\underset{{}_{1\le j\le p}}{\max }{\Vert {\mathit{X}}^j\Vert }_2=O(\sqrt{n}), \\ \underset{\mathit{\beta }\in {\mathcal{N}}_0}{\inf }{\lambda }_{\min }\left({\mathit{X}}_{S\cup \mathcal{M}}^T\mathbf{\Sigma }(\mathit{X}\mathit{\beta }){\mathit{X}}_{S\cup \mathcal{M}}\right)\ge cn, \\ {\lambda }_{\max }\left({\mathit{X}}_{S\cup \mathcal{M}}^T\mathbf{\Sigma }\left(\mathit{X}{\mathit{\beta }}_0\right){\mathit{X}}_{S\cup \mathcal{M}}\right)=O(n), \\ {\Vert {\mathit{X}}_{{(S\cup \mathcal{M})}^c}^T\mathbf{\Sigma }\left({\mathit{X}}^T{\mathit{\beta }}_0\right){\mathit{X}}_{S\cup \mathcal{M}}\Vert }_{2,\infty }=O(n), \\ \underset{{}_{1\le j\le p}}{\max }\underset{\mathit{\beta }\in {\mathcal{N}}_0}{\sup }{\lambda }_{\max }\left({\mathit{X}}_{S\cup \mathcal{M}}^T\mathrm{diag}\left\{\left|{\mathit{X}}^j\right|\circ \left|b^{\prime \prime \prime }(\mathit{X}\mathit{\beta })\right|\right\}{\mathit{X}}_{S\cup \mathcal{M}}\right)=O(n), \end{gathered}$$

for some constant c > 0, where for any vector v = (v₁,...,v_q)^T, diag(v) denotes a diagonal matrix with the j-th diagonal elements being v_j, |v| = (|v₁|,...,|v_q|)^T, and ∥B∥2_,∞ = sup_v:∥v∥2₌₁ ∥Bv∥ for any matrix B with q rows.

(A2) Assume that $d_n\gg {\lambda }_{n,j}\gg \max \{\sqrt{(s+m)/n},\sqrt{(\log p)/n}\},p_{{\lambda }_{n,j}}^{\prime }\left(d_n\right)=o\left({(s+m)}^{-1/2}{n}^{-1/2}\right),{\lambda }_{n,j}{\kappa }_{0,j}=o(1)$ where ${\kappa }_{0,j}={\max }_{\mathit{\beta }\in {\mathcal{N}}_0}\kappa \left(\rho ,\mathit{\beta },{\lambda }_{n,j}\right)$, for j = 0,a.

(A3) Assume that there exist some constants M and v₀ such that

$$\underset{{}_{1\le i\le n}}{\max }\text{E}\left\{\exp \left(\frac{\left|Y_i-\mu \left({\mathit{\beta }}_0^T{\mathit{X}}_i\right)\right|}{M}\right)-1-\frac{\left|Y_i-\mu \left({\mathit{\beta }}_0^T{\mathit{X}}_i\right)\right|}{M}\right\}{M}^2\le \frac{v_0}{2}.$$

(A4) Assume that ${\Vert {\mathit{h}}_n\Vert }_2=O\left(\sqrt{\min (s+m-r,r)/n}\right)$, and λ_max ((CC^T)⁻¹) = O(1).

In Section S4.1 of the supplementary material, we show that Condition (A1) holds with probability tending to 1 if the covariate vectors X₁,...,X_n are uniformly bounded or realizations from a sub-Gaussian distribution. The first condition in (A2) is a minimum signal assumption imposed on nonzero elements in ${\mathcal{M}}^c$ only. This is due to partial penalization, which enables us to evaluate the uncertainty of the estimation for small signals. Such conditions are not assumed in van de Geer et al. (2014) and Ning and Liu (2017) for testing $H_0:{\mathit{\beta }}_{0,\mathcal{M}}=0$. However, we note that these authors impose some additional assumptions on the design matrix. For example, the validity of the decorrelated score statistic depends on the sparsity of w^*. For testing univariate parameters, this requires the degree of a particular node in the graph to be relatively small when the covariate follows a Gaussian graphical model (see Remark 6 in Ning and Liu, 2017). In Section S4.3 of the supplementary material, we show Condition (A3) holds for linear, logistic, and Poisson regression models.

Theorem 2.1. Suppose that Conditions (A1)-(A4) hold, and $s+m=o(\sqrt{n})$, then the following holds: (i) With probability tending to 1, ${\widehat{\mathit{\beta }}}_0$ and ${\widehat{\mathit{\beta }}}_a$ defined in (2.2) and (2.3) must satisfy ${\widehat{\mathit{\beta }}}_{0,{(S\cup \mathcal{M})}^c}={\widehat{\mathit{\beta }}}_{a,{(S\cup \mathcal{M})}^c}=0$. (ii) ${\Vert {\widehat{\mathit{\beta }}}_{a,S\cup \mathcal{M}}-{\mathit{\beta }}_{a,S\cup \mathcal{M}}\Vert }_2=O_p\left(\sqrt{(s+m)/n}\right)$ and ${\Vert {\widehat{\mathit{\beta }}}_{0,S\cup \mathcal{M}}-{\mathit{\beta }}_{0,S\cup \mathcal{M}}\Vert }_2=O_p\left(\sqrt{(s+m-r)/n}\right)$. If further s + m =o(n^1/3), then we have

$$\begin{gathered} \sqrt{n}\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-{\mathit{\beta }}_{0,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{a,S}-{\mathit{\beta }}_{0,S}\end{array}\right)=\frac{1}{\sqrt{n}}{\mathit{K}}_n^{-1}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\}+o_p(1), \\ \sqrt{n}\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}-{\mathit{\beta }}_{0,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{0,S}-{\mathit{\beta }}_{0,S}\end{array}\right)=\frac{1}{\sqrt{n}}{\mathit{K}}_n^{-1/2}\left(\mathit{I}-{\mathit{P}}_n\right){\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\} \\ -\sqrt{n}{\mathit{K}}_n^{-1/2}{\mathit{P}}_n{\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{C}}^T{\left(\mathit{C}{\mathit{C}}^T\right)}^{-1}{\mathit{h}}_n\\ \mathbf{0}\end{array}\right)+o_p(1), \end{gathered}$$

where I is the identity matrix, K_n is the (m + s) × (m + s) matrix

$${\mathit{K}}_n=\frac{1}{n}\left(\begin{array}{cc}{\mathit{X}}_{\mathcal{M}}^T\mathbf{\Sigma }\left(\mathit{X}{\mathit{\beta }}_0\right){\mathit{X}}_{\mathcal{M}}& {\mathit{X}}_{\mathcal{M}}^T\mathbf{\Sigma }\left(\mathit{X}{\mathit{\beta }}_0\right){\mathit{X}}_{\mathcal{S}}\\ {\mathit{X}}_{\mathcal{S}}^T\mathbf{\Sigma }\left(\mathit{X}{\mathit{\beta }}_0\right){\mathit{X}}_{\mathcal{M}}& {\mathit{X}}_{\mathcal{S}}^T\mathbf{\Sigma }\left(\mathit{X}{\mathit{\beta }}_0\right){\mathit{X}}_{\mathcal{S}}\end{array}\right),$$

and P_n is the (m × s) × (m × s) projection matrix

$${\mathit{P}}_n={\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right){\left\{\left(\mathit{C}{\mathit{O}}_{r\times s}\right){\mathit{K}}_n^{-1}\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right)\right\}}^{-1}\left(\mathit{C}{\mathit{O}}_{r\times s}\right){\mathit{K}}_n^{-1/2},$$

where O_r×s is an r × s zero matrix.

Remark 2.1. Since $d_n\gg \sqrt{(s+m)/n}$, Theorem 2.1(ii) implies that each element in ${\widehat{\mathit{\beta }}}_{0,S}$ and ${\widehat{\mathit{\beta }}}_{a,S}$ is nonzero. This together with result (i) shows the sign consistency of ${\widehat{\mathit{\beta }}}_{0,{\mathcal{M}}^c}$ and ${\widehat{\mathit{\beta }}}_{a,{\mathcal{M}}^c}$.

Remark 2.2. Theorem 2.1 implies that the constrained estimator ${\widehat{\mathit{\beta }}}_0$ converges at a rate of $O_p(\sqrt{s+m-r}/\sqrt{n})$. In contrast, the unconstrained estimator converges at a rate of $O_p(\sqrt{s+m}/\sqrt{n})$. This suggests that when h_n is relatively small, the constrained estimator ${\widehat{\mathit{\beta }}}_0$ converges faster than the unconstrained ${\widehat{\mathit{\beta }}}_a$ defined in (2.3), when s+m−r ≪ s+m. This result is expected with the following intuition: the more information about β₀ we have, the more accurate the estimator will be.

Remark 2.3. Under certain regularity conditions, Theorem 2.1 implies that

$$\sqrt{n}\left\{{\left({\widehat{\mathit{\beta }}}_{0,\mathcal{M}}-{\mathit{\beta }}_{0,\mathcal{M}}\right)}^T,{\left({\widehat{\mathit{\beta }}}_{0,S}-{\mathit{\beta }}_{0,S}\right)}^T\right\}\to N\left(-{\mathit{\xi }}_0,{\mathit{V}}_0\right),$$

where ξ₀and V0 are limits of $\sqrt{n}{\mathit{K}}_n^{-1/2}{\mathit{P}}_n{\mathit{K}}_n^{-1/2}{\left({\mathit{h}}_n^T,{\mathbf{0}}^T\right)}^T$ and ${\mathit{K}}_n^{-1/2}(\mathit{I}-{\mathit{P}}_n){\mathit{K}}_n^{-1/2}$, respectively. Similarly, we can show

$$\sqrt{n}\left\{{\left({\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-{\mathit{\beta }}_{0,\mathcal{M}}\right)}^T,{\left({\widehat{\mathit{\beta }}}_{a,S}-{\mathit{\beta }}_{0,S}\right)}^T\right\}\to N\left(\mathbf{0},{\mathit{V}}_a\right),$$

where ${\mathit{V}}_a={\lim }_n{\mathit{K}}_n^{-1}$. Note that a^T V₀a ≤ a^T V_aa for any a ∈ ℝ^s+m. Under the null, we have ξ₀ = 0, which suggests that ${\widehat{\mathit{\beta }}}_0$ is more efficient than ${\widehat{\mathit{\beta }}}_a$ in terms of a smaller asymptotic variance. Under the alternative, ${\widehat{\mathit{\beta }}}_{0,\mathcal{M}}$ is asymptotically biased. This can be interpreted as a bias-variance trade-off between ${\widehat{\mathit{\beta }}}_0$ and ${\widehat{\mathit{\beta }}}_a$.

3. Partial penalized Wald, score and likelihood ratio statistics.

3.1. Test statistics.

We begin by introducing our partial penalized like-lihood ratio statistic,

$$T_L=2n\left\{L_n\left({\widehat{\mathit{\beta }}}_a\right)-L_n\left({\widehat{\mathit{\beta }}}_0\right)\right\}/\widehat{\varphi },$$ (3.1)

where L_n(β) = ∑_i{Y_iβ^T X_i − b(β^T X_i)}/n, ${\widehat{\mathit{\beta }}}_0$ and ${\widehat{\mathit{\beta }}}_a$ are defined in (2.2) and (2.3) respectively, and $\widehat{\varphi }$ is some consistent estimator for ϕ₀ in (1.1). For Gaussian linear models, ϕ₀ corresponds to the error variance. For logistic or Poisson regression models, ϕ₀ = 1.

The partial penalized Wald statistic is based on $\sqrt{n}(\mathit{C}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-t)$. Define ${\mathbf{\Omega }}_n={\mathit{K}}_n^{-1}$ and denote Ω_mm as the first m rows and columns of Ω_n. It follows from Theorem 2.1 that its asymptotic variance is equal to CΩ_mmC^T. Let ${\widehat{S}}_a=\left\{j\in {\mathcal{M}}^c:{\widehat{\mathit{\beta }}}_{a,j}\ne 0\right\}$. Then, with probability tending to 1, we have ${\widehat{S}}_a=S$. Define

$${\widehat{\mathbf{\Omega }}}_a=n{\left(\begin{array}{cc}{\mathit{X}}_{\mathcal{M}}^T\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_a\right){\mathit{X}}_{\mathcal{M}}& {\mathit{X}}_{\mathcal{M}}^T\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_a\right){\mathit{X}}_{{\widehat{S}}_a}\\ {\mathit{X}}_{{\widehat{S}}_a}^T\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_a\right){\mathit{X}}_{\mathcal{M}}& {\mathit{X}}_{{\widehat{S}}_a}^T\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_a\right){\mathit{X}}_{{\widehat{S}}_a}\end{array}\right)}^{-1},$$

and ${\widehat{\mathbf{\Omega }}}_{a,mm}$ as its submatrix formed by its first m rows and columns. The partial penalized Wald statistic is defined by

$$T_W={(\mathit{C}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-\mathit{t})}^T{\left(\mathit{C}{\widehat{\mathbf{\Omega }}}_{a,mm}{\mathit{C}}^T\right)}^{-1}(\mathit{C}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-\mathit{t})/\widehat{\varphi }.$$ (3.2)

Analogous to the classical score statistic, we define our partial penalized score statistic as

$$T_S={\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\widehat{\mathit{\beta }}}_0\right)\right\}}^T\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}\\ {\mathit{X}}_{{\widehat{S}}_0}\end{array}\right){\widehat{\mathbf{\Omega }}}_0{\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}\\ {\mathit{X}}_{{\widehat{S}}_0}\end{array}\right)}^T\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\widehat{\mathit{\beta }}}_0\right)\right\}/\widehat{\varphi },$$ (3.3)

where ${\widehat{S}}_0=\{j\in {\mathcal{M}}^c:{\widehat{\beta }}_{0,j}\ne 0\}$, and

$${\widehat{\mathbf{\Omega }}}_0=n{\left(\begin{array}{cc}{\mathit{X}}_{\mathcal{M}}^T\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_0\right){\mathit{X}}_{\mathcal{M}}& {\mathit{X}}_{\mathcal{M}}^T\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_0\right){\mathit{X}}_{{\widehat{S}}_0}\\ {\mathit{X}}_{{\widehat{S}}_0}^T\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_0\right){\mathit{X}}_{\mathcal{M}}& {\mathit{X}}_{{\widehat{S}}_0}^T\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_0\right){\mathit{X}}_{{\widehat{S}}_0}\end{array}\right)}^{-1}.$$

3.2. Limiting distributions of the test statistics.

For a given significance level α, we reject the null hypothesis when $T>{\chi }_{\alpha }^2(r)$ for T = T_L,T_W or T_S where ${\chi }_{\alpha }^2(r)$ is the upper α-quantile of a central χ² distribution with r degrees of freedom and r is the number of constraints. Assume r is fixed. When $\widehat{\varphi }$ is consistent to ϕ₀, it follows from Theorem 2.1 that T_L, T_W and T_S converge asymptotically to a (non-central) χ² distribution with r degrees of freedom. However, when r diverges with n, there is no such theoretical guarantee. This is because the concept of weak convergence is not well defined in such settings. To resolve this issue, we observe that when the following holds,

$$\underset{x}{\sup }\left|\mathrm{Pr}(T\le x)-\mathrm{Pr}\left({\chi }^2\left(r,{\gamma }_n\right)\le x\right)\right|\to 0,$$

where χ²(r,γ_n) is a chi square random variable with r degrees of freedom and noncentrality parameter γ_n which is allowed to vary with n, our testing procedure is still valid using χ² approximation.

Theorem 3.1. Assume Conditions (A1)-(A4) hold, s + m = o(n^1/3), and $\left|\widehat{\varphi }-{\varphi }_0\right|=o_p(1)$. Further assume the following holds:

$$\frac{r^{1/4}}{n^{3/2}}\sum _{i=1}^n{\left\{{\left({\mathit{X}}_{i,\mathcal{M}\cup \mathcal{S}}\right)}^T{\mathit{K}}_n^{-1}{\mathit{X}}_{i,\mathcal{M}\cup \mathcal{S}}\right\}}^{3/2}\to 0.$$ (3.4)

Then, we have

$$\underset{x}{\sup }\left|Pr(T\le x)-Pr\left({\chi }^2\left(r,{\gamma }_n\right)\le x\right)\right|\to 0,$$ (3.5)

for T = T_W ,T_S or T_L, where ${\gamma }_n=n{\mathit{h}}_n^T{\left(\mathit{C}{\mathbf{\Omega }}_{mm}{\mathit{C}}^T\right)}^{-1}{\mathit{h}}_n/{\varphi }_0.$

REMARK 3.1. By (3.5), it is immediate to see that

$$\underset{x}{\sup }\left|\mathrm{Pr}\left(T_1\le x\right)-\mathrm{Pr}\left(T_2\le x\right)\right|\to 0,$$

for any T₁,T₂ ∈ {T_W ,T_S,T_L}. This establish the equivalence between the partial penalized Wald, score and likelihood-ratio statistics. Condition (3.4) is the key to guarantee χ² approximation in (3.5). When r = O(1), this condition is equivalent to

$$\frac{1}{n^{3/2}}\sum _{i=1}^n{\left\{{\left({\mathit{X}}_{i,\mathcal{M}\cup \mathcal{S}}\right)}^T{\mathit{K}}_n^{-1}{\mathit{X}}_{i,\mathcal{M}\cup \mathcal{S}}\right\}}^{3/2}\to 0,$$

which corresponds to the Lyaponuv condition that ensures the asymptotic normality of ${\widehat{\mathit{\beta }}}_{0,\mathcal{M}\cup S}$ and ${\widehat{\mathit{\beta }}}_{a,\mathcal{M}\cup S}$. When r diverges, (3.4) guarantees that the following Lyaponuv type bound goes to 0,

$$\underset{\mathcal{C}}{\sup }\left|\mathrm{Pr}\left({\left(\mathit{C}{\mathbf{\Omega }}_{mm}{\mathit{C}}^T\right)}^{-1/2}\left(\mathit{C}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-\mathit{t}\right)/{\sqrt{\varphi }}_0\in \mathcal{C}\right)-\mathrm{Pr}(\mathit{Z}\in \mathcal{C})\right|\to 0,$$

where Z represents an r-dimensional multivariate normal with identity covariance matrix, and the supremum is taken over all convex subsets $\mathcal{C}$ in ℝ^m. The scaling factor r^1/4 accounts for the dependence of the above Lyaponuv type estimate on the dimension and it remains unknown whether the factor r^1/4 can be improved (see related discussions in Bentkus, 2004).

REMARK 3.2. Theorem 3.1 implies that our testing procedures are consistent. When the null holds, we have h_n = 0 and hence γ_n = 0. This together with equation (2.1) suggests that our tests have correct size under the null. Under the alternative, we have h_n ≠ 0 and hence γ_n ≠ 0. Since χ²(r,0) is stochastically smaller than χ²(r,γ_n), (3.5) implies that our tests have non-negligible powers under H_a. We summarize these results in the following corollary.

COROLLARY 3.1. Assume Conditions (A1)-(A3) and (3.4) hold, s + m = o(n^1/3), λ_max((CC^T )^⎺1) = O(1), and $\left|\widehat{\varphi }-{\varphi }_0\right|=o_p(1)$. Then, under the null hypothesis, for any 0 < α < 1, we have

$$\underset{n}{\lim }Pr\left(T>{\chi }_{\alpha }^2(r)\right)=\alpha ,$$

for T = T_W ,T_L and T_S, where ${\chi }_{\alpha }^2(r)$ is the critical value of χ²-distribution with r degrees of freedom at level α. Under the alternative $\mathit{C}{\mathit{\beta }}_{0,\mathcal{M}}-\mathit{t}={\mathit{h}}_n$ for some h_n satisfying ${\mathit{h}}_n=O\left(\sqrt{\min (s+m-r,r)/n}\right)$, we have for any 0 < α < 1, and T = T_W ,T_S and T_L,

$$\underset{n}{\lim }\left|Pr\left(T>{\chi }_{\alpha }^2(r)\right)-Pr\left({\chi }^2\left(r,{\gamma }_n\right)>{\chi }_{\alpha }^2(r)\right)\right|=0,$$

where ${\gamma }_n=n{\mathit{h}}_n^T{\left(\mathit{C}{\mathbf{\Omega }}_{mm}{\mathit{C}}^T\right)}^{-1}{\mathit{h}}_n/{\varphi }_0.$

REMARK 3.3. Corollary 3.1 shows that the asymptotic power functions of the proposed test statistics are

$$\mathrm{Pr}\left({\chi }^2\left(r,{\gamma }_n\right)>{\chi }_{\alpha }^2(r)\right).$$ (3.6)

It follows from Theorem 2 in Ghosh (1973) that the asymptotic power function decreases as r increases for a given γ_n. This is the same as that for traditional likelihood ratio test, score test and Wald’s test. However, h_n is an r-dimensional vector in our setting. Thus, one may easily construct an example in which γ_n grows as r increases. As a result, the asymptotic power function may not be monotone increasing function of r.

In Section S3 of Shi et al. (2018), we study in depth that how the penalty on individual coefficient affects the power, and find that the tests are most advantageous if each unpenalized variable is either an important variable (i.e., in $\mathcal{S}$) or a variable in $\mathcal{M}$.

REMARK 3.4. Notice that the null hypothesis reduces to ${\mathit{\beta }}_{0,\mathcal{M}}=\mathbf{0}$ if we set C to be the identity matrix and t = 0. The Wald test based on the desparsified Lasso estimator (van de Geer et al., 2014) and the decorrelated score test (Ning and Liu, 2017) can also be applied to testing such hypothesis. Based on (3.6), we show that these two tests achieve less power than the proposed partial penalized tests in Section S1 of Shi et al. (2018). This is due to the increased variances of the de-sparsified Lasso estimator and the decorrelated score statistic after the debiasing procedure.

3.3. Some implementation issues.

3.3.1. Constrained partial penalized regression.

To construct our test statistics, we need to compute the partial penalized estimators ${\widehat{\mathit{\beta }}}_0$ and ${\widehat{\mathit{\beta }}}_a$. Our algorithm is based upon the alternating direction method of multipliers (ADMM), which is a variant of standard augmented Lagrangian method. Below, we present our algorithm for estimating ${\widehat{\mathit{\beta }}}_0$. The unconstrained estimator ${\widehat{\mathit{\beta }}}_a$ can be similarly computed. For a fixed regularization parameter λ, define

$${\widehat{\mathit{\beta }}}_0^{\lambda }=\arg \underset{\mathit{\beta }}{\min }\left(-L_n(\mathit{\beta })+\sum _{j\in {\mathcal{M}}^c}{p}_{\lambda }\left(\left|{\mathit{\beta }}_j\right|\right)\right),\text{subject}\text{to}\mathit{C}{\mathit{\beta }}_{\mathcal{M}}=\mathit{t}.$$

The above optimization problem is equivalent to

$$\begin{gathered} ({\widehat{\mathit{\beta }}}_0^{\lambda },{\widehat{\mathit{\theta }}}_0^{\lambda })=\arg \underset{\mathit{\theta }\in {ℝ}^{p-m}}{\underset{\mathit{\beta }\in {ℝ}^p}{\min }}\left(-L_n(\mathit{\beta })+\sum _{j=1}^{p-m}{p}_{\lambda }\left(\left|{\theta }_j\right|\right)\right), \\ \text{subject}\text{to}\mathit{C}{\mathit{\beta }}_{\mathcal{M}}=\mathit{t},{\mathit{\beta }}_{{\mathcal{M}}^c}=\mathit{\theta }. \end{gathered}$$ (3.7)

The augmented Lagrangian for (3.7) is

$$\begin{gathered} L_{\rho }(\mathit{\beta },\mathit{\theta },\mathit{v})=-L_n(\mathit{\beta })+\sum _{j=1}^{p-m}{p}_{\lambda }\left(\left|{\theta }_j\right|\right)+{\mathit{v}}^T\left(\begin{array}{c}\mathit{C}{\mathit{\beta }}_{\mathcal{M}}-\mathit{t}\\ {\mathit{\beta }}_{{\mathcal{M}}^c}-\mathit{\theta }\end{array}\right) \\ +\frac{\rho }{2}{\Vert \mathit{C}{\mathit{\beta }}_{\mathcal{M}}-\mathit{t}\Vert }_2^2+\frac{\rho }{2}{\Vert {\mathit{\beta }}_{{\mathcal{M}}^c}-\mathit{\theta }\Vert }_2^2, \end{gathered}$$

for a given ρ > 0. Applying dual ascent method yields the following algorithm:

$$\begin{gathered} {\mathit{\beta }}^{k+1}=\arg \underset{{}_{\mathit{\beta }}}{\min }\left\{{\left({\mathit{v}}^k\right)}^T\left(\begin{array}{c}\mathit{C}{\mathit{\beta }}_{\mathcal{M}}-\mathit{t}\\ {\mathit{\beta }}_{{\mathcal{M}}^c}-{\mathit{\theta }}^k\end{array}\right)+\frac{\rho }{2}{\Vert \begin{array}{c}\mathit{C}{\mathit{\beta }}_{\mathcal{M}}-\mathit{t}\\ {\mathit{\beta }}_{{\mathcal{M}}^c}-{\mathit{\theta }}^k\end{array}\Vert }_2^2-L_n(\mathit{\beta })\right\}, \\ {\mathit{\theta }}^{k+1}=\arg \underset{{}_{{}_{\mathit{\theta }}}}{\min }\left\{\sum _{j=1}^{p-m}{p}_{\lambda }\left(\left|{\theta }_j\right|\right)+\frac{\rho }{2}{\Vert {\mathit{\beta }}_{{\mathcal{M}}^c}^{k+1}-\mathit{\theta }\Vert }_2^2+{\left({\mathit{v}}^k\right)}^T\left(\begin{array}{c}\mathit{C}{\mathit{\beta }}_{\mathcal{M}}^{k+1}-\mathit{t}\\ {\mathit{\beta }}_{{\mathcal{M}}^c}^{k+1}-\mathit{\theta }\end{array}\right)\right\}, \\ {\mathit{v}}^{k+1}={\mathit{v}}^k+\rho \left(\begin{array}{c}\mathit{C}{\mathit{\beta }}_{\mathcal{M}}^{k+1}-\mathit{t}\\ {\mathit{\beta }}_{{\mathcal{M}}^c}^{k+1}-{\mathit{\theta }}^{k+1}\end{array}\right), \end{gathered}$$

for the (k + 1)th iteration.

Since L_n is twice differentiable, β^k+1 can be obtained by the Newton-Raphson algorithm. θ^k+1 may have a closed form for some popular penalties such as Lasso, SCAD or MCP penalty. In our implementation, we use the SCAD penalty,

$$p_{\lambda }\left(\left|{\beta }_j\right|\right)=\lambda {\int }_0^{\left|{\beta }_j\right|}\left\{I(t\le \lambda )+\frac{{(a\lambda -t)}_{+}}{a-1}I(t>\lambda )\right\}dt,$$

and set a = 3.7, 𝜌=1,

To obtain ${\widehat{\mathit{\beta }}}_0$, we compute ${\widehat{\mathit{\beta }}}_0^{\lambda }$ for a series of log-spaced values in [−λ_min, λ_max] for some λ_min < λ_max. Then we choose ${\widehat{\mathit{\beta }}}_0={\widehat{\mathit{\beta }}}_0^{\widehat{\lambda }}$ by minimizing the following information criterion:

$$\widehat{\lambda }=\arg \underset{{}_{\lambda }}{\min }\left(-n{L}_n(\lambda )+c_n{\Vert {\widehat{\mathit{\beta }}}^{\lambda }\Vert }_0\right),$$

where c_n = max{logn, log(log(n))log(p)}. Using similar arguments in Schwarz (1978) and Fan and Tang (2013), we can show such information criterion is consistent in both fixed p and ultrahigh dimension setting.

3.3.2. Estimation of the nuisance parameter.

It can be shown that ϕ₀ = 1 for logistic or Poisson regression models. In linear regression models, we have ${\varphi }_0=\text{E}{\left(Y_i-{\mathit{\beta }}_0^T{\mathit{X}}_i\right)}^2$. In our implementation, we estimate ϕ₀ by

$$\widehat{\varphi }=\frac{1}{n-\left|{\widehat{S}}_a\right|-m}\sum _{i=1}^n{\left(Y_i-{\widehat{\mathit{\beta }}}_a^T{\mathit{X}}_i\right)}^2,$$

where ${\widehat{\mathit{\beta }}}_a$ is defined in (2.3).

In Section S2 of the supplementary material (Shi et al., 2018), we show $\widehat{\varphi }={\varphi }_0+O_p\left(n^{-1/2}\right)$, under the conditions in Theorem 2.1, which implies selection consistency. Alternatively, one can estimate ϕ₀ using refitted cross-validation (Fan, Guo and Hao, 2012) or scaled lasso (Sun and Zhang, 2013).

4. Numerical Examples.

In this section, we examine the finite sample performance of the proposed tests. Simulation results for linear regression and logistic regression are presented in the main text. In the supplementary material (Shi et al., 2018), we present simulation results for Poisson log-linear model and illustrate the proposed methodology by a real data example.

4.1. Linear regression.

Simulated data with sample size n = 100 were generated from

$$Y=2X_1-\left(2+h^{(1)}\right)X_2+h^{(2)}{X}_3+\epsilon$$

where ε ∼ N(0,1) and X ∼ N(0_p,Σ), and h⁽¹⁾ and h⁽²⁾ are some constants. The true value ${\mathit{\beta }}_0={\left(2,-2-h^{(1)},h^{(2)},{\mathbf{0}}_{p-3}^T\right)}^T$ where 0_q denotes a zero vector of length q.

4.1.1. Testing linear hypothesis.

We focus on testing the following three pairs of hypotheses:

$$\begin{gathered} H_0^{(1)}:{\beta }_{0,1}+{\beta }_{0,2}=0,v.s{H}_a^{(1)}:{\beta }_{0,1}+{\beta }_{0,2}\ne 0. \\ {H}_0^{(2)}:{\beta }_{0,2}+{\beta }_{0,3}=-2,v.s{H}_a^{(2)}:{\beta }_{0,2}+{\beta }_{0,3}\ne -2. \\ {H}_0^{(3)}:{\beta }_{0,3}+{\beta }_{0,4}=0,v.s{H}_a^{(3)}:{\beta }_{0,3}+{\beta }_{0,4}\ne 0. \end{gathered}$$

These hypotheses test linear structures between two regression coefficients. When testing $H_0^{(1)}$, we set h⁽²⁾ = 0, and hence $H_0^{(1)}$ holds if and only if h⁽¹⁾ = 0. Similarly when testing $H_0^{(2)}$ and $H_0^{(3)}$, we set h⁽¹⁾ = 0, and hence the hull hypotheses hold if and only if h⁽²⁾ = 0.

We consider two different dimensions, p = 50 and p = 200, and two different covariance matrices Σ, corresponding to Σ = I and $\mathbf{\Sigma }={\{{0.5}^{|i-j|}\}}_{i,j=1,\dots ,p}$. This yields a total of 4 settings. For each hypothesis and each setting, we further consider four scenarios, by setting h^(j) = 0,0.1,0.2,0.4. Therefore, the null holds under the first scenario and the alternative holds under the rest three. Table 1 summarizes the rejection probabilities for $H_0^{(1)},H_0^{(2)}$ and $H_0^{(3)}$ under the settings where Σ = {0.5^|i–j|}. Rejection probabilities of the proposed tests under the settings where Σ = I are given in Table S1 in the supplementary material. The rejection probabilities are evaluated via 600 simulation replications.

Table 1.

Rejection probabilities (%) of the partial penalized Wald, score and likelihood ratio statistics with standard errors in parenthesis (%), under the setting where $\mathbf{\Sigma }={\{{0.5}^{|i-j|}\}}_{i,j=1,\dots ,p}$.

	p = 50			p = 200
	TL	TW	TS	TL	TW	TS
h(1)	$H_0^{(1)}$
0	4.33(0.83)	4.33(0.83)	4.67(0.86)	5.67(0.94)	5.67(0.94)	5.67(0.94)
0.1	13.17(1.38)	13.50(1.40)	13.50(1.40)	11.67(1.31)	11.67(1.31)	11.67(1.31)
0.2	39.83(2.00)	40.17(2.00)	40.00(2.00)	39.67(2.00)	39.67(2.00)	39.67(2.00)
0.4	92.33(1.09)	93.17(1.03)	93.17(1.03)	92.67(1.06)	92.67(1.06)	92.67(1.06)
h(2)	$H_0^{(2)}$
0	5.17(0.90)	5.17(0.90)	5.67(0.94)	5.33(0.92)	5.33(0.92)	5.33(0.92)
0.1	11.00(1.28)	11.00(1.28)	11.33(1.29)	12.50(1.35)	12.50(1.35)	12.50(1.35)
0.2	30.67(1.88)	30.67(1.88)	31.00(1.89)	33.67(1.93)	33.67(1.93)	33.67(1.93)
0.4	85.17(1.45)	85.00(1.46)	85.00(1.46)	87.83(1.33)	87.83(1.33)	87.83(1.33)
h(2)	$H_0^{(3)}$
0	6.50 (1.01)	6.33(0.99)	6.50(1.01)	5.67(0.94)	5.67(0.94)	5.67(0.94)
0.1	11.83 (1.32)	11.67(1.31)	11.67(1.31)	11.00(1.28)	11.00(1.28)	11.00(1.28)
0.2	31.67 (1.90)	31.50(1.90)	31.67(1.90)	33.17(1.92)	33.17(1.92)	33.17(1.92)
0.4	84.33 (1.48)	84.17(1.49)	84.50(1.48)	86.00(1.42)	86.17(1.41)	86.17(1.41)

Based on the results, it can be seen that under these null hypotheses, Type I error rates of the three tests are well controlled and close to the nominal level for all four settings. Under the alternative hypotheses, the powers of these three test statistics increase as h⁽¹⁾ or h⁽²⁾ increases, showing the consistency of our testing procedure. Moreover, the empirical rejection rates between these three test statistics are very close across all different scenarios and settings. For example, the rejection rates are exactly the same for testing $H_0^{(1)}$ and $H_0^{(2)}$ when p = 200 in Table 1, although we observed that the values of these three statistics in our simulation are slightly different. This is consistent with our theoretical findings that these statistics are asymptotically equivalent even in high dimensional settings. Figures S1, S2 and S3 in the supplementary material depicts the kernel density estimates of three test statistics under $H_0^{(1)}$ and $H_0^{(2)}$ with different combinations of p and the covariance matrices respectively. It can be seen that these three test statistics converge to their limiting distributions under the null hypotheses.

4.1.2. Testing univariate parameter.

Consider testing the following two pairs of hypotheses:

$$\begin{gathered} H_0^{(4)}:{\beta }_{0,2}=-2,v.s{H}_a^{(1)}:{\beta }_{0,2}\ne -2. \\ {H}_0^{(5)}:{\beta }_{0,3}=0,v.s{H}_a^{(2)}:{\beta }_{0,3}\ne 0. \end{gathered}$$

We set h⁽²⁾ = 0 when testing $H_0^{(4)}$, and set h⁽¹⁾ = 0 when testing $H_0^{(5)}$. Therefore, $H_0^{(4)}$ is equivalent to h⁽¹⁾ = 0 and $H_0^{(5)}$ is equivalent to h⁽²⁾ = 0. We use the same 4 settings described in Section 4.1.1. For each setting, we set h⁽¹⁾ = 0.1,0.2,0.4 under $H_a^{(4)}$ and h⁽²⁾ = 0.1,0.2,0.4 under $H_a^{(5)}$. Comparison is made among the following test statistics:

1. The proposed likelihood ratio (T_L), Wald (T_W ) and score (T_S) statistic.

2. The Wald test statistic based on the de-sparisfied Lasso estimator $\left(T_W^D\right)$.

3. The decorrelated score statistic. $\left(T_S^D\right)$.

The test statistic $T_W^D$ is computed via the R package hdi (Dezeure et al., 2015). We calculate $T_S^D$ according to Section 4.1 in Ning and Liu (2017). More specifically, the initial estimator $\widehat{\mathit{\beta }}$ is computed by a penalized linear regression with SCAD penalty function, and $\widehat{\mathit{\omega }}$ is computed by a penalized linear regression with l₁ penalty function (see Equation (4.4) in Ning and Liu, 2017). These penalized regressions are implemented via the R package ncvreg (Breheny and Huang, 2011). The tuning parameters are selected via 10-folded cross-validation. The rejection probabilities of these test statistics under the settings where Σ = {0.5^|i–j|} are reported in Table 2. In the supplementary material, we report the rejection probabilities of these test statistics under the settings where Σ = I in Table S2. Results are averaged over 600 simulation replications.

Table 2.

Rejection probabilities (%) of the partial penalized Wald, score and likelihood ratio statistics, the Wald test statistic based on the de-sparsified Lasso estimator and the decorrelated score statistic under the settings where $\mathbf{\Sigma }={\{{0.5}^{|i-j|}\}}_{i,j=1,\dots ,p}$, with standard errors in parenthesis (%).

	TL	TW	TS	$T_W^D$	$T_S^D$
h(1)	$H_0^{(4)}$ and p = 50
0	5.17(0.90)	5.33(0.92)	5.50(0.93)	12.67(1.36)	7.00(1.04)
0.1	15.67(1.48)	16.00(1.50)	16.00(1.50)	6.00(0.97)	14.67(1.44)
0.2	41.00(2.01)	41.33(2.01)	41.50(2.01)	14.83(1.45)	38.83(1.99)
0.4	92.50(1.08)	93.00(1.04)	93.00(1.04)	67.67(1.91)	88.67(1.29)
	$H_0^{(4)}$ and p = 200
0	4.83(0.88)	4.83(0.88)	4.83(0.88)	21.83(1.69)	5.50(0.93)
0.1	11.00(1.28)	11.00(1.28)	11.00(1.28)	5.83(0.96)	10.83(1.27)
0.2	40.50(2.00)	40.50(2.00)	40.50(2.00)	6.17(0.98)	37.83(1.98)
0.4	91.50(1.14)	91.50(1.14)	91.50(1.14)	49.33(2.04)	88.00(1.33)
h(2)	$H_0^{(5)}$ and p = 50
0	6.33(0.99)	6.00(0.97)	6.50(1.00)	5.33(0.92)	3.00(0.70)
0.1	13.67(1.40)	13.50(1.40)	14.00(1.42)	5.33(0.92)	9.17(1.18)
0.2	40.17(2.00)	40.33(2.00)	40.50(2.00)	15.67(1.48)	28.50(1.84)
0.4	90.83(1.18)	91.33(1.15)	91.67(1.13)	69.17(1.89)	83.33(1.52)
	$H_0^{(5)}$ and p = 200
0	5.67(0.94)	5.67(0.94)	5.67(0.94)	6.50(1.01)	2.67(0.66)
0.1	13.67(1.40)	13.67(1.40)	13.67(1.40)	3.67(0.77)	8.17(1.12)
0.2	39.17(1.99)	39.17(1.99)	39.17(1.99)	9.67(1.21)	24.67(1.76)
0.4	91.50(1.14)	91.50(1.14)	91.50(1.14)	51.33(2.04)	80.50(1.62)

From Table 2, it can be seen that $T_W^D$ failed to test $H_0^{(4)}$ under the settings where Σ = {0.5^|i–j|}. Under the null hypotheses, the Type I error rates of $T_W^D$ are greater than 12%. Under the alternative hypotheses, the proposed test statistics and the decorrelated score test are more powerful than $T_W^D$ in almost all cases. Besides, we note that T_L, T_W, T_S and $T_W^D$ perform comparable under the settings where Σ = I. When Σ = {0.5^|i–j|} however, the proposed test statistics achieve greater power than $T_S^D$. This is in line with our theoretical findings (see Section S1 of the supplementary material for details).

4.1.3. Effects on m.

In Section 4.1.1, we consider linear hypotheses involving two parameters only. As suggested by one of the referee, we further examine our test statistics under settings where more regression parameters are involved in the hypotheses. More specifically, we consider the following three pairs of hypotheses:

$$\begin{gathered} H_0^{(6)}:\sum _{j=1}^4{\beta }_{0,j}=0,v.s{H}_a^{(6)}:\sum _{j=1}^4{\beta }_{0,j}\ne 0. \\ {H}_0^{(7)}:\sum _{j=1}^8{\beta }_{0,j}=0,v.s{H}_a^{(7)}:\sum _{j=1}^8{\beta }_{0,j}\ne 0. \\ {H}_0^{(8)}:\sum _{j=1}^{12}{\beta }_{0,j}=0,v.s{H}_a^{(8)}:\sum _{j=1}^{12}{\beta }_{0,j}\ne 0. \end{gathered}$$

The numbers of parameters involved in $H_0^{(6)},H_0^{(7)}$ and $H_0^{(8)}$ are equal to 4, 8 and 12, respectively. We consider the same 4 settings described in Section 4.1.1. For each setting, we set h⁽¹⁾ = 0,0.2,0.4,0.8 and h⁽²⁾ = 0. Hence, the null hypotheses hold when h⁽¹⁾ = 0 and the alternatives hold when h⁽¹⁾ > 0. We report the rejection probabilities over 600 replications in Table 3, under the settings where Σ = {0.5^|i–j|}. Rejection probabilities under the settings where Σ = I are reported in Table S3 in the supplementary material.

Table 3.

Rejection probabilities (%) of the partial penalized Wald, score and likelihood ratio statistics with standard errors in parenthesis (%), under the settings where $\mathbf{\Sigma }={\{{0.5}^{|i-j|}\}}_{i,j=1,\dots ,p}$.

	p = 50			p = 200
	TL	TW	TS	TL	TW	TS
h(1)	$H_0^{(6)}$
0	4.83(0.88)	4.50(0.85)	4.67(0.86)	4.83(0.88)	4.83(0.88)	4.83(0.88)
0.2	28.17(1.84)	28.17(1.84)	28.50(1.84)	28.50(1.84)	28.50(1.84)	28.50(1.84)
0.4	80.33(l.62)	80.17(1.63)	80.33(1.62)	79.83(1.64)	79.83(1.64)	79.83(1.64)
0.8	99.83(0.17)	100.00(0.00)	100.00(0.00)	100.00(0.00)	100.00(0.00)	100.00(0.00)
h(1)	$H_0^{(7)}$
0	4.50(0.85)	4.50(0.85)	4.50(0.85)	5.00(0.89)	5.00(0.89)	5.00(0.89)
0.2	18.17(1.57)	18.33(1.58)	18.33(1.58)	18.33(1.58)	18.33(1.58)	18.33(1.58)
0.4	53.83(2.04)	54.17(2.03)	54.00(2.03)	57.33(2.02)	57.33(2.02)	57.33(2.02)
0.8	98.50(0.50)	99.00(0.41)	99.00(0.41)	98.50(0.50)	98.50(0.50)	98.50(0.50)
h(1)	$H_0^{(8)}$
0	5.17 (0.90)	5.00(0.89)	5.17(0.90)	5.67(0.94)	5.67(0.94)	5.67(0.94)
0.2	14.33 (1.43)	14.33(1.43)	14.33(1.43)	13.67(1.40)	13.67(1.40)	13.67(1.40)
0.4	42.00 (2.01)	42.17(2.02)	42.17(2.02)	41.67(2.01)	41.67(2.01)	41.67(2.01)
0.8	92.83 (1.05)	92.83(1.05)	92.83(1.05)	93.00(1.04)	93.00(1.04)	93.00(1.04)

The Type I error rates of the three test statistics are close to the nominal level under the null hypotheses. The powers of the test statistics increase as h⁽¹⁾ increases, under the alternative hypotheses. Moreover, we note that the powers decrease as m increases. This is in line with Corollary 3.1 which states that the asymptotic power function of our test statistics is a function of r and γ_n. Recall that ${\gamma }_n=n{\mathit{h}}_n^T{\left(\mathit{C}{\mathbf{\Omega }}_{mm}{\mathit{C}}^T\right)}^{-1}{\mathit{h}}_n/{\varphi }_0$. Consider the following sequence of null hypotheses indexed by m ≥ 2: C_mβ₀ = 0 where C_m = (1,···,1,0_p−m). Let ${\gamma }_{n,m}=n{h}_n^T{\left(\mathit{C}{\mathbf{\Omega }}_{mm}{\mathit{C}}_m^T\right)}^{-1}{h}_n/{\varphi }_0$. Under the given settings, we have Ω_mm = (ω_ij) is a banded matrix with ω_ij = 0 for |i − j| ≥ 2, w_ij = –1/(1-ρ²) for |i − j| = 1, ω₁₁ = ω_mm = 1/{ρ(1 − ρ²)}, and ω_jj = (1 + ρ²)/{ρ(1 − ρ²)} for j ≠ 1 and m, where ρ is the auto-correlation between X₁ and X₂. It is immediate to see γ_n,m decreases as m increases.

4.2. Logistic regression.

In this example, we generate data with sample size n = 300 from the logistic regression model

$$\mathrm{logit}\{\mathrm{Pr}(Y=1|\mathit{X})\}=2X_1-\left(2+h^{(1)}\right)X_2+h^{(2)}{X}_3,$$

where logit(p) = log{p/(1−p)}, the logit link function, and X ∼ N(0_p,Σ).

4.2.1. Testing linear hypothesis.

We consider the same linear hypotheses as those in Section 4.1.1:

$$\begin{gathered} H_0^{(1)}:{\beta }_{0,1}+{\beta }_{0,2}=0,v.s{H}_a^{(1)}:{\beta }_{0,1}+{\beta }_{0,2}\ne 0. \\ {H}_0^{(2)}:{\beta }_{0,2}+{\beta }_{0,3}=-2,v.s{H}_a^{(2)}:{\beta }_{0,2}+{\beta }_{0,3}\ne -2. \\ {H}_0^{(3)}:{\beta }_{0,3}+{\beta }_{0,4}=0,v.s{H}_a^{(3)}:{\beta }_{0,3}+{\beta }_{0,4}\ne 0. \end{gathered}$$

Similarly, we set h⁽²⁾ = 0 when testing $H_0^{(1)}$, and set h⁽¹⁾ = 0 when testing $H_0^{(2)}$. Therefore, $H_0^{(1)}$ is equivalent to h⁽¹⁾ = 0 and $H_0^{(2)}$ is equivalent to h⁽²⁾ = 0. We use the same 4 settings described in Section 4.1.1. For each of the four settings, we set h^(j) = 0.2,0.4,0.8 under $H_a^{(j)}$. The rejection probabilities for $H_0^{(1)}$ and $H_0^{(2)}$ over 600 replications are given in Table S4 in the supplementary material. We also plot the kernel density estimates of three test statistics under $H_0^{(1)}$ and $H_0^{(2)}$ in Figures S4, S5 and S6 in the supplementary material. The findings are very similar to those in the previous examples.

4.2.2. Testing univariate parameter.

To compare the proposed partial penalized Wald (T_W ), score (T_S) and likelihood ratio (T_L) test statistics with the Wald test based on the de-sparsified Lasso estimator $\left(T_W^D\right)$ and the decorrelated score test $\left(T_S^D\right)$, we consider testing the following hypotheses:

$$\begin{gathered} H_0^{(4)}:{\beta }_{0,2}=-2,v.s{H}_a^{(4)}:{\beta }_{0,2}\ne -2. \\ {H}_0^{(5)}:{\beta }_{0,3}=0,v.s{H}_a^{(5)}:{\beta }_{0,3}\ne 0. \end{gathered}$$

Similar to Section 4.1.2, we set h⁽²⁾ = 0 when testing $H_0^{(4)}$, and set h⁽¹⁾ = 0 when testing $H_0^{(5)}$. We set h⁽¹⁾ = 0 under $H_0^{(4)}$, h⁽¹⁾ = 0.2, 0.4, 0.8 under $H_a^{(4)}$ and set h⁽²⁾ = 0 under $H_0^{(5)}$, h⁽²⁾ = 0.2, 0.4, 0.8 under $H_a^{(5)}$. We consider the same 4 settings described in Section 4.1.1. The test statistic $T_W^D$ is computed via the R package hdi and $T_S^D$ is obtained according to Section 4.2 of Ning and Liu (2017). We compute the initial estimator $\widehat{\mathit{\beta }}$ in $T_S^D$ by fitting a penalized logistic regression with SCAD penalty function, and calculate $\widehat{\mathit{\omega }}$ by fitting a penalized linear regression with l₁ penalty function. These penalized regressions are implemented via the R package ncvreg. We report the rejection probabilities of T_W, T_S,T_L, $T_W^D$ and $T_S^D$ in Table S5 in the supplementary article, based on 600 simulation replications.

Based on the results, it can be seen that the Type I error rates of $T_W^D$ and $T_S^D$ are significantly larger than the nominal level in almost all cases for testing $H_0^{(4)}$. On the other hand, the Type I error rates of the proposed test statistics are close to the nominal level under $H_0^{(4)}$. Besides, under $H_a^{(5)}$, the powers of the proposed test statistics are greater than or equal to $T_W^D$ and $T_S^D$ in all cases.

4.2.3. Effects on m.

As in Section 4.1.3, we further examine the proposed test statistics by allowing more regression coefficients to appear in the linear hypotheses. Similarly, we consider the following three pairs of hypotheses:

$$\begin{gathered} H_0^{(6)}:\sum _{j=1}^4{\beta }_{0,j}=0,v.s{H}_a^{(6)}:\sum _{j=1}^4{\beta }_{0,j}\ne 0. \\ {H}_0^{(7)}:\sum _{j=1}^8{\beta }_{0,j}=0,v.s{H}_a^{(7)}:\sum _{j=1}^8{\beta }_{0,j}\ne 0. \\ {H}_0^{(8)}:\sum _{j=1}^{12}{\beta }_{0,j}=0,v.s{H}_a^{(8)}:\sum _{j=1}^{12}{\beta }_{0,j}\ne 0. \end{gathered}$$

We set h⁽²⁾ = 0, and set h⁽¹⁾ = 0 under the null hypotheses, h⁽¹⁾ = 0.4,0.8,1.6 under the alternative hypotheses. The same 4 settings described in Section 4.1.1 are used. The rejection probabilities of the proposed test statistics are reported in Table S6 in the supplementary article. Results are averaged over 600 replications. Findings are very similar to those in Section 4.1.3.

5. Technical proofs.

This section consists of the proof of Theorem 3.1. To establish Theorem 3.1, we need the following lemma. The proof of this lemma is given in Section 5.1. For any symmetric and positive definite matrix A ∈ ℝ^q×^q, it follows from the spectral theorem that A = U^T ΛU for some orthogonal matrix U and diagonal matrix Λ = diag(λ₁,...,λ_q). Since the diagonal elements in Λ are positive, we use Λ^1/2 and Λ−^1/2 to denote the diagonal matrices $\text{diag}({\lambda }_1^{1/2},\dots ,{\lambda }_q^{1/2})$ and $\text{diag}({\lambda }_1^{-1/2},\dots ,{\lambda }_q^{-1/2})$, respectively. In addition, we define A^1/2 = U^T Λ^1/2U and A^‒1/2 = U^T Λ⎺^1/2U.

LEMMA 5.1. Under the conditions in Theorem 3.1, we have

$${\lambda }_{\max }\left({\mathit{K}}_n\right)=O(1),$$ (5.1)

$${\lambda }_{\max }\left({\mathit{K}}_n^{1/2}\right)=O(1),$$ (5.2)

$${\lambda }_{\max }\left({\mathit{K}}_n^{-1/2}\right)=O(1),$$ (5.3)

$${\lambda }_{\max }\left({\left(\mathit{C}{\mathbf{\Omega }}_{mm}{\mathit{C}}^T\right)}^{-1}\right)=O(1),$$ (5.4)

$${\Vert {\mathbf{\Psi }}^{-1/2}\mathit{C}\Vert }_2=O(1),$$ (5.5)

$${\Vert {\mathbf{\Psi }}^{1/2}{(\mathit{C}{\widehat{\mathbf{\Omega }}}_{a,mm}{\mathit{C}}^T)}^{-1}{\mathbf{\Psi }}^{1/2}-\mathit{I}\Vert }_2=O_p\left(\frac{s+m}{\sqrt{n}}\right),$$ (5.6)

$${\Vert I-{\mathit{K}}_n^{1/2}{\widehat{\mathit{K}}}_{n,0}^{-1}{\mathit{K}}_n^{1/2}\Vert }_2=O_p\left(\frac{s+m}{\sqrt{n}}\right),$$ (5.7)

where Ψ = CΩ_mm C^T and

$${\widehat{\mathit{K}}}_{n,0}=\frac{1}{n}\left(\begin{array}{cc}{\mathit{X}}_{\mathcal{M}}^T\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_0\right){\mathit{X}}_{\mathcal{M}}& {\mathit{X}}_{\mathcal{M}}^T\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_0\right){\mathit{X}}_S\\ {\mathit{X}}_S^T\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_0\right){\mathit{X}}_{\mathcal{M}}& {\mathit{X}}_S^T\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_0\right){\mathit{X}}_S\end{array}\right).$$

We break the proof into four steps. In the first three steps, we show T_W /r, T_S/r and T_L/r are equivalent to T₀/r, respectively, where

$$T_0=\frac{1}{{\varphi }_0}{\left({\mathit{\omega }}_n+\sqrt{n}{\mathit{h}}_n\right)}^T{\left(\mathit{C}{\mathbf{\Omega }}_{mm}{\mathit{C}}^T\right)}^{-1}\left({\mathit{\omega }}_n+\sqrt{n}{\mathit{h}}_n\right),$$

and

$${\mathit{\omega }}_n=\frac{1}{\sqrt{n}}{\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{s\times r}^T\end{array}\right)}^T{\mathit{K}}_n^{-1}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\}.$$

In the final step, we show the χ² approximation (3.5) holds for T_W ,T_S and T_L.

Step 1: We first show that T_W /r is equivalent to T₀/r. It follows from Theorem 2.1 that

$$\sqrt{n}\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-{\mathit{\beta }}_{0,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{a,S}-{\mathit{\beta }}_{0,S}\end{array}\right)=\frac{1}{\sqrt{n}}{\mathit{K}}_n^{-1}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\}+{\mathit{R}}_a,$$

for some vector R_a that satisfies

$${\Vert {\mathit{R}}_a\Vert }_2=o_p(1).$$ (5.8)

Therefore, we have

$$\sqrt{n}\mathit{C}({\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-{\mathit{\beta }}_{0,\mathcal{M}})={\mathit{\omega }}_n+\mathit{C}{\mathit{R}}_{a,J_0},$$ (5.9)

where J₀ = [1,...,m]. Since $\mathit{C}{\mathit{\beta }}_{0,\mathcal{M}}=\mathit{t}+{\mathit{h}}_n$, it follows from (5.9) that

$$\sqrt{n}(\mathit{C}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-\mathit{t})={\mathit{\omega }}_n+\mathit{C}{\mathit{R}}_{a,J_0}+\sqrt{n}{\mathit{h}}_n,$$

and hence

$$\sqrt{n}{\mathbf{\Psi }}^{-1/2}(\mathit{C}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-\mathit{t})={\mathbf{\Psi }}^{-1/2}({\mathit{\omega }}_n+\mathit{C}{\mathit{R}}_{a,J_0}+\sqrt{n}{\mathit{h}}_n).$$ (5.10)

By (5.8) and (5.5) in Lemma 5.1, we have

$${\Vert {\mathbf{\Psi }}^{-1/2}\mathit{C}{\mathit{R}}_{a,J_0}\Vert }_2\le \Vert {\mathbf{\Psi }}^{-1/2}\mathit{C}\Vert {\Vert {\mathit{R}}_{a,J_0}\Vert }_2\le \Vert {\mathbf{\Psi }}^{-1/2}\mathit{C}\Vert {\Vert {\mathit{R}}_a\Vert }_2=o_p(1).$$

This together with (5.10) gives

$$\sqrt{n}{\mathbf{\Psi }}^{-1/2}(\mathit{C}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-\mathit{t})={\mathbf{\Psi }}^{-1/2}({\mathit{\omega }}_n+\sqrt{n}{\mathit{h}}_n)+o_p(1).$$ (5.11)

Note that

$$\text{E}{\Vert {\mathbf{\Psi }}^{-1/2}{\mathit{\omega }}_n\Vert }_2^2=\mathrm{tr}\left({\mathbf{\Psi }}^{-1/2}\text{E}{\mathit{\omega }}_n{\mathit{\omega }}_n^T{\mathbf{\Psi }}^{-1/2}\right)={\varphi }_0\mathrm{tr}\left({\mathbf{\Psi }}^{-1/2}\mathbf{\Psi }{\mathbf{\Psi }}^{-1/2}\right)=r{\varphi }_0.$$

By Markov’s inequality, we have

$${\Vert {\mathbf{\Psi }}^{-1/2}{\mathit{\omega }}_n\Vert }_2=O_p(\sqrt{r}).$$ (5.12)

Besides, it follows from (5.4) in Lemma 5.1 and Condition (A4) that

$${\Vert \sqrt{n}{\mathbf{\Psi }}^{-1/2}{\mathit{h}}_n\Vert }_2=O(\sqrt{r}).$$ (5.13)

This together with (5.11) and (5.12) implies that

$${\Vert \sqrt{n}{\mathbf{\Psi }}^{-1/2}(\mathit{C}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-\mathit{t})\Vert }_2=O_p(\sqrt{r}).$$ (5.14)

Combining this together with (5.6) in Lemma 5.1 gives

$$\begin{gathered} {\Vert {\{\sqrt{n}{\mathbf{\Psi }}^{-1/2}(\mathit{C}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-\mathit{t})\}}^T\{{\mathbf{\Psi }}^{1/2}{(\mathit{C}{\widehat{\mathbf{\Omega }}}_{a,mm}{\mathit{C}}^T)}^{-1}{\mathbf{\Psi }}^{1/2}-\mathit{I}\}\{\sqrt{n}{\mathbf{\Psi }}^{-1/2}(\mathit{C}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-\mathit{t})\}\Vert }_2^2 \\ \le {\Vert \{\sqrt{n}{\mathbf{\Psi }}^{-1/2}(\mathit{C}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-\mathit{t})\}\Vert }_2^2{\Vert \{{\mathbf{\Psi }}^{1/2}{(\mathit{C}{\widehat{\mathbf{\Omega }}}_{a,mm}{\mathit{C}}^T)}^{-1}{\mathbf{\Psi }}^{1/2}-\mathit{I}\}\Vert }_2=O_p\left(\frac{r(s+m)}{\sqrt{n}}\right). \end{gathered}$$

The last term is o_p(r) under the condition s+m = o(n^1/3). By the definition of T_W, we have shown that

$$\widehat{\varphi }\left|T_W-T_{W,0}\right|=o_p(r),$$ (5.15)

where

$$T_{W,0}=\frac{n{(\mathit{C}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-\mathit{t})}^T{\mathbf{\Psi }}^{-1}(\mathit{C}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-\mathit{t})}{\widehat{\varphi }}.$$

Under the conditions in Theorem 3.1, we have $\widehat{\varphi }={\varphi }_0+o_p(1)$. Since ϕ₀ > 0, we have

$$1/\widehat{\varphi }=O_p(1),$$ (5.16)

which together with (5.15) entails that T_W = T_W,0 + o_p(r).

It follows from (5.10)-(5.13) and the condition s + m = o(n^1/3) that

$$\begin{gathered} \widehat{\varphi }{T}_{W,0}={\Vert {\mathbf{\Psi }}^{-1/2}{\mathit{\omega }}_n+\sqrt{n}{\mathbf{\Psi }}^{-1/2}{\mathit{h}}_n+o_p(1)\Vert }_2^2 \\ ={\Vert {\mathbf{\Psi }}^{-1/2}{\mathit{\omega }}_n+\sqrt{n}{\mathbf{\Psi }}^{-1/2}{\mathit{h}}_n\Vert }_2^2+o_p(1)+o_p\left({\mathbf{\Psi }}^{-1/2}\left({\mathit{\omega }}_n+\sqrt{n}{\mathit{h}}_n\right)\right) \\ ={\Vert {\mathbf{\Psi }}^{-1/2}{\mathit{\omega }}_n+\sqrt{n}{\mathbf{\Psi }}^{-1/2}{\mathit{h}}_n\Vert }_2^2+o_p(1)+o_p(r) \\ ={\Vert {\mathbf{\Psi }}^{-1/2}{\mathit{\omega }}_n+\sqrt{n}{\mathbf{\Psi }}^{-1/2}{\mathit{h}}_n\Vert }_2^2+o_p(r)=\widehat{\varphi }{T}_{W,1}+o_p(r), \end{gathered}$$ (5.17)

where

$$T_{W,1}=\frac{{\Vert {\mathbf{\Psi }}^{-1/2}{\mathit{\omega }}_n+\sqrt{n}{\mathbf{\Psi }}^{-1/2}{\mathit{h}}_n\Vert }_2^2}{\widehat{\varphi }}.$$

By (5.16), we obtain T_W,0 = T_W,1 + o_p(r) and hence T_W = T_W,1 + o_p(r). In the following, we show T_W,1 = T₀ + o_p(r).

Observe that

$$\left|T_{W,1}-T_0\right|=\frac{\left|{\varphi }_0-\widehat{\varphi }\right|}{\widehat{\varphi }{\varphi }_0}{\Vert {\mathbf{\Psi }}^{-1/2}{\mathit{\omega }}_n+\sqrt{n}{\mathbf{\Psi }}^{-1/2}{\mathit{h}}_n\Vert }_2^2.$$ (5.18)

It follows from (5.12), (5.13), (5.16) and the condition $|\widehat{\varphi }-{\varphi }_0|=o_p(1)$ that right-hand side (RHS) of (5.18) is of the order o_p(r). This proves T_W,1 = T₀ + o_p(r).

Step 2: We show that T_S/r is equivalent to T₀/r. Based on the proof of Theorem 2.1 in Section S5.1 of the supplementary article, we have

$$\begin{gathered} \frac{1}{\sqrt{n}}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\widehat{\mathit{\beta }}}_0\right)\right\}=\frac{1}{\sqrt{n}}{\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)}^T\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\} \\ -\frac{1}{\sqrt{n}}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\mathbf{\Sigma }\left(\mathit{X}{\mathit{\beta }}_0\right){\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)}^T\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}-{\mathit{\beta }}_{0,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{0,S}-{\mathit{\beta }}_{0,S}\end{array}\right)+o_p(1), \end{gathered}$$ (5.19)

and

$$\begin{gathered} \sqrt{n}\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}-{\mathit{\beta }}_{0,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{0,S}-{\mathit{\beta }}_{0,S}\end{array}\right)=\frac{1}{\sqrt{n}}{\mathit{K}}_n^{-1/2}\left(\mathit{I}-{\mathit{P}}_n\right){\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\} \\ -\sqrt{n}{\mathit{K}}_n^{-1}\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right){\mathbf{\Psi }}^{-1}{\mathit{h}}_n+o_p(1). \end{gathered}$$ (5.20)

Combining (5.1) with (5.20) gives

$$\begin{gathered} \sqrt{n}{\mathit{K}}_n\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}-{\mathit{\beta }}_{0,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{0,S}-{\mathit{\beta }}_{0,S}\end{array}\right)=\frac{1}{\sqrt{n}}{\mathit{K}}_n^{1/2}\left(\mathit{I}-{\mathit{P}}_n\right){\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\} \\ -\sqrt{n}\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right){\mathbf{\Psi }}^{-1}{\mathit{h}}_n+o_p(1), \end{gathered}$$

which together with (5.19) implies that

$$\begin{gathered} \frac{1}{\sqrt{n}}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\widehat{\mathit{\beta }}}_0\right)\right\}=\frac{1}{\sqrt{n}}{\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)}^T\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\}+o_p(1) \\ -\frac{1}{\sqrt{n}}{\mathit{K}}_n^{1/2}\left(\mathit{I}-{\mathit{P}}_n\right){\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\}+\sqrt{n}\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right){\mathbf{\Psi }}^{-1}{\mathit{h}}_n \\ =\frac{1}{\sqrt{n}}{\mathit{K}}_n^{1/2}{\mathit{P}}_n{\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\}+\sqrt{n}\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right){\mathbf{\Psi }}^{-1}{\mathit{h}}_n+o_p(1). \end{gathered}$$

By (5.3), we have

$$\begin{gathered} \frac{1}{\sqrt{n}}{\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\widehat{\mathit{\beta }}}_0\right)\right\}=\frac{1}{\sqrt{n}}{\mathit{P}}_n{\mathit{K}}_n^{-1/2}{\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)}^T\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\} \\ +\sqrt{n}{\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right){\mathbf{\Psi }}^{-1}{\mathit{h}}_n+o_p(1). \end{gathered}$$ (5.21)

It follows from (5.5) and (5.13) that

$${\Vert \left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right){\mathbf{\Psi }}^{-1}{\mathit{h}}_n\Vert }_2\le {\Vert {\mathit{C}}^T{\mathbf{\Psi }}^{-1/2}\Vert }_2{\Vert {\mathbf{\Psi }}^{-1/2}{\mathit{h}}_n\Vert }_2=O_p(\sqrt{r/n}).$$

This together with (5.3) yields

$$\sqrt{n}{\Vert {\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right){\mathbf{\Psi }}^{-1}{\mathit{h}}_n\Vert }_2=O_p(\sqrt{r}).$$ (5.22)

Notice that

$$\mathrm{E}{\Vert \frac{1}{\sqrt{n}}{\mathit{P}}_n{\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\}\Vert }_2^2=\mathrm{tr}\left({\mathit{P}}_n\right)=\mathrm{rank}\left({\mathit{P}}_n\right)=r.$$

It follows from Markov’s equality that

$${\Vert \frac{1}{\sqrt{n}}{\mathit{P}}_n{\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\}\Vert }_2=O_p(\sqrt{r}).$$

Combining this with (5.21) and (5.22) yields

$${\Vert \frac{1}{\sqrt{n}}{\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\widehat{\mathit{\beta }}}_0\right)\right\}\Vert }_2=O_p(\sqrt{r}).$$ (5.23)

This together with (5.7) and the condition s + m = o(n^1/3) gives that

$$\begin{gathered} \left|\frac{1}{n}{\left\{\right(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array})\{\mathit{Y}-\mathit{\mu }(\mathit{X}{\widehat{\mathit{\beta }}}_0)\}\}}^T({\mathit{K}}_n^{-1}-{\widehat{\mathit{K}}}_{n,0}^{-1}){\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)}^T\{\mathit{Y}-\mathit{\mu }(\mathit{X}{\widehat{\mathit{\beta }}}_0)\}\right|\le \\ \frac{1}{n}{\Vert {\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\{\mathit{Y}-\mathit{\mu }(\mathit{X}{\widehat{\mathit{\beta }}}_0)\}\Vert }_2^2{\Vert \mathit{I}-{\mathit{K}}_n^{1/2}{\widehat{\mathit{K}}}_{n,0}^{-1}{\mathit{K}}_n^{1/2}\Vert }_2=O_p\left(\frac{r(s+m)}{\sqrt{n}}\right)=o_p(r). \end{gathered}$$

When ${\widehat{S}}_0=S$, we have

$$\widehat{\varphi }{T}_S=\frac{1}{n}{\left\{\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\widehat{\mathit{\beta }}}_0\right)\right\}\right\}}^T{\widehat{\mathit{K}}}_{n,0}^{-1}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{Y-\mathit{\mu }\left(\mathit{X}{\widehat{\mathit{\beta }}}_0\right)\right\}.$$

Since $\mathrm{Pr}({\widehat{S}}_0=S)\to 1$, we obtain $\widehat{\varphi }\left|T_S-T_{S,0}\right|=o_p(r)$, where

$$T_{S,0}=\frac{1}{n\widehat{\varphi }}{\left\{\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\widehat{\mathit{\beta }}}_0\right)\right\}\right\}}^T{\mathit{K}}_n^{-1}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{Y-\mu \left(\mathit{X}{\widehat{\mathit{\beta }}}_0\right)\right\}.$$

This together with (5.16) implies that |T_S − T_S,0| = o_p(r). Using similar arguments in (5.17) and (5.18), we can show that T_S,0/r is equivalent to T_S,1/r, where T_S,1 is defined as

$$\frac{1}{{\varphi }_0}{\Vert \frac{1}{\sqrt{n}}{\mathit{P}}_n{\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\}+\sqrt{n}{\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right){\mathbf{\Psi }}^{-1}{\mathit{h}}_n\Vert }_2^2.$$

Recall that

$${\mathit{P}}_n={\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right){\mathbf{\Psi }}^{-1}{\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right)}^T{\mathit{K}}_n^{-1/2},$$

we have

$$\begin{gathered} T_{S,1}=\frac{1}{{\varphi }_0}{\Vert {\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right){\mathbf{\Psi }}^{-1}{\mathit{\omega }}_n+\sqrt{n}{\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right){\mathbf{\Psi }}^{-1}{\mathit{h}}_n\Vert }_2^2 \\ =\frac{1}{{\varphi }_0}{\Vert {\mathbf{\Psi }}^{-1/2}{\mathit{\omega }}_n+\sqrt{n}{\mathbf{\Psi }}^{-1/2}{\mathit{h}}_n\Vert }_2^2=T_0. \end{gathered}$$

This proves the equivalence between T_S/r and T₀/r.

Step 3: By Theorem 2.1, we have

$$\begin{gathered} \sqrt{n}\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{a,S}-{\widehat{\mathit{\beta }}}_{0,S}\end{array}\right)=\frac{1}{\sqrt{n}}{\mathit{K}}_n^{-1/2}{\mathit{P}}_n{\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\} \\ +\sqrt{n}{\mathit{K}}_n^{-1/2}{\mathit{P}}_n{\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{C}}^T{\left(\mathit{C}{\mathit{C}}^T\right)}^{-1}{\mathit{h}}_n\\ \mathbf{0}\end{array}\right)+o_p(1). \end{gathered}$$

Notice that

$$\begin{gathered} {\mathit{K}}_n^{-1/2}{\mathit{P}}_n{\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{C}}^T{\left(\mathit{C}{\mathit{C}}^T\right)}^{-1}{\mathit{h}}_n\\ \mathbf{0}\end{array}\right) \\ ={\mathit{K}}_n^{-1}\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right){\mathbf{\Psi }}^{-1}{\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right)}^T\left(\begin{array}{c}{\mathit{C}}^T{\left(\mathit{C}{\mathit{C}}^T\right)}^{-1}{\mathit{h}}_n\\ 0\end{array}\right)={\mathit{K}}_n^{-1}\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right){\mathbf{\Psi }}^{-1}{\mathit{h}}_n. \end{gathered}$$

It follows that

$$\begin{gathered} \sqrt{n}\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{a,S}-{\widehat{\mathit{\beta }}}_{0,S}\end{array}\right)=\frac{1}{\sqrt{n}}{\mathit{K}}_n^{-1/2}{\mathit{P}}_n{\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\} \\ +\sqrt{n}{\mathit{K}}_n^{-1}\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right){\mathbf{\Psi }}^{-1}{\mathit{h}}_n+o_p(1). \end{gathered}$$ (5.24)

Similar to (5.23), we can show that

$$n{\Vert {\widehat{\mathit{\beta }}}_{a,\mathcal{M}\cup S}-{\widehat{\mathit{\beta }}}_{0,\mathcal{M}\cup \mathcal{S}}\Vert }_2^2=O_p(r).$$ (5.25)

Under the event ${\widehat{\mathit{\beta }}}_{0,\mathcal{M}\cup S}={\widehat{\mathit{\beta }}}_{a,\mathcal{M}\cup \mathcal{S}}=0$, using third-order Taylor expansion, we obtain that

$$\begin{gathered} L_n\left({\widehat{\mathit{\beta }}}_0\right)-L_n\left({\widehat{\mathit{\beta }}}_a\right)=\frac{1}{n}{\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}-{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{0,S}-{\widehat{\mathit{\beta }}}_{a,S}\end{array}\right)}^T\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\widehat{\mathit{\beta }}}_a\right)\right\} \\ -\frac{1}{2n}{\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}-{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{0,S}-{\widehat{\mathit{\beta }}}_{a,S}\end{array}\right)}^T\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_a\right){\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)}^T\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}-{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{0,S}-{\widehat{\mathit{\beta }}}_{a,S}\end{array}\right) \\ +{\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}-{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{0,S}-{\widehat{\mathit{\beta }}}_{a,S}\end{array}\right)}^T\mathit{R}, \end{gathered}$$

where n∥R∥_∞ is upper bounded by

$$\begin{gathered} \underset{{}_{j\in \mathcal{M}\cup S}}{\max }\left|{({\widehat{\mathit{\beta }}}_{0,\mathcal{M}\cup S}-{\widehat{\mathit{\beta }}}_{a,\mathcal{M}\cup S})}^T{\mathit{X}}_{\mathcal{M}\cup S}^T\mathrm{diag}\{\left|{\mathit{X}}^j\right|\circ \left|b^{\prime \prime \prime }(\mathit{X}{\mathit{\beta }}^{*})\right|\}{\mathit{X}}_{\mathcal{M}\cup \mathcal{S}}({\widehat{\mathit{\beta }}}_{0,\mathcal{M}\cup S}-{\widehat{\mathit{\beta }}}_{a,\mathcal{M}\cup S})\right| \\ \le {\Vert {\widehat{\mathit{\beta }}}_{0,\mathcal{M}\cup S}-{\widehat{\mathit{\beta }}}_{a,\mathcal{M}\mathcal{U}\mathcal{S}}\Vert }_2^2\underset{{}_{j\in \mathcal{M}\cup S}}{\max }{\lambda }_{\max }\left({\mathit{X}}_{\mathcal{M}\cup S}^T\mathrm{diag}\left\{\left|{\mathit{X}}^j\right|\circ \left|b^{\prime \prime \prime }\left(\mathit{X}{\mathit{\beta }}^{*}\right)\right|\right\}{\mathit{X}}_{\mathcal{M}\cup \mathcal{S}}\right), \end{gathered}$$

for some β* lying on the line segment between ${\widehat{\mathit{\beta }}}_a$ and ${\widehat{\mathit{\beta }}}_0$. By Theorem 2.1, we have ${\mathit{\beta }}_{{(\mathcal{M}\cup \mathcal{S})}^c}^{*}=0$ and ${\Vert {\mathit{\beta }}_{\mathcal{M}\cup \mathcal{S}}^{*}-{\mathit{\beta }}_{0,\mathcal{M}\cup \mathcal{S}}\Vert }_2\le \sqrt{(s+m)\log n/n}$ with probability tending to 1. By Condition (A1), we obtain

$$\Vert \mathit{R}{\Vert }_{\infty }=O_p\left(\frac{r}{n}\right).$$

This together with (5.25) yields that

$${\Vert {\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{a,S}-{\widehat{\mathit{\beta }}}_{0,S}\end{array}\right)}^T\mathit{R}\Vert }_2\le {\Vert {\widehat{\mathit{\beta }}}_{a,\mathcal{M}\cup \mathcal{S}}-{\widehat{\mathit{\beta }}}_{0,\mathcal{M}\cup S}\Vert }_1\Vert \mathit{R}{\Vert }_{\infty }=o_p\left(\frac{r}{n}\frac{\sqrt{r}}{\sqrt{n}}\sqrt{s+m}\right).$$

The last term is $o_p(\sqrt{r}/n)$ since r ≤ s + m and s + m = o(n^1/3).

Similarly, we can show

$$\begin{gathered} \Vert {\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{a,S}-{\widehat{\mathit{\beta }}}_{0,S}\end{array}\right)}^T\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_a\right){\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)}^T\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{a,S}-{\widehat{\mathit{\beta }}}_{0,S}\end{array}\right)- \\ {{\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{a,S}-{\widehat{\mathit{\beta }}}_{0,S}\end{array}\right)}^T\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\mathbf{\Sigma }\left(\mathit{X}{\mathit{\beta }}_0\right){\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)}^T\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}-{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{a,S}-{\widehat{\mathit{\beta }}}_{0,S}\end{array}\right)\Vert }_2=o_p(\sqrt{r}). \end{gathered}$$

As a result, we have

$$\begin{gathered} n\left\{L_n\left({\widehat{\mathit{\beta }}}_0\right)-L_n\left({\widehat{\mathit{\beta }}}_a\right)\right\}={\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}-{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{0,S}-{\widehat{\mathit{\beta }}}_{a,S}\end{array}\right)}^T\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\widehat{\mathit{\beta }}}_a\right)\right\} \\ -\frac{n}{2}{\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}-{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{0,S}-{\widehat{\mathit{\beta }}}_{a,S}\end{array}\right)}^T{\mathit{K}}_n{\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}-{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{0,S}-{\widehat{\mathit{\beta }}}_{a,S}\end{array}\right)}^T+o_p(\sqrt{r}). \end{gathered}$$ (5.26)

Recall that ${\widehat{\mathit{\beta }}}_a$ is the maximizer of $n{L}_n(\mathit{\beta })-n{\sum }_{j\notin \mathcal{M}}{p}_{{\lambda }_{n,a}}(|{\mathit{\beta }}_j|)$ Theorem 2.1, we have with probability tending to 1 that ${\min }_{j\in S}\left|{\widehat{\mathit{\beta }}}_{a,j}\right|\ge d_n$. Under the condition $p_{{\lambda }_{n,a}}^{\prime }\left(d_n\right)=o\left({(s+m)}^{-1/2}{n}^{-1/2}\right)$ we have

$$\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\widehat{\mathit{\beta }}}_a\right)\right\}=n\left(\begin{array}{c}\mathbf{0}\\ \overline{\mathit{\rho }}({\widehat{\mathit{\beta }}}_{a,S},{\lambda }_{n,a})\end{array}\right)=o_p\left(n^{1/2}\right).$$

This together with (5.25) yields

$${\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}-{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{0,S}-{\widehat{\mathit{\beta }}}_{a,S}\end{array}\right)}^T\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\widehat{\mathit{\beta }}}_a\right)\right\}=o_p(\sqrt{r}).$$

By (5.26), we obtain that

$$n\left\{L_n\left({\widehat{\mathit{\beta }}}_0\right)-L_n\left({\widehat{\mathit{\beta }}}_a\right)\right\}=-\frac{n}{2}{\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}-{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{0,S}-{\widehat{\mathit{\beta }}}_{a,S}\end{array}\right)}^T{\mathit{K}}_n\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}-{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{0,S}-{\widehat{\mathit{\beta }}}_{a,S}\end{array}\right)+o_p(\sqrt{r}).$$

In view of (5.24), using similar arguments in (5.17), we can show that

$$\begin{gathered} |n{\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}-{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{0,S}-{\widehat{\mathit{\beta }}}_{a,S}\end{array}\right)}^T{\mathit{K}}_n\left(\begin{array}{c}{\widehat{\mathit{\beta }}}_{0,\mathcal{M}}-{\widehat{\mathit{\beta }}}_{a,\mathcal{M}}\\ {\widehat{\mathit{\beta }}}_{0,S}-{\widehat{\mathit{\beta }}}_{a,S}\end{array}\right) \\ -\frac{1}{n}{\Vert P_n{\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{Y-\mu \left(\mathit{X}{\mathit{\beta }}_0\right)\right\}+n{\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right){\mathbf{\Psi }}^{-1}{\mathit{h}}_n\Vert }_2^2|=o_p(\sqrt{r}). \end{gathered}$$

As a result, we have

$$\begin{gathered} \frac{1}{n}{\Vert {\mathit{P}}_n{\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\}+n{\mathit{K}}_n^{-1/2}\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right){\mathbf{\Psi }}^{-1}{\mathit{h}}_n\Vert }_2^2 \\ -2n\left\{L_n\left({\widehat{\mathit{\beta }}}_a\right)-L_n\left({\widehat{\mathit{\beta }}}_0\right)\right\}=o_p(\sqrt{r}). \end{gathered}$$

By (5.16), this shows

$$\left|T_L-\frac{{\varphi }_0}{\widehat{\varphi }}{T}_0\right|=o_p(r).$$

Under the condition $|\widehat{\varphi }-{\varphi }_0|=o_p(1)$, we can show $|T_0(1-{\varphi }_0/\widehat{\varphi })|=o_p(r)$. As a result, we have T_L = T₀ + o_p(r).

Step 4: We first show the χ² approximation (3.5) holds for T = T₀. Recall that

$$T_0=\frac{1}{{\varphi }_0}{\Vert \frac{1}{\sqrt{n}}{\mathbf{\Psi }}^{-1/2}{\mathit{\omega }}_n+\sqrt{n}{\mathbf{\Psi }}^{-1/2}{\mathit{h}}_n\Vert }_2^2.$$

By the definition of ω_n, we have

$$\begin{gathered} \frac{1}{\sqrt{n{\varphi }_0}}{\mathbf{\Psi }}^{-1/2}{\mathit{\omega }}_n=\frac{1}{\sqrt{n{\varphi }_0}}{\mathbf{\Psi }}^{-1/2}{\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right)}^T{\mathit{K}}_n^{-1}\left(\begin{array}{c}{\mathit{X}}_{\mathcal{M}}^T\\ {\mathit{X}}_S^T\end{array}\right)\left\{\mathit{Y}-\mathit{\mu }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\} \\ =\sum _{i=1}^n\frac{1}{\sqrt{n{\varphi }_0}}{\mathbf{\Psi }}^{-1/2}{\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right)}^T{\mathit{K}}_n^{-1}\left\{{\mathit{Y}}_i-\mathit{\mu }\left({\mathit{\beta }}_0^T{\mathit{X}}_i\right)\right\}\left(\begin{array}{c}{\mathit{X}}_{i,\mathcal{M}}\\ {\mathit{X}}_{i,S}\end{array}\right)=\sum _{i=1}^n{\mathit{\xi }}_i. \end{gathered}$$

With some calculation, we can show that

$$\sum _i^n\mathrm{cov}\left({\mathit{\xi }}_i\right)={\mathit{I}}_r.$$ (5.27)

It follows from Condition (A3) that

$$\begin{gathered} \underset{{}_{i=1,\dots ,n}}{\max }\text{E}\left(\frac{{\left|Y_i-\mu \left({\mathit{\beta }}_0^T{\mathit{X}}_i\right)\right|}^3}{6M^3}{M}^2\right) \\ \le \underset{{}_{i=1,\dots ,n}}{\max }\text{E}\left\{\exp \left(\frac{\left|Y_i-\mu \left({\mathit{\beta }}_0^T{\mathit{X}}_i\right)\right|}{M}\right)-1-\frac{\left|Y_i-\mu \left({\mathit{\beta }}_0^T{\mathit{X}}_i\right)\right|}{M}\right\}{M}^2\le \frac{v_0}{2}. \end{gathered}$$

This implies max_{i = 1,...,n} $\text{E}{\left|Y_i-\mu \left({\mathit{\beta }}_0^T{\mathit{X}}_i\right)\right|}^3=O(1).$

Hence, with some calculations, we have

$$\begin{gathered} r^{1/4}\sum _i^n\text{E}{\Vert {\mathit{\xi }}_i\Vert }_2^3 \\ =\frac{r^{1/4}}{{\left(n{\varphi }_0\right)}^{3/2}}\sum _i^n\text{E}{\Vert {\mathbf{\Psi }}^{-1/2}{\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right)}^T{\mathit{K}}_n^{-1}{\mathit{X}}_{i,\mathcal{M}\cup S}\left\{Y_i-\mu \left({\mathit{\beta }}_0^T{\mathit{X}}_i\right)\right\}\Vert }_2^3 \\ =\frac{r^{1/4}}{{\left(n{\varphi }_0\right)}^{3/2}}\sum _i^n{\Vert {\mathbf{\Psi }}^{-1/2}{\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right)}^T{\mathit{K}}_n^{-1}{\mathit{X}}_{i,\mathcal{M}\cup S}\Vert }_2^3\mathrm{E}{\left|Y_i-\mu \left({\mathit{\beta }}_0^T{\mathit{X}}_i\right)\right|}^3 \\ =O(1)\frac{r^{1/4}}{{\left(n{\varphi }_0\right)}^{3/2}}\sum _i^n{\Vert {\mathbf{\Psi }}^{-1/2}{\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right)}^T{\mathit{K}}_n^{-1}{\mathit{X}}_{i,\mathcal{M}\cup S}\Vert }_2^3 \\ \le O(1)\frac{r^{1/4}}{{\left(n{\varphi }_0\right)}^{3/2}}\sum _i^n{\Vert {\mathbf{\Psi }}^{-1/2}{\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right)}^T{\mathit{K}}_n^{-1/2}\Vert }_2^3{\Vert {\mathit{K}}_n^{-1/2}{\mathit{X}}_{i,\mathcal{M}\cup S}\Vert }_2^3 \\ \le O(1)\frac{r^{1/4}}{{\left(n{\varphi }_0\right)}^{3/2}}\sum _i^n{\left({\mathit{X}}_{i,\mathcal{M}\cup \mathcal{S}}\right)}^T{\mathit{K}}_n^{-1}{\mathit{X}}_{i,\mathcal{M}\cup \mathcal{S}}{\}}^{3/2}=o(1), \end{gathered}$$

where O(1) denotes some positive constant, the first inequality follows from Cauchy-Schwarz inequality, the last inequality follows from the fact that

$$\Vert {\mathbf{\Psi }}^{-1/2}{\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right)}^T{\mathit{K}}_n^{-1/2}\Vert ={\lambda }_{\max }\left\{{\mathbf{\Psi }}^{-1/2}{\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right)}^T{\mathit{K}}_n^{-1}\left(\begin{array}{c}{\mathit{C}}^T\\ {\mathit{O}}_{r\times s}^T\end{array}\right){\mathbf{\Psi }}^{-1/2}\right\}=1,$$

and the last equality is due to Condition (3.4).

This together with (5.27) and an application of Lemma S.3 in the supplementary material gives that

$$\underset{\mathcal{C}}{\sup }\left|\mathrm{Pr}\left(\frac{1}{\sqrt{n{\varphi }_0}}{\mathbf{\Psi }}^{-1/2}{\mathit{\omega }}_0\in \mathcal{C}\right)-\mathrm{Pr}(\mathit{Z}\in \mathcal{C})\right|\to 0,$$ (5.28)

where Z ∈ ℝ^r stands for a mean zero Gaussian random vector with identity covariance matrix, and the supremum is taken over all convex sets $\mathcal{C}$ ∈ ℝ^r.

Consider the following class of sets:

$${\mathcal{C}}_x=\left\{\mathit{z}\in {ℝ}^r:{\Vert \mathit{z}-\sqrt{\frac{n}{{\varphi }_0}}{\mathbf{\Psi }}^{-1/2}{\mathit{h}}_n\Vert }_2\le x\right\},$$

indexed by x ∈ ℝ. It follows from (5.28) that

$$\underset{x}{\sup }\left|\mathrm{Pr}\left(\frac{1}{\sqrt{n{\varphi }_0}}{\mathbf{\Psi }}^{-1/2}{\mathit{\omega }}_0\in {\mathcal{C}}_x\right)-\mathrm{Pr}\left(\mathit{Z}\in {\mathcal{C}}_x\right)\right|\to 0.$$

Note that $\frac{1}{\sqrt{n{\varphi }_0}}{\mathbf{\Psi }}^{-1}{\mathit{\omega }}_0\in {\mathcal{C}}_x$ is equivalent to T₀ ≤ x, and Pr(Z ∈ ${\mathcal{C}}_x$) = Pr(χ²(r,γ_n) ≤ x) where ${\gamma }_n=n{\mathit{h}}_n^T{\mathbf{\Psi }}^{-1/2}{\mathit{h}}_n/{\varphi }_0$. This implies

$$\underset{x}{\sup }\left|\mathrm{Pr}\left(T_0\le x\right)-\mathrm{Pr}\left({\chi }^2\left(r,{\gamma }_n\right)\le x\right)\right|\to 0.$$ (5.29)

Consider any statistic T^* = T₀ + o_p(r). For any x and ε > 0, it follows from (5.29) that

$$\begin{gathered} \mathrm{Pr}\left({\chi }^2\left(r,{\gamma }_n\right)\le x-r\epsilon \right)+o(1)\le \mathrm{Pr}\left(T_0\le x-r\epsilon \right)+o(1) \\ \le \mathrm{Pr}\left(T^{*}\le x\right)\le \mathrm{Pr}\left(T_0\le x+r\epsilon \right)+o(1)\le \mathrm{Pr}\left({\chi }^2\left(r,{\gamma }_n\right)\le x+r\epsilon \right)+o(1). \end{gathered}$$ (5.30)

Besides, by Lemma S.4, we have

$$\underset{\epsilon \to 0}{\lim }\underset{n}{\lim \sup }\left|\mathrm{Pr}\left({\chi }^2\left(r,{\gamma }_n\right)\le x+r\epsilon \right)-\mathrm{Pr}\left({\chi }^2\left(r,{\gamma }_n\right)\le x-r\epsilon \right)\right|\to 0.$$ (5.31)

Combining (5.30) with (5.31), we obtain that

$$\underset{x}{\sup }\left|\mathrm{Pr}\left(T^{*}\le x\right)-\mathrm{Pr}\left({\chi }^2\left(r,{\gamma }_n\right)\le x\right)\right|\to 0.$$ (5.32)

In the first three steps, we have shown T₀ = T_S + o_p(1) = T_W + o_p(1) = T_L + o_p(1). This together with (5.32) implies that the χ² approximation holds for our partial penalized Wald, score and likelihood ratio statistics. The proof is hence completed.

5.1. Proof of Lemma 5.1.

Assertion (5.1) is directly implies by Condition (A1). This means the square root of the maximum eigenvalue of K_n is O(1). By definition, this proves (5.2). Under Condition (A1), we have ${\lambda }_{\max }({\mathit{K}}_n^{-1})=O(1)$. Using the same arguments, we have ${\lambda }_{\max }({\mathit{K}}_n^{-1/2})=O(1)$. Hence, (5.3) is proven. We now show (5.4) holds. It follows from the condition λ_max ((CC^T )⎺¹) = O(1) in Condition (A4) that liminf_n λ_min(CC^T )⎺¹ > 0, and hence

$$a_0\triangleq \underset{n}{\lim \inf }\underset{\mathit{a}\in {ℝ}^r:\Vert \mathit{a}{\Vert }_2=1}{\inf }{\Vert {\mathit{C}}^T\mathit{a}\Vert }_2^2=\underset{n}{\lim \inf }\underset{a\in {ℝ}^r:\Vert a{\Vert }_2=1}{\inf }{\mathit{a}}^T\mathit{C}{\mathit{C}}^T\mathit{a}>0.$$

This implies that for sufficiently large n, we have

$${\Vert {\mathit{C}}^T\mathit{a}\Vert }_2>\sqrt{a_0/2}\Vert \mathit{a}{\Vert }_2,\forall \mathit{a}\ne 0.$$ (5.33)

By (5.1), we have liminf_n λ_min(Ω_n) > 0, or equivalently,

$$\underset{\mathit{a}\in {ℝ}^{m+s}:\Vert \mathit{a}{\Vert }_2=1}{\inf }\underset{n}{\lim \inf }{\mathit{a}}^T{\mathbf{\Omega }}_n\mathit{a}>0.$$

Hence, we have

$$\underset{\mathit{a}\in {ℝ}^{m+s}:\Vert \mathit{a}{\Vert }_2=1,{\mathit{a}}_{J_0}^c=0}{\inf }\underset{n}{\lim \inf }{\mathit{a}}^T{\mathbf{\Omega }}_n\mathit{a}>0,$$

where J₀ = [1,...,m]. Note that this implies

$$\underset{\mathit{a}\in {ℝ}^{m+s}:\Vert \mathit{a}{\Vert }_2=1}{\inf }\underset{n}{\lim \inf }{\mathit{a}}^T{\mathbf{\Omega }}_n\mathit{a}>0.$$

Therefore, we obtain

$$\underset{n}{\lim \inf }{\lambda }_{\min }\left({\mathbf{\Omega }}_{mm}\right)>0.$$ (5.34)

Combining this together with (5.33) yields

$$\underset{\mathit{a}\in {ℝ}^r:\Vert \mathit{a}{\Vert }_2=1}{\inf }\underset{n}{\lim \inf }{\mathit{a}}^T\mathit{C}{\mathbf{\Omega }}_{mm}{\mathit{C}}^T\mathit{a}\ge \underset{\mathit{a}\in {ℝ}^m:\Vert \mathit{a}{\Vert }_2=\sqrt{{\mathit{a}}_0/2}}{\inf }\underset{n}{\lim \inf }{\mathit{a}}^T{\mathbf{\Omega }}_{mm}\mathit{a}>0.$$

By definition, this suggests

$$\underset{n}{\lim \inf }{\lambda }_{\min }\left(\mathit{C}{\mathbf{\Omega }}_{mm}{\mathit{C}}^T\right)>0,$$

or equivalently,

$${\lambda }_{\max }\left({\left(\mathit{C}{\mathbf{\Omega }}_{mm}{\mathit{C}}^T\right)}^{-1}\right)=O(1).$$

This gives (5.4).

Using Cauchy-Schwarz inequality, we have

$${\Vert {\left(\mathit{C}{\mathbf{\Omega }}_{mm}{\mathit{C}}^T\right)}^{-1/2}\mathit{C}\Vert }_2\le \underset{I_2}{\underbrace{{\Vert {\left(\mathit{C}{\mathbf{\Omega }}_{mm}{\mathit{C}}^T\right)}^{-1/2}\mathit{C}{\mathbf{\Omega }}_{mm}^{1/2}\Vert }_2}}\underset{I_2}{\underbrace{{\Vert {\mathbf{\Omega }}_{mm}^{-1/2}\Vert }_2.}}$$

Observe that

$$I_1^2={\lambda }_{\max }\left({\left(\mathit{C}{\mathbf{\Omega }}_{mm}{\mathit{C}}^T\right)}^{-1/2}\mathit{C}{\mathbf{\Omega }}_{mm}{\mathit{C}}^T{\left(\mathit{C}{\mathbf{\Omega }}_{mm}{\mathit{C}}^T\right)}^{-1/2}\right)=1.$$ (5.35)

Besides, by (5.34), we have

$$I_2^2={\lambda }_{\max }\left({\left({\mathbf{\Omega }}_{mm}\right)}^{-1}\right)=O(1),$$

which together with (5.35) implies that I₁I₂ = O(1). This proves (5.5).

We now show (5.6) holds. Assume for now, we have

$${\Vert {\mathit{K}}_n-{\widehat{\mathit{K}}}_{n,\mathit{a}}\Vert }_2=O_p\left(\frac{s+m}{\sqrt{n}}\right),$$ (5.36)

where

$${\widehat{\mathit{K}}}_{n,a}=\frac{1}{n}\left(\begin{array}{cc}{\mathit{X}}_{\mathcal{M}}^T\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_a\right){\mathit{X}}_{\mathcal{M}}& {\mathit{X}}_{\mathcal{M}}^T\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_a\right){\mathit{X}}_S\\ {\mathit{X}}_S^T\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_a\right){\mathit{X}}_{\mathcal{M}}& {\mathit{X}}_S^T\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_a\right){\mathit{X}}_S\end{array}\right).$$

Note that

$$\begin{gathered} \underset{n}{\lim \inf }{\lambda }_{\min }\left({\widehat{\mathit{K}}}_{n,a}\right)\ge \underset{n}{\lim \inf }\underset{\Vert \mathit{a}{\Vert }_2=1}{\underset{\mathit{a}\in {ℝ}^{m+s}}{\inf }}{\mathit{a}}^T{\mathit{K}}_n\mathit{a}-\underset{n}{\lim \sup }\underset{\Vert \mathit{a}{\Vert }_2=1}{\underset{\mathit{a}\in {ℝ}^{m+s}}{\sup }}\left|{\mathit{a}}^T\left({\widehat{\mathit{K}}}_{n,\mathit{a}}-{\mathit{K}}_n\right)\mathit{a}\right| \\ \ge \underset{n}{\lim \inf }{\lambda }_{\min }\left({\mathit{K}}_{n,a}\right)-\underset{n}{\lim \sup }{\Vert {\mathit{K}}_n-{\widehat{\mathit{K}}}_{n,\mathit{a}}\Vert }_2. \end{gathered}$$

Under Condition (A1), we have lim inf_n λ_min(K_n) > 0. Under the condition max(s,m) = o(n^1/2), this together with (5.36) implies

$$\underset{n}{\lim \inf }{\lambda }_{\min }\left({\widehat{\mathit{K}}}_{n,a}\right)>0,$$ (5.37)

with probability tending to 1. Hence, we have

$$\begin{gathered} {\Vert {\mathit{K}}_n^{-1}-{\widehat{\mathit{K}}}_{n,a}^{-1}\Vert }_2={\Vert {\mathit{K}}_n^{-1}({\mathit{K}}_n-{\widehat{\mathit{K}}}_{n,a}){\widehat{\mathit{K}}}_{n,a}^{-1}\Vert }_2 \\ \le {\lambda }_{\max }({\mathit{K}}_n^{-1}){\Vert {\mathit{K}}_n-{\widehat{\mathit{K}}}_{n,a}\Vert }_2{\lambda }_{\max }({\widehat{\mathit{K}}}_n^{-1})=O_p\left(\frac{s+m}{\sqrt{n}}\right). \end{gathered}$$ (5.38)

By Lemma S.2, this gives

$$\underset{\mathit{a}\in {ℝ}^{m+s}:\Vert \mathit{a}{\Vert }_2=1}{\sup }\left|{\mathit{a}}^T\left({\mathit{K}}_n^{-1}-{\widehat{\mathit{K}}}_{n,\mathit{a}}^{-1}\right)\mathit{a}\right|=O_p\left(\frac{s+m}{\sqrt{n}}\right),$$

and hence,

$$\underset{\mathit{a}\in {ℝ}^{m+s}:\Vert \mathit{a}{\Vert }_2=1,{\mathit{a}}_{J_0}^c=0}{\sup }\left|{\mathit{a}}^T\left({\mathit{K}}_n^{-1}-{\widehat{\mathit{K}}}_{n,\mathit{a}}^{-1}\right)\mathit{a}\right|=O_p\left(\frac{s+m}{\sqrt{n}}\right),$$

where J₀ = [1,...,m]. Using Lemma S.2 again, we obtain

$${\Vert {({\mathit{K}}_n^{-1})}_{J_0,J_0}-{\left({\widehat{\mathit{K}}}_{n,\mathit{a}}^{-1}\right)}_{J_0,J_0}\Vert }_2=O_p\left(\frac{s+m}{\sqrt{n}}\right).$$ (5.39)

By definition, we have ${\mathbf{\Omega }}_{mm}={\left({\mathit{K}}_n^{-1}\right)}_{J_0,J_0}$. According to Theorem 2.1, we have that with probability tending to 1, ${\widehat{S}}_a=S$ where ${\widehat{S}}_a=\left\{j\in {\mathcal{M}}^c:{\widehat{\beta }}_{a,j}\ne 0\right\}$. When ${\widehat{S}}_a=S$, we have ${\widehat{\mathit{K}}}_{n,a}^{-1}={\widehat{\mathbf{\Omega }}}_a$ and ${\widehat{\mathit{K}}}_{n,a}^{-1}{)}_{J_0,J_0}={\widehat{\mathbf{\Omega }}}_{a,mm}$. Therefore, by (5.39), we have

$${\Vert {\mathbf{\Omega }}_{mm}-{\widehat{\mathbf{\Omega }}}_{a,mm}\Vert }_2=O_p\left(\frac{s+m}{\sqrt{n}}\right).$$

Using Cauchy-Schwarz inequality, we obtain

$$\begin{gathered} {\Vert {\mathbf{\Omega }}_{mm}^{-1/2}\left({\mathbf{\Omega }}_{mm}-{\widehat{\mathbf{\Omega }}}_{a,mm}\right){\mathbf{\Omega }}_{mm}^{-1/2}\Vert }_2 \\ \le {\Vert {\mathbf{\Omega }}_{mm}^{-1/2}\Vert }_2^2{\Vert {\mathbf{\Omega }}_{mm}-{\widehat{\mathbf{\Omega }}}_{a,mm}\Vert }_2=O_p\left(\frac{s+m}{\sqrt{n}}\right), \end{gathered}$$ (5.40)

by (5.34). Let Ψ = CΩ_mmC^T, we obtain

$$\begin{gathered} {\Vert {\mathbf{\Psi }}^{-1/2}\mathit{C}\left({\mathbf{\Omega }}_{mm}-{\widehat{\mathbf{\Omega }}}_{a,mm}\right){\mathit{C}}^T{\mathbf{\Psi }}^{-1/2}\Vert }_2 \\ \le {\Vert {\mathbf{\Psi }}^{-1/2}\mathit{C}{\mathbf{\Omega }}_{mm}^{1/2}{\mathbf{\Omega }}_{mm}^{-1/2}\left({\mathbf{\Omega }}_{mm}-{\widehat{\mathbf{\Omega }}}_{a,mm}\right){\mathbf{\Omega }}_{mm}^{-1/2}{\mathbf{\Omega }}_{mm}^{1/2}{\mathit{C}}^T{\mathbf{\Psi }}^{-1/2}\Vert }_2 \\ \le {\Vert {\mathbf{\Psi }}^{-1/2}\mathit{C}{\mathbf{\Omega }}_{mm}^{1/2}\Vert }_2^2\Vert {\mathbf{\Omega }}_{mm}^{-1/2}\left({\mathbf{\Omega }}_{mm}-{\widehat{\mathbf{\Omega }}}_{a,mm}\right){\mathbf{\Omega }}_{mm}^{-1/2}\Vert 2=O_p\left(\frac{s+m}{\sqrt{n}}\right), \end{gathered}$$ (5.41)

by(5.40) and that

$${\Vert {\mathbf{\Psi }}^{-1/2}\mathit{C}{\mathbf{\Omega }}_{mm}^{1/2}\Vert }_2^2={\lambda }_{\max }\left({\mathbf{\Psi }}^{-1/2}\mathbf{\Psi }{\mathbf{\Psi }}^{-1/2}\right)=O(1).$$

Similar to (5.37), by (5.41), we can show that

$$\underset{n}{\lim \inf }{\lambda }_{\min }\left({\mathbf{\Psi }}^{-1/2}\mathit{C}{\widehat{\mathbf{\Omega }}}_{a,mm}{\mathit{C}}^T{\mathbf{\Psi }}^{-1/2}\right)>0.$$ (5.42)

Combining (5.41) together with (5.42), we obtain

$$\begin{gathered} {\Vert {\left({\mathbf{\Psi }}^{-1/2}\mathit{C}{\widehat{\mathbf{\Omega }}}_{a,mm}{\mathit{C}}^T{\mathbf{\Psi }}^{-1/2}\right)}^{-1}-{\mathit{I}}_m\Vert }_2 \\ \le {\Vert {\left({\mathbf{\Psi }}^{-1/2}\mathit{C}{\widehat{\mathbf{\Omega }}}_{a,mm}{\mathit{C}}^T{\mathbf{\Psi }}^{-1/2}\right)}^{-1}\Vert }_2{\Vert {\mathbf{\Psi }}^{-1/2}\mathit{C}\left({\mathbf{\Omega }}_{mm}-{\widehat{\mathbf{\Omega }}}_{a,mm}\right){\mathit{C}}^T{\mathbf{\Psi }}^{-1/2}\Vert }_2 \\ =O_p\left(\frac{s+m}{\sqrt{n}}\right). \end{gathered}$$

This proves (5.6).

Similar to (5.38), we can show

$${\Vert {\mathit{K}}_n^{-1}-{\widehat{\mathit{K}}}_{n,0}^{-1}\Vert }_2=O_p\left(\frac{s+m}{\sqrt{n}}\right).$$

By (5.2), we obtain

$${\Vert \mathit{I}-{\mathit{K}}_n^{1/2}{\widehat{\mathit{K}}}_{n,0}^{-1}{\mathit{K}}_n^{1/2}\Vert }_2\le {\Vert {\mathit{K}}_n^{1/2}\Vert }_2{\Vert {\mathit{K}}_n^{-1}-{\widehat{\mathit{K}}}_{n,0}^{-1}\Vert }_2{\Vert {\mathit{K}}_n^{1/2}\Vert }_2=O_p\left(\frac{s+m}{\sqrt{n}}\right).$$

This proves (5.7).

It remains to show (5.36). Since K_n and ${\widehat{\mathit{K}}}_{n,a}$ are symmetric, by Lemma S.5, it suffices to show

$${\Vert {\mathit{K}}_n-{\widehat{\mathit{K}}}_{n,a}\Vert }_{\infty }=O_p\left(\frac{s+m}{\sqrt{n}}\right).$$

By definition, this requires to show

$$\underset{j\in \mathcal{S}\cup \mathcal{M}}{\max }{\Vert {\left({\mathit{X}}^j\right)}^T\left\{\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_a\right)-\Sigma \left(\mathit{X}{\mathit{\beta }}_0\right)\right\}{\mathit{X}}_{\mathcal{M}\cup S}\Vert }_1=O_p(\sqrt{n}(s+m)),$$

For any vector a ∈ ℝ^q, we have $\Vert a{\Vert }_1\le \sqrt{q}\Vert a{\Vert }_2$. Hence, it suffices to show

$$\underset{j\in S\cup \mathcal{M}}{\max }{\Vert {\left({\mathit{X}}^j\right)}^T\left\{\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_a\right)-\mathbf{\Sigma }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\}{\mathit{X}}_{\mathcal{M}\mathcal{U}\mathcal{S}}\Vert }_2=O_p\left(\sqrt{n(s+m)}\right).$$ (5.43)

Using Taylor’s theorem, we have

$$\begin{gathered} {\left({\mathit{X}}^j\right)}^T\left\{\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_a\right)-\mathbf{\Sigma }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\}{\mathit{X}}_{\mathcal{M}\cup \mathcal{S}} \\ \le {\int }_0^1{\left({\widehat{\mathit{\beta }}}_a-{\mathit{\beta }}_0\right)}^T\mathit{X}\mathrm{diag}\left\{{\mathit{X}}^j\circ b^{\prime \prime \prime }\left(\mathit{X}\left\{t{\widehat{\mathit{\beta }}}_a+(1-t){\mathit{\beta }}_0\right\}\right)\right\}{\mathit{X}}_{\mathcal{M}\cup \mathcal{S}}dt. \end{gathered}$$ (5.44)

By Theorem 2.1, we have $\mathrm{Pr}\left({\widehat{\mathit{\beta }}}_a\in {\mathcal{N}}_0\right)\to 1$. Hence, we have

$$\mathrm{Pr}\left(\bigcup _{t\in [0,1]}\left\{t{\widehat{\mathit{\beta }}}_a+(1-t){\mathit{\beta }}_0\in {\mathcal{N}}_0\right\}\right)\to 1.$$

By Condition (A1),

$$\underset{t\in [0,1]}{\sup }{\lambda }_{\max }\left\{{\mathit{X}}_{\mathcal{M}\cup \mathcal{S}}\mathrm{diag}\left({\mathit{X}}^j\circ b^{\prime \prime \prime }\left(\mathit{X}\left\{t{\widehat{\mathit{\beta }}}_a+(1-t){\mathit{\beta }}_0\right\}\right)\right){\mathit{X}}_{\mathcal{M}\mathcal{U}\mathcal{S}}\right\}=O_p(n).$$

By Cauchy-Schwarz inequality, we have

$$\begin{gathered} {\Vert {\left({\mathit{X}}^j\right)}^T\left\{\mathbf{\Sigma }\left(\mathit{X}{\widehat{\mathit{\beta }}}_a\right)-\mathbf{\Sigma }\left(\mathit{X}{\mathit{\beta }}_0\right)\right\}{\mathit{X}}_{\mathcal{M}\cup \mathcal{S}}\Vert }_2 \\ \le \underset{t\in [0,1]}{\sup }{\Vert {\left({\widehat{\mathit{\beta }}}_a-{\mathit{\beta }}_0\right)}^T\mathit{X}\mathrm{diag}\left\{{\mathit{X}}^j\circ b^{\prime \prime \prime }\left(\mathit{X}\left\{t{\widehat{\mathit{\beta }}}_a+(1-t){\mathit{\beta }}_0\right\}\right)\right\}{\mathit{X}}_{\mathcal{M}\cup \mathcal{S}}\Vert }_2 \\ \le {\Vert {\widehat{\mathit{\beta }}}_a-{\mathit{\beta }}_0\Vert }_2\underset{t\in [0,1]}{\sup }{\lambda }_{\max }\left\{{\mathit{X}}_{\mathcal{M}\cup \mathcal{S}}\mathrm{diag}\left({\mathit{X}}^j\circ b^{\prime \prime \prime }\left(\mathit{X}\left\{t{\widehat{\mathit{\beta }}}_a+(1-t){\mathit{\beta }}_0\right\}\right)\right){\mathit{X}}_{\mathcal{M}\cup \mathcal{S}}\right\} \\ =O_p(\sqrt{n(s+m)}). \end{gathered}$$

This proves (5.43).