Nonuniformity of P-values Can Occur Early in Diverging Dimensions

Abstract

Evaluating the joint significance of covariates is of fundamental importance in a wide range of applications. To this end, p-values are frequently employed and produced by algorithms that are powered by classical large-sample asymptotic theory. It is well known that the conventional p-values in Gaussian linear model are valid even when the dimensionality is a non-vanishing fraction of the sample size, but can break down when the design matrix becomes singular in higher dimensions or when the error distribution deviates from Gaussianity. A natural question is when the conventional p-values in generalized linear models become invalid in diverging dimensions. We establish that such a breakdown can occur early in nonlinear models. Our theoretical characterizations are confirmed by simulation studies.

1. Introduction

In many applications it is often desirable to evaluate the significance of covariates in a predictive model for some response of interest. Identifying a set of significant covariates can facilitate domain experts to further probe their causal relationships with the response. Ruling out insignificant covariates can also help reduce the fraction of false discoveries and narrow down the scope of follow-up experimental studies by scientists. These tasks certainly require an accurate measure of feature significance in finite samples. The tool of p-values has provided a powerful framework for such investigations.

As p-values are routinely produced by algorithms, practitioners should perhaps be aware that those p-values are usually based on classical large-sample asymptotic theory. For example, marginal p-values have been employed frequently in large-scale applications when the number of covariates p greatly exceeds the number of observations n. Those p-values are based on marginal regression models linking each individual covariate to the response separately. In these marginal regression models, the ratio of sample size to model dimensionality is equal to n, which results in justified p-values as sample size increases. Yet due to the correlations among the covariates, we often would like to investigate the joint significance of a covariate in a regression model conditional on all other covariates, which is the main focus of this paper. A natural question is whether conventional joint p-values continue to be valid in the regime of diverging dimensionality p.

It is well known that fitting the linear regression model with p > n using the ordinary least squares can lead to perfect fit giving rise to zero residual vector, which renders the p-values undefined. When p ≥ n and the design matrix is nonsingular, the p-values in the linear regression model are well defined and valid thanks to the exact normality of the least-squares estimator when the random error is Gaussian and the design matrix is deterministic. When the error is non-Gaussian, Huber (1973) showed that the least-squares estimator can still be asymptotically normal under the assumption of p = o(n), but is generally no longer normal when p = o(n) fails to hold, making the conventional p-values inaccurate in higher dimensions. For the asymptotic properties of M-estimators for robust regression, see, for example, Huber (1973); Portnoy (1984, 1985) for the case of diverging dimensionality p = o(n) and Karoui et al. (2013); Bean et al. (2013) for the scenario when the dimensionality p grows proportionally to sample size n.

We have seen that the conventional p-values for the least-squares estimator in linear regression model can start behaving wildly and become invalid when the dimensionality p is of the same order as sample size n and the error distribution deviates from Gaussianity. A natural question is whether similar phenomenon holds for the conventional p-values for the maximum likelihood estimator (MLE) in the setting of diverging-dimensional nonlinear models. More specifically, we aim to answer the question of whether p ~ n is still the breakdown point of the conventional p-values when we move away from the regime of linear regression model, where ~ stands for asymptotic order. To simplify the technical presentation, in this paper we adopt the generalized linear model (GLM) as a specific family of nonlinear models (McCullagh and Nelder, 1989). The GLM with a canonical link assumes that the conditional distribution of y given X belongs to the canonical exponential family, having the following density function with respect to some fixed measure

$$f_n(y;X,\mathit{\beta })\equiv \prod _{i=1}^n{f}_0\left(y_i;{\theta }_i\right)=\prod _{i=1}^n\left\{c\left(y_i\right)\text{exp}\left[\frac{y_i{\theta }_i-b\left({\theta }_i\right)}{\varphi }\right]\right\},$$ (1)

where X = (x₁, ⋯ , x_p) is an n × p design matrix with x_j = (x_1j, ⋯ , x_nj)^T, j = 1, ⋯ , p, y = (y₁, ⋯ , y_n)^T is an n-dimensional response vector, β = (β₁, ⋯ , β_p)^T is a p-dimensional regression coefficient vector, $\left\{f_0(y;\theta ):\theta \in ℝ\right\}$ is a family of distributions in the regular exponential family with dispersion parameter ϕ ∈ (0, ∞), and θ = (θ₁, ⋯ , θ_n)^T = Xβ. As is common in GLM, the function b(θ) in (1) is implicitly assumed to be twice continuously differentiable with b″ (θ) always positive. Popularly used GLMs include the linear regression model, logistic regression model, and Poisson regression model for continuous, binary, and count data of responses, respectively.

The key innovation of our paper is the formal justification that the conventional p-values in nonlinear models of GLMs can become invalid in diverging dimensions and such a breakdown can occur much earlier than in linear models, which spells out a fundamental difference between linear models and nonlinear models. To begin investigating p-values in diverging-dimensional GLMs, let us gain some insights into this problem by looking at the specific case of logistic regression. Recently, Candès (2016) established an interesting phase transition phenomenon of perfect hyperplane separation for high-dimensional classification with an elegant probabilistic argument. Suppose we are given a random design matrix X ~ N(0, I_n ⊗ I_p) and arbitrary binary y_i’s that are not all the same. The phase transition of perfect hyperplane separation happens at the point p/n = 1/2. With such a separating hyperplane, there exist some ${\mathit{\beta }}^{*}\in {ℝ}^p$ and $t\in ℝ$ such that $x_i^T{\mathit{\beta }}^{*}>t$ for all cases y_i = 1 and $x_i^T{\mathit{\beta }}^{*}<t$ for all controls y_i = 0. Let us fit a logistic regression model with an intercept. It is easy to show that multiplying the vector (−t, (β*)^T)^T by a divergence sequence of positive numbers c, we can obtain a sequence of logistic regression fits with the fitted response vector approaching y = (y₁, ⋯ , y_n)^T as c → ∞. As a consequence, the MLE algorithm can return a pretty wild estimate that is close to infinity in topology when the algorithm is set to stop. Clearly, in such a case the p-value of the MLE is no longer justified and meaningful. The results in Candès (2016) have two important implications. First, such results reveal that unlike in linear models, p-values in nonlinear models can break down and behave wildly when p/n is of order 1/2; see Karoui et al. (2013); Bean et al. (2013) and discussions below. Second, these results motivate us to characterize the breakdown point of p-values in nonlinear GLMs with $p~n^{{\alpha }_0}$ in the regime of α₀ ∈ [0, 1/2). In fact, our results show that the breakdown point can be even much earlier than n/2.

It is worth mentioning that our work is different in goals from the limited but growing literature on p-values for high-dimensional nonlinear models, and makes novel contributions to such a problem. The key distinction is that existing work has focused primarily on identifying the scenarios in which conventional p-values or their modifications continue to be valid with some sparsity assumption limiting the growth of intrinsic dimensions. For example, Fan and Peng (2004) established the oracle property including the asymptotic normality for nonconcave penalized likelihood estimators in the scenario of p = o(n^1/5), while Fan and Lv (2011) extended their results to the GLM setting of non-polynomial (NP) dimensionality. In the latter work, the p-values were proved to be valid under the assumption that the intrinsic dimensionality s = o(n^1/3). More recent work on high-dimensional inference in nonlinear model settings includes van de Geer et al. (2014); Athey et al. (2016) under sparsity assumptions. In addition, two tests were introduced in Guo and Chen (2016) for high-dimensional GLMs without or with nuisance regression parameters, but the p-values were obtained for testing the global hypothesis for a given set of covariates, which is different from our goal of testing the significance of individual covariates simultaneously. Portnoy (1988) studied the asymptotic behavior of the MLE for exponential families under the classical i.i.d. non-regression setting, but with diverging dimensionality. In contrast, our work under the GLM assumes the regression setting in which the design matrix X plays an important role in the asymptotic behavior of the MLE $\widehat{\beta }$. The validity of the asymptotic normality of the MLE was established in Portnoy (1988) under the condition of p = o(n^1/2), but the precise breakdown point in diverging dimensionality was not investigated therein. Another line of work is focused on generating asymptotically valid p-values when p/n converges to a fixed positive constant. For instance, Karoui et al. (2013) and Bean et al. (2013) considered M-estimators in the linear model and showed that their variance is greater than classically predicted. Based on this result, it is possible to produce p-values by making adjustments for the inflated variance in high dimensions. Recently, Sur and Candès (2018) showed that similar adjustment is possible for the likelihood ratio test (LRT) for logistic regression. Our work differs from this line of work in two important aspects. First, our focus is on the classical p-values and their validity. Second, their results concern dimensionality that is comparable to sample size, while we aim to analyze the problem for a lower range of dimensionality and pinpoint the exact breakdown point of p-values.

The rest of the paper is organized as follows. Section 2 provides characterizations of p-values in low dimensions. We establish the nonuniformity of GLM p-values in diverging dimensions in Section 3. Section 4 presents several simulation examples verifying the theoretical phenomenon. We discuss some implications of our results in Section 5. The proofs of all the results are relegated to the Appendix.

2. Characterizations of P-values in Low Dimensions

To pinpoint the breakdown point of GLM p-values in diverging dimensions, we start with characterizing p-values in low dimensions. In contrast to existing work on the asymptotic distribution of the penalized MLE, our results in this section focus on the asymptotic normality of the unpenalized MLE in diverging-dimensional GLMs, which justifies the validity of conventional p-values. Although Theorems 1 and 4 to be presented in Sections 2.2 and A are in the conventional sense of relatively small p, to the best of our knowledge such results are not available in the literature before in terms of the maximum range of dimensionality p without any sparsity assumption.

2.1. Maximum likelihood estimation

For the GLM (1), the log-likelihood log f_n(y; X, β) of the sample is given, up to an affine transformation, by

$${\mathcal{l}}_n(\mathit{\beta })=n^{-1}\left[y^{T}X\mathit{\beta }-1^{T}b(X\mathit{\beta })\right],$$ (2)

where b(θ) = (b(θ₁), ⋯ , b(θ_n))^T for $\mathit{\theta }={\left({\theta }_1,\cdots ,{\theta }_n\right)}^T\in {ℝ}^n$. Denote by $\widehat{\mathit{\beta }}={\left({\widehat{\beta }}_1,\cdots ,{\widehat{\beta }}_p\right)}^T\in {ℝ}^p$ the MLE which is the maximizer of (2), and

$$\mathit{\mu }(\mathit{\theta })={\left(b^{\prime }\left({\theta }_1\right),\cdots ,b^{\prime }\left({\theta }_n\right)\right)}^T\text{\hspace{1em}and\hspace{1em}}\Sigma (\mathit{\theta })=\text{diag}\left\{b^{\prime \prime }\left({\theta }_1\right),\cdots ,b^{\prime \prime }\left({\theta }_n\right)\right\}.$$ (3)

A well-known fact is that the n-dimensional response vector y in GLM (1) has mean vector μ(θ) and covariance matrix ϕΣ(θ). Clearly, the MLE $\widehat{\mathit{\beta }}$ is given by the unique solution to the score equation

$$X^T[y-\mathit{\mu }(X\mathit{\beta })]=0$$ (4)

when the design matrix X is of full column rank p.

It is worth mentioning that for the linear model, the score equation (4) becomes the well-known normal equation X^Ty = X^TXβ which admits a closed form solution. On the other hand, equation (4) does not admit a closed form solution in general nonlinear models. This fact due to the nonlinearity of the mean function μ(·) causes the key diffierence between the linear and nonlinear models. In future presentations, we will occasionally use the term nonlinear GLMs to exclude the linear model from the family of GLMs when necessary.

We will present in the next two sections some sufficient conditions under which the asymptotic normality of MLE holds. In particular, Section 2.2 concerns the case of fixed design and Section A deals with the case of random design. In addition, Section 2.2 allows for general regression coefficient vector β₀ and the results extend some existing ones in the literature, while Section A assumes the global null β₀ = 0 and Gaussian random design which enable us to pinpoint the exact breakdown point of the asymptotic normality for the MLE.

2.2. Conventional p-values in low dimensions under fixed design

Recall that we condition on the design matrix X in this section. We first introduce a deviation probability bound that facilitates our technical analysis. Consider both cases of bounded responses and unbounded responses. In the latter case, assume that there exist some constants M, υ₀ > 0 such that

$$\underset{1\le i\le n}{\text{max}}E\left\{\text{exp}\left[\frac{\left|y_i-b^{\prime }\left({\theta }_{0,i}\right)\right|}{M}\right]-1-\frac{\left|y_i-b^{\prime }\left({\theta }_{0,i}\right)\right|}{M}\right\}{M}^2\le \frac{v_0}{2}$$ (5)

with (θ_0,1, ⋯ , θ_0,n)^T = θ₀ = Xβ₀, where β₀ = (β_0,1, ⋯ , β_0,p)^T denotes the true regression coefficient vector in model (1). Then by Fan and Lv (2011, 2013), it holds that for any $a\in {ℝ}^n$,

$$P\left(|a^{T}Y-a^T\mathit{\mu }\left({\mathit{\theta }}_0\right)|>\Vert a{\Vert }_2\epsilon \right)\le \phi (\epsilon ),$$ (6)

where $\phi (\epsilon )=2e^{-c_1{\epsilon }^2}$ with c₁ > 0 some constant, and ε ∈ (0, ∞) if the responses are bounded and ε ∈ (0, ‖a‖₂/‖a‖_∞] if the responses are unbounded.

For nonlinear GLMs, the MLE $\widehat{\mathit{\beta }}$ solves the nonlinear score equation (4) whose solution generally does not admit an explicit form. To address such a challenge, we construct a solution to equation (4) in an asymptotically shrinking neighborhood of β₀ that meets the MLE $\widehat{\mathit{\beta }}$ thanks to the uniqueness of the solution. Specifically, define a neighborhood of β₀ as

$${\mathcal{N}}_0=\left\{\mathit{\beta }\in {ℝ}^p:{\Vert \mathit{\beta }-{\mathit{\beta }}_0\Vert }_{\infty }\le n^{-\gamma }\text{log}n\right\}$$ (7)

for some constant γ ∈ (0, 1/2]. Assume that $p=O\left(n^{{\alpha }_0}\right)$ for some α₀ ∈ (0, γ) and let $b_n=o\left\{\text{min}(n^{1/2-\gamma }\sqrt{\text{log}n},s_n^{-1}{n}^{2\gamma -{\alpha }_0-1/2}/{(\text{log}n)}^2\right\}$ be a diverging sequence of positive numbers, where s_n is a sequence of positive numbers that will be specified in heorem 1 below. We need some basic regularity conditions to establish the asymptotic normality of the MLE $\widehat{\mathit{\beta }}$.

Condition 1

The design matrix X satisfies

$${\Vert {\left[X^T\Sigma \left({\mathit{\theta }}_0\right)X\right]}^{-1}\Vert }_{\infty }=O\left(b_n{n}^{-1}\right),$$ (8)

$$\underset{\mathit{\beta }\in {\mathcal{N}}_0}{\text{max}}{\text{max}}_{j=1}^p{\lambda }_{\text{max}}\left[X^T\text{diag}\left\{\left|x_j\right|\circ \left|{\mathit{\mu }}^{\prime \prime }(X\mathit{\beta })\right|\right\}X\right]=O(n)$$ (9)

with ∘ denoting the Hadamard product and derivatives understood componentwise. Assume that ${\text{max}}_{j=1}^p{\Vert x_j\Vert }_{\infty }<c_1^{1/2}{\{n/(\text{log}n)\}}^{1/2}$ if the responses are unbounded.

Condition 2

The eigenvalues of n⁻¹A_n are bounded away from 0 and ∞, ${\sum }_{i=1}^n{\left(z_i^T{A}_n^{-1}{z}_i\right)}^{3/2}=o\left(1\right)$, and ${\text{max}}_{i=1}^{n}E{\left|y_i-b^{\prime }\left({\theta }_{0,i}\right)\right|}^3=O(1)$, where A_n = X^TΣ(θ₀)X and (z₁, ⋯ , z_n)^T = X.

Conditions 1 and 2 put some basic restrictions on the design matrix X and a moment condition on the responses. For the case of linear model, bound (8) becomes ‖(X^TX)⁻¹‖∞ = O(b_n/n) and bound (9) holds automatically since b′′′ (θ) ≡ 0. Condition 2 is related to the Lyapunov condition.

Theorem 1 (Asymptotic normality)

Assume that Conditions 1–2 and probability bound (6) hold. Then

1. there exists a unique solution $\widehat{\mathit{\beta }}$ to score equation (4) in ${\mathcal{N}}_0$ with asymptotic probability one;

2. the MLE $\widehat{\mathit{\beta }}$ satisfies that for each vector $u\in {ℝ}^p$ with ‖u‖₂ = 1 and ‖u‖₁ = O(s_n),

   $${\left(u^T{A}_n^{-1}u\right)}^{-1/2}\left(u^T\widehat{\mathit{\beta }}-u^T{\mathit{\beta }}_0\right)\stackrel{\mathcal{D}}{\to }N(0,\varphi )$$ (10)

   and specifically for each 1 ≤ j ≤ p,

   $${\left(A_n^{-1}\right)}_{jj}^{-1/2}\left({\widehat{\beta }}_j-{\beta }_{0,j}\right)\stackrel{\mathcal{D}}{\to }N(0,\varphi ),$$ (11)

   where A_n = X^TΣ(θ₀)X and ${\left(A_n^{-1}\right)}_{jj}$ denotes the jth diagonal entry of matrix $A_n^{-1}$.

Theorem 1 establishes the asymptotic normality of the MLE and consequently justifies the validity of the conventional p-values in low dimensions. Note that for simplicity, we present here only the marginal asymptotic normality, and the joint asymptotic normality also holds for the projection of the MLE onto any fixed-dimensional subspace. This result can also be extended to the case of misspecified models; see, for example, Lv and Liu (2014).

As mentioned in the Introduction, the asymptotic normality was shown in Fan and Lv (2011) for nonconcave penalized MLE having intrinsic dimensionality s = o(n^1/3). In contrast, our result in Theorem 1 allows for the scenario of p = o(n^1/2) with no sparsity assumption in view of our technical conditions. In particular, we see that the conventional p-values in GLMs generally remain valid in the regime of slowly diverging dimensionality p = o(n^1/2).

3. Nonuniformity of GLM P-values in Diverging Dimensions

So far we have seen that for nonlinear GLMs, the p-values can be valid when p = o(n^1/2) as shown in Section 2, and can become meaningless when p ≥ n/2 as discussed in the Introduction. Apparently, there is a big gap between these two regimes of growth of dimensionality p. To provide some guidance on the practical use of p-values in nonlinear GLMs, it is of crucial importance to characterize their breakdown point. To highlight the main message with simplified technical presentation, hereafter we content ourselves with the specific case of logistic regression model for binary response. Moreover, we investigate the distributional property in (11) for the scenario of true regression coefficient vector β₀ = 0, that is, under the global null. We argue that this specific model is sufficient for our purpose because if the conventional p-values derived from MLEs fail (i.e., (11) fails) for at least one β₀ (in particular β₀ = 0), then conventional p-values are not justified. Therefore, the breakdown point for logistic regression is at least the breakdown point for general nonlinear GLMs. This argument is fundamentally different from that of proving the overall validity of conventional p-values, where one needs to prove the asymptotic normality of MLEs under general GLMs rather than any specific model.

3.1. The wild side of nonlinear regime

For the logistic regression model (1), we have b(θ) = log(1 + e^θ), $\theta \in ℝ$ and ϕ = 1. The mean vector μ(θ) and covariance matrix ϕΣ(θ) of the n-dimensional response vector y given by (3) now take the familiar form of $\mathit{\mu }(\mathit{\theta })={\left(\frac{e^{{\theta }_1}}{1+e^{{\theta }_1}},\cdots ,\frac{e^{{\theta }_n}}{1+e^{{\theta }_n}}\right)}^T$ and

$$\Sigma (\mathit{\theta })=\text{diag}\left\{\frac{e^{{\theta }_1}}{{\left(1+e^{{\theta }_1}\right)}^2},\cdots ,\frac{e^{{\theta }_n}}{{\left(1+e^{{\theta }_n}\right)}^2}\right\}$$

with θ = (θ₁, ⋯ , θ_n)^T = Xβ. In many real applications, one would like to interpret the significance of each individual covariate produced by algorithms based on the conventional asymptotic normality of the MLE as established in Theorem 1. As argued at the beginning of this section, in order to justify the validity of p-values in GLMs, the underlying theory should at least ensure that the distributional property (11) holds for logistic regression under the global null. As we will see empirically in Section 4, as the dimensionality increases, p-values from logistic regression under the global null have a distribution that is skewed more and more toward zero. Consequently, classical hypothesis testing methods which reject the null hypothesis when p-value is less than the pre-specified level α would result in more false discoveries than the desired level of α. As a result, practitioners may simply lose the theoretical backup and the resulting decisions based on the p-values can become ineffective or even misleading. For this reason, it is important and helpful to identify the breakdown point of p-values in diverging-dimensional logistic regression model under the global null.

Characterizing the breakdown point of p-values in nonlinear GLMs is highly nontrivial and challenging. First, the nonlinearity generally causes the MLE to take no analytical form, which makes it di cult to analyze its behavior in diverging dimensions. Second, conventional probabilistic arguments for establishing the central limit theorem of MLE only enable us to see when the distributional property holds, but not exactly at what point it fails. To address these important challenges, we introduce novel geometric and probabilistic arguments presented later in the proofs of Theorems 2–3 that provide a rather delicate analysis of the MLE. In particular, our arguments unveil that the early breakdown point of p-values in nonlinear GLMs is essentially due to the nonlinearity of the mean function μ(·). This shows that p-values can behave wildly much early on in diverging dimensions when we move away from linear regression model to nonlinear regression models such as the widely applied logistic regression; see the Introduction for detailed discussions on the p-values in diverging-dimensional linear models.

Before presenting the main results, let us look at the specific case of logistic regression model under the global null. In such a scenario, it holds that θ₀ = Xβ₀ = 0 and thus Σ(θ₀) = 4⁻¹I_n, which results in

$$A_n=X^T\Sigma \left({\mathit{\theta }}_0\right)X=4^{-1}{X}^{T}X.$$

In particular, we see that when n⁻¹X^TX is close to the identity matrix I_p, the asymptotic standard deviation of the jth component ${\widehat{\beta }}_j$ of the MLE $\widehat{\mathit{\beta }}$ is close 2n^−1/2 when the asymptotic theory in (11) holds. As mentioned in the Introduction, when p ≥ n/2 the MLE can blow up with excessively large variance, a strong evidence against the distributional property in (11). In fact, one can also observe inflated variance of the MLE relative to what is predicted by the asymptotic theory in (11) even when the dimensionality p grows at a slower rate with sample size n. As a consequence, the conventional p-values given by algorithms according to property (11) can be much biased toward zero and thus produce more significant discoveries than the truth. Such a breakdown of conventional p-values is delineated clearly in the simulation examples presented in Section 4.

3.2. Main results

We now present the formal results on the invalidity of GLM p-values in diverging dimensions.

Theorem 2 (Uniform orthonormal design)¹

Assume that n^−1/2X is uniformly distributed on the Stiefel manifold $V_p\left({ℝ}^n\right)$ consisting of all n × p orthonormal matrices. Then for the logistic regression model under the global null, the asymptotic normality of the MLE established in (11) fails to hold when p ~ n^2/3, where ~ stands for asymptotic order.

Theorem 3 (Correlated Gaussian design)

Assume that X ~ N(0, I_n ⊗ Σ) with covariance matrix Σ nonsingular. Then for the logistic regression model under the global null, the same conclusion as in Theorem 2 holds.

Theorem 4 in Appendix A states that under the global null in GLM with Gaussian design, the p-value based on the MLE remains valid as long as the dimensionality p diverges with n at a slower rate than n^2/3. This together with Theorems 2 and 3 shows that under the global null, the exact breakdown point for the uniformity of p-value is n^2/3. We acknowledge that these results are mainly for theoretical interests because in practice one cannot check precisely whether the global null assumption holds or not. However, these results clearly suggest that in GLM with diverging dimensionality, one needs to be very cautious when using p-values based on the MLE.

The key ingredients of our new geometric and probabilistic arguments are demonstrated in the proof of Theorem 2 in Section B.3. The assumption that the rescaled random design matrix n^−1/2X has the Haar measure on the Stiefel manifold $V_p\left({ℝ}^n\right)$ greatly facilitates our technical analysis. The major theoretical finding is that the nonlinearity of the mean function μ(·) can be negligible in determining the asymptotic distribution of MLE as given in (11) when the dimensionality p grows at a slower rate than n^2/3, but such nonlinearity can become dominant and deform the conventional asymptotic normality when p grows at rate n^2/3 or faster. See the last paragraph of Section B.3 for more detailed in-depth discussions on such an interesting phenomenon. Furthermore, the global null assumption is a crucial component of our geometric and probabilistic argument. The global null assumption along with the distributional assumption on the design matrix ensures the symmetry property of the MLE and the useful fact that the MLE can be asymptotically independent of the random design matrix. In the absence of such an assumption, we may suspect that p-values of the noise variables can be affected by the signal variables due to asymmetry. Indeed, our simulation study in Section 4 reveals that as the number of signal variables increases, the breakdown point of the p-values occurs even earlier.

Theorem 3 further establishes that the invalidity of GLM p-values in high dimensions beyond the scenario of orthonormal design matrices considered in Theorem 2. The breakdown of the conventional p-values occurs regardless of the correlation structure of the covariates.

Our theoretical derivations detailed in the Appendix also suggest that the conventional p-values in nonlinear GLMs can generally fail to be valid when $p~n^{{\alpha }_0}$ with α₀ ranging between 1/2 and 2/3, which differs significantly from the phenomenon for linear models as discussed in the Introduction. The special feature of logistic regression model that the variance function b″ (θ) takes the maximum value 1/4 at natural parameter θ = 0 leads to a higher transition point of $p~n^{{\alpha }_0}$ with α₀ = 2/3 for the case of global null β₀ = 0.

4. Numerical Studies

We now investigate the breakdown point of p-values for nonlinear GLMs in diverging dimensions as predicted by our major theoretical results in Section 3 with several simulation examples. Indeed, these theoretical results are well supported by the numerical studies.

4.1. Simulation examples

Following Theorems 2–3 in Section 3, we consider three examples of the logistic regression model (1). The response vector y = (y₁, ⋯ , y_n)^T has independent components and each y_i has Bernoulli distribution with parameter $e^{{\theta }_i}/\left(1+e^{{\theta }_i}\right)$, where θ = (θ₁, ⋯ ,θ_n)^T = Xβ₀. In example 1, we generate the n × p design matrix X = (x₁, ⋯ , x_p) such that n^−1/2X is uniformly distributed on the Stiefel manifold $V_p\left({ℝ}^n\right)$ as in Theorem 2, while examples 2 and 3 assume that X ~ N(0, I_n ⊗ Σ) with covariance matrix Σ as in Theorem 3. In particular, we choose Σ = (ρ^|j−k|)_1≤j,k≤p with ρ = 0, 0.5, and 0.8 to reflect low, moderate, and high correlation levels among the covariates. Moreover, examples 1 and 2 assume the global null model with β₀ = 0 following our theoretical results, whereas example 3 allows sparsity s = ‖β₀‖₀ to vary.

To examine the asymptotic results we set the sample size n = 1000. In each example, we consider a spectrum of dimensionality p with varying rate of growth with sample size n. As mentioned in the Introduction, the phase transition of perfect hyperplane separation happens at the point p/n = 1/2. Recall that Theorems 2–3 establish that the conventional GLM p-values can become invalid when p ~ n^2/3. We set $p=\left[n^{{\alpha }_0}\right]$ with α₀ in the grid {2/3 – 4δ, ⋯ , 2/3 – δ, 2/3, 2/3 + δ, ⋯ ,2/3 + 4δ, (log(n) – log(2))/log(n)} for δ = 0.05. For example 3, we pick s signals uniformly at random among all but the first components, where a random half of them are chosen as 3 and the other half are set as −3.

The goal of the simulation examples is to investigate empirically when the conventional GLM p-values could break down in diverging dimensions. When the asymptotic theory for the MLE in (11) holds, the conventional p-values would be valid and distributed uniformly on the interval [0, 1] under the null hypothesis. Note that the first covariate x₁ is a null variable in each simulation example. Thus in each replication, we calculate the conventional p-value for testing the null hypothesis H₀ : β_0,1 = 0. To check the validity of these p-values, we further test their uniformity.

For each simulation example, we first calculate the p-values for a total of 1, 000 replications as described above and then test the uniformity of these 1, 000 p-values using, for example, the Kolmogorov–Smirnov (KS) test (Kolmogorov, 1933; Smirnov, 1948) and the Anderson–Darling (AD) test (Anderson and Darling, 1952, 1954). We repeat this procedure 100 times to obtain a final set of 100 new p-values from each of these two uniformity tests. Specifically, the KS and AD test statistics for testing the uniformity on [0, 1] are defined as

$$\text{KS}=\underset{x\in [0,1]}{\text{sup}}\left|F_m(x)-x\right|\text{\hspace{1em}and\hspace{1em}AD}=m{\int }_0^1\frac{{\left[F_m(x)-x\right]}^2}{x(1-x)}dx,$$

respectively, where $F_m(x)=m^{-1}{\sum }_{i=1}^m{I}_{(-\infty ,x]}\left(x_i\right)$ is the empirical distribution function for a given sample ${\left\{x_i\right\}}_{i=1}^m$.

4.2. Testing results

For each simulation example, we apply both KS and AD tests to verify the asymptotic theory for the MLE in (11) by testing the uniformity of conventional p-values at significance level 0.05. As mentioned in Section 4.1, we end up with two sets of 100 new p-values from the KS and AD tests. Figures 1–3 depict the boxplots of the p-values obtained from both KS and AD tests for simulation examples 1–3, respectively. In particular, we observe that the numerical results shown in Figures 1–2 for examples 1–2 are in line with our theoretical results established in Theorems 2–3, respectively, for diverging-dimensional logistic regression model under global null that the conventional p-values break down when $p~n^{{\alpha }_0}$ with α₀ = 2/3. Figure 3 for example 3 examines the breakdown point of p-values with varying sparsity s. It is interesting to see that the breakdown point shifts even earlier when s increases as suggested in the discussions in Section 3.2. The results from the AD test are similar so we present only the results from the KS test for simplicity.

Figure 1:

Results of KS and AD tests for testing the uniformity of GLM p-values in simulation example 1 for diverging-dimensional logistic regression model with uniform orthonormal design under global null. The vertical axis represents the p-value from the KS and AD tests, and the horizontal axis stands for the growth rate α₀ of dimensionality p = [n^α0].

Figure 3:

Results of KS test for testing the uniformity of GLM p-values in simulation example 3 for diverging-dimensional logistic regression model with uncorrelated Gaussian design under global null for varying sparsity s. The vertical axis represents the p-value from the KS test, and the horizontal axis stands for the growth rate α₀ of dimensionality p = [n^α0].

Figure 2:

Results of KS and AD tests for testing the uniformity of GLM p-values in simulation example 2 for diverging-dimensional logistic regression model with correlated Gaussian design under global null for varying correlation level ρ. The vertical axis represents the p-value from the KS and AD tests, and the horizontal axis stands for the growth rate α₀ of dimensionality p = [n^α0].

To gain further insights into the nonuniformity of the null p-values, we next provide an additional figure in the setting of simulation example 1. Specifically, in Figure 4 we present the histograms of the 1,000 null p-values from the first simulation repetition (out of 100) for each value of α₀. It is seen that as the dimensionality increases (i.e., α₀ increases), the null p-values have a distribution that is skewed more and more toward zero, which is prone to produce more false discoveries if these p-values are used naively in classical hypothesis testing methods.

Figure 4:

Histograms of null p-values in simulation example 1 from the first simulation repetition for different growth rates α₀ of dimensionality p = [n^α0].

To further demonstrate the severity of the problem, we estimate the probability of making type I error at significance level a, as the fraction of p-values below a. The means and standard deviations of the estimated probabilities are reported in Table 1 for a = 0.05 and 0.1. When the null p-values are distributed uniformly, the probabilities of making type I error should all be close to the target level a. However, Table 1 shows that when the growth rate of dimensionality α₀ approaches or exceeds 2/3, these probabilities can be much larger than a, which again supports our theoretical findings. Also it is seen that when α₀ is close to but still smaller than 2/3, the averages of estimated probabilities exceed slightly a, which could be the effect of finite sample size.

Table 1:

Means and standard deviations (SD) for estimated probabilities of making type I error in simulation example 1 with α₀ the growth rate of dimensionality p = [n^α0]. Two significance levels a = 0.05 and 0.1 are considered.

	α 0	0.10	0.47	0.57	0.67	0.77	0.87
a = 0.05	Mean	0.050	0.052	0.055	0.063	0.082	0.166
	SD	0.006	0.007	0.007	0.007	0.001	0.011
a = 0.1	Mean	0.098	0.104	0.107	0.118	0.144	0.247
	SD	0.008	0.010	0.009	0.011	0.012	0.013

5. Discussions

In this paper we have provided characterizations of p-values in nonlinear GLMs with diverging dimensionality. The major findings are that the conventional p-values can remain valid when p = o(n^1/2), but can become invalid much earlier in nonlinear models than in linear models, where the latter case can allow for p = o(n). In particular, our theoretical results pinpoint the breakdown point of p ~ n^2/3 for p-values in diverging-dimensional logistic regression model under global null with uniform orthonormal design and correlated Gaussian design, as evidenced in the numerical results. It would be interesting to investigate such a phenomenon for more general class of random design matrices.

The problem of identifying the breakdown point of p-values becomes even more complicated and challenging when we move away from the setting of global null. Our technical analysis suggests that the breakdown point $p~n^{{\alpha }_0}$ can shift even earlier with α₀ ranging between 1/2 and 2/3. But the exact breakdown point can depend upon the number of signals s, the signal magnitude, and the correlation structure among the covariates in a rather complicated fashion. Thus more delicate mathematical analysis is needed to obtain the exact relationship. We leave such a problem for future investigation. Moving beyond the GLM setting will further complicate the theoretical analysis.

As we routinely produce p-values using algorithms, the phenomenon of nonuniformity of p-values occurring early in diverging dimensions unveiled in the paper poses useful cautions to researchers and practitioners when making decisions in real applications using results from p-value based methods. For instance, when testing the joint significance of covariates in diverging-dimensional nonlinear models, the effective sample size requirement should be checked before interpreting the testing results. Indeed, statistical inference in general high-dimensional nonlinear models is particularly challenging since obtaining accurate p-values is generally not easy. One possible route is to bypass the use of p-values in certain tasks including the false discovery rate (FDR) control; see, for example, Barber and Candès (2015); Candès et al. (2018); Fan et al. (2018) for some initial efforts made along this line.

Appendix A. Conventional P-values in Low Dimensions under Random Design

Under the specific assumption of Gaussian design and global null β₀ = 0, we can show that the asymptotic normality of MLE continues to hold without previous Conditions 1–2.

Theorem 4

Assume that β₀ = 0, the rows of X are i.i.d. from N(0, Σ), b⁽⁵⁾(·) is uniformly bounded in its domain, and y − μ₀ has uniformly sub-Gaussian components. Then if p = O(n^α) with some α ∈ [0, 2/3), we have the componentwise asymptotic normality

$${\left(A_n^{-1}\right)}_{jj}^{-1/2}{\widehat{\beta }}_j\stackrel{\mathcal{D}}{\to }N(0,\varphi ),$$

where all the notation is the same as in (11).

Theorem 4 shows that the conclusions of Theorem 1 continue to hold for the case of random design and global null with the major difference that the dimensionality can be pushed as far as p ~ n^2/3. The main reasons for presenting Theorem 4 under Gausssian design are twofold. First, Gaussian design is a widely used assumption in the literature. Second, our results on the nonuniformity of GLM p-values in diverging dimensions use geometric and probabilistic arguments which require random design setting; see Section 3 for more details. To contrast more accurately the two regimes and maintain self-contained theory, we have chosen to present Theorem 4 under Gaussian design. On the other hand, we would like to point out that Theorem 4 is not for practitioners who want to justify the usage of classical p-values. The global null assumption of β₀ = 0 restricts the validity of Theorem 4 in many practical scenarios.

Appendix B. Proofs of Main Results

We provide the detailed proofs of Theorems 1–3 in this Appendix.

B.1. Proof of Theorem 1

To ease the presentation, we split the proof into two parts, where the first part locates the MLE $\widehat{\mathit{\beta }}$ in an asymptotically shrinking neighborhood ${\mathcal{N}}_0$ of the true regression coefficient vector β₀ with significant probability and the second part further establishes its asymptotic normality.

Part 1: Existence of a unique solution to score equation (4) in ${\mathcal{N}}_0$ under Condition 1 and probability bound (6). For simplicity, assume that the design matrix X is rescaled columnwise such that ${\Vert x_j\Vert }_2=\sqrt{n}$ for each 1 ≤ j ≤ p. Consider an event

$$\mathcal{E}=\left\{\Vert \xi {\Vert }_{\infty }\le c_1^{-1/2}\sqrt{n\text{log}n}\right\},$$ (12)

where ξ = (ξ₁, ⋯ , ξ_p)^T = X^T [y − μ(θ₀)]. Note that for unbounded responses, the assumption of ${\text{max}}_{j=1}^p{\Vert x_j\Vert }_{\infty }<c_1^{1/2}{\{n/(\text{log}n)\}}^{1/2}$ in Condition 1 entails that $c_1^{-1/2}\sqrt{\text{log}n}<{\text{min}}_{j=1}^p\left\{{\Vert x_j\Vert }_2/{\Vert x_j\Vert }_{\infty }\right\}$. Thus by $\Vert x_j{\Vert }_2=\sqrt{n}$, probability bound (6), and Bonferroni’s inequality, we deduce

$$\begin{gathered} P(\mathcal{E})\ge 1-\sum _{j=1}^{p}P\left(\left|{\xi }_j\right|>c_1^{-1/2}\sqrt{n\text{log}n}\right) \\ \ge 1-2p{n}^{-1}=1-O\left\{n^{-\left(1-{\alpha }_0\right)}\right\}, \end{gathered}$$ (13)

since $p=O\left(n^{{\alpha }_0}\right)$ for some α₀ ∈ (0,γ) with γ ∈ (0, 1/2] by assumption. Hereafter we condition on the event $\mathcal{E}$ defined in (12) which holds with significant probability.

We will show that for sufficiently large n, the score equation (4) has a solution in the neighborhood ${\mathcal{N}}_0$ which is a hypercube. Define two vector-valued functions

$$\mathit{\gamma }(\mathit{\beta })={\left({\gamma }_1(\mathit{\beta }),\cdots ,{\gamma }_p(\mathit{\beta })\right)}^T=X^T\mu (X\mathit{\beta })$$

and

$$\Psi (\mathit{\beta })=\mathit{\gamma }(\mathit{\beta })-\mathit{\gamma }\left({\mathit{\beta }}_0\right)-\xi ,\mathit{\beta }\in {ℝ}^p.$$

Then equation (4) is equivalent to Ψ(β) = 0. We need to show that the latter has a solution inside the hypercube ${\mathcal{N}}_0$. To this end, applying a second order Taylor expansion of γ(β) around β₀ with the Lagrange remainder term componentwise leads to

$$\mathit{\gamma }(\mathit{\beta })=\mathit{\gamma }\left({\mathit{\beta }}_0\right)+X^T\Sigma \left({\theta }_0\right)X\left(\mathit{\beta }-{\mathit{\beta }}_0\right)+r,$$ (14)

where r = (r₁, ⋯ , r_p)^T and for each 1 ≤ j ≤ p,

$$r_j=\frac{1}{2}{\left(\mathit{\beta }-{\mathit{\beta }}_0\right)}^T{\nabla }^2{\gamma }_j\left({\mathit{\beta }}_j\right)\left(\mathit{\beta }-{\mathit{\beta }}_0\right)$$

with β_j some p-dimensional vector lying on the line segment joining β and β₀. It follows from (9) in Condition 1 that

$$\begin{gathered} {\Vert r\Vert }_{\infty }\le \underset{\delta \in {\mathcal{N}}_0}{\text{max}}\underset{j=1}{\overset{p}{\text{max}}}\frac{1}{2}{\lambda }_{\text{max}}\left[X^T\text{diag}\left\{\left|x_j\right|\circ \left|{\mu }^{″}(X\mathit{\delta })\right|\right\}X\right]{\Vert \mathit{\beta }-{\mathit{\beta }}_0\Vert }_2^2 \\ =O\left\{p{n}^{1-2\gamma }{(\text{log}n)}^2\right\}. \end{gathered}$$ (15)

Let us define another vector-valued function

$$\overline{\Psi }(\mathit{\beta })\equiv {\left[X^T\mathbf{\Sigma }\left({\mathit{\theta }}_0\right)X\right]}^{-1}\mathbf{\Psi }(\mathit{\beta })=\mathit{\beta }-{\mathit{\beta }}_0+u,$$ (16)

where u = −[X^TΣ(θ₀)X]⁻¹(ξ − r). It follows from (12), (15), and (8) in Condition 1 that for any $\mathit{\beta }\in {\mathcal{N}}_0$,

$$\begin{gathered} {\Vert u\Vert }_{\infty }\le {\Vert {\left[X^T\Sigma \left({\theta }_0\right)X\right]}^{-1}\Vert }_{\infty }\left(\Vert \xi {\Vert }_{\infty }+\Vert r{\Vert }_{\infty }\right) \\ =O\left[b_n{n}^{-1/2}\sqrt{\text{log}n}+b_{n}p{n}^{-2\gamma }{(\text{log}n)}^2\right]. \end{gathered}$$ (17)

By the assumptions of $p=O\left(n^{{\alpha }_0}\right)$ with constant α₀ ∈ (0, γ) and $b_n=o\left\{\text{min}(n^{1/2-\gamma }\sqrt{\text{log}n},n^{2\gamma -{\alpha }_0-1/2}/{(\text{log}n)}^2\right\}$, We have

$$\Vert u{\Vert }_{\infty }=o\left(n^{-\gamma }\text{log}n\right).$$

Thus in light of (16), it holds for large enough n that when ${\left(\mathit{\beta }-{\mathit{\beta }}_0\right)}_j=n^{-\gamma }\sqrt{\text{log}n}$,

$${\overline{\mathbf{\Psi }}}_j\left(\mathit{\beta }\right)\ge n^{-\gamma }\sqrt{\text{log}n}-{\Vert u\Vert }_{\infty }\ge 0,$$ (18)

and when ${\left(\mathit{\beta }-{\mathit{\beta }}_0\right)}_j=-n^{-\gamma }\sqrt{\text{log}n}$,

$${\overline{\mathbf{\Psi }}}_j\left(\mathit{\beta }\right)\le -n^{-\gamma }\sqrt{\text{log}n}+{\Vert u\Vert }_{\infty }\le 0,$$ (19)

where $\overline{\Psi }(\mathit{\beta })={\left({\overline{\Psi }}_1(\mathit{\beta }),\cdots ,{\overline{\Psi }}_p(\mathit{\beta })\right)}^T$.

By the continuity of the vector-valued function $\overline{\Psi }(\mathit{\beta })$, (18), and (19), Miranda’s existence theorem Vrahatis (1989) ensures that equation $\overline{\Psi }(\mathit{\beta })=0$ has a solution $\widehat{\mathit{\beta }}$ in ${\mathcal{N}}_0$. Clearly, $\widehat{\mathit{\beta }}$ also solves equation $\Psi (\mathit{\beta })=0$ in view of (16). Therefore, we have shown that score equation (4) indeed has a solution $\widehat{\mathit{\beta }}$ in ${\mathcal{N}}_0$. The strict concavity of the log-likelihood function (2) by assumptions for model (1) entails that $\widehat{\mathit{\beta }}$ is the MLE.

Part 2: Conventional asymptotic normality of the MLE $\widehat{\mathit{\beta }}$. Fix any 1 ≤ j ≤ p. In light of (16), we have $\widehat{\mathit{\beta }}-{\mathit{\beta }}_0=A_n^{-1}(\xi -r)$, which results in

$${\left(A_n^{-1}\right)}_{jj}^{-1/2}\left({\widehat{\beta }}_j-{\beta }_{0,j}\right)={\left(A_n^{-1}\right)}_{jj}^{-1/2}{e}_j^T{A}_n^{-1}\xi -{\left(A_n^{-1}\right)}_{jj}^{-1/2}{e}_j^T{A}_n^{-1}r$$ (20)

with $e_j\in {ℝ}^p$ having one for the jth component and zero otherwise. Note that since the smallest and largest eigenvalues of n⁻¹A_n are bounded away from 0 and ∞ by Condition 2, it is easy to show that ${\left(A_n^{-1}\right)}_{jj}^{-1/2}$ is of exact order n^1/2. In view of (17), it holds on the event $\mathcal{E}$ defined in (12) that

$$\begin{gathered} {\Vert A_n^{-1}r\Vert }_{\infty }\le {\Vert {\left[X^T\Sigma \left({\theta }_0\right)X\right]}^{-1}\Vert }_{\infty }{\Vert r\Vert }_{\infty } \\ =O\left[b_{n}p{n}^{-2\gamma }{(\text{log}n)}^2\right]=o\left(n^{-1/2}\right), \end{gathered}$$

since $b_n=o\left\{n^{2\gamma -{\alpha }_0-1/2}/{(\text{log}n)}^2\right\}$ by assumption. This leads to

$${\left(A_n^{-1}\right)}_{jj}^{-1/2}{e}_j^T{A}_n^{-1}r=O\left(n^{1/2}\right)\cdot o_P\left(n^{-1/2}\right)=o_P(1).$$ (21)

It remains to consider the term ${\left(A_n^{-1}\right)}_{jj}^{-1/2}{e}_j^T{A}_n^{-1}\xi =\sum _{i=1}^n{\eta }_i$, where ${\eta }_i={\left(A_n^{-1}\right)}_{jj}^{-1/2}{e}_j^T{A}_n^{-1}{z}_i\left[y_i-b^{\prime }\left({\theta }_{0,i}\right)\right]$. Clearly, the n random variables η_i’s are independent with mean 0 and

$$\sum _{i=1}^n\text{var}\left({\eta }_i\right)={\left(A_n^{-1}\right)}_{jj}^{-1}{e}_j^T{A}_n^{-1}\left(\varphi A_n\right)A_n^{-1}{e}_j=\varphi .$$

It follows from Condition 2 and the Cauchy–Schwarz inequality that

$$\begin{gathered} \sum _{i=1}^{n}E{\left|{\eta }_i\right|}^3=\sum _{i=1}^n{\left|{\left(A_n^{-1}\right)}_{jj}^{-1/2}{e}_j^T{A}_n^{-1}{z}_i\right|}^{3}E{\left|y_i-b^{\prime }\left({\theta }_{0,i}\right)\right|}^3 \\ =O(1)\sum _{i=1}^n{\left|{\left(A_n^{-1}\right)}_{jj}^{-1/2}{e}_j^T{A}_n^{-1}{z}_i\right|}^3 \\ \le O(1)\sum _{i=1}^n{\Vert {\left(A_n^{-1}\right)}_{jj}^{-1/2}{e}_j^T{A}_n^{-1/2}\Vert }_2^3{\Vert A_n^{-1/2}{z}_i\Vert }_2^3 \\ =O(1)\sum _{i=1}^n{\left(z_i^T{A}_n^{-1}{z}_i\right)}^{3/2}=o(1). \end{gathered}$$

Thus an application of Lyapunov’s theorem yields

$${\left(A_n^{-1}\right)}_{jj}^{-1/2}{e}_j^T{A}_n^{-1}\xi =\sum _{i=1}^n{\eta }_i\stackrel{\mathcal{D}}{\to }N(0,\varphi ).$$ (22)

By Slutsky’s lemma, we see from (20)–(22) that

$${\left(A_n^{-1}\right)}_{jj}^{-1/2}\left({\widehat{\beta }}_j-{\beta }_{0,j}\right)\stackrel{\mathcal{D}}{\to }N(0,\varphi ),$$

showing the asymptotic normality of each component ${\widehat{\beta }}_j$ of the MLE $\widehat{\mathit{\beta }}$.

We further establish the asymptotic normality for the one-dimensional projections of the MLE $\widehat{\mathit{\beta }}$. Fix an arbitrary vector $u\in {ℝ}^p$ with ‖u‖₂ = 1 satisfying the L₁ sparsity bound ‖u‖₁ = O(s_n). In light of (16), we have $\widehat{\mathit{\beta }}-{\mathit{\beta }}_0=A_n^{-1}\left(\xi -r\right)$, which results in

$${\left(u^T{A}_n^{-1}u\right)}^{-1/2}\left(u^T\widehat{\mathit{\beta }}-u^T{\mathit{\beta }}_0\right)={\left(u^T{A}_n^{-1}u\right)}^{-1/2}{u}^T{A}_n^{-1}\xi -{\left(u^T{A}_n^{-1}u\right)}^{-1/2}{u}^T{A}_n^{-1}r.$$ (23)

Note that since the smallest and largest eigenvalues of n⁻¹A_n are bounded away from 0 and ∞ by Condition 2, it is easy to show that ${\left(u^T{A}_n^{-1}u\right)}^{-1/2}$ is of exact order n^1/2. In view of (17), it holds on the event $\mathcal{E}$ defined in (12) that

$$\begin{gathered} {\Vert A_n^{-1}r\Vert }_{\infty }\le {\Vert {\left[X^T\Sigma \left({\theta }_0\right)X\right]}^{-1}\Vert }_{\infty }{\Vert r\Vert }_{\infty } \\ =O\left\{b_{n}p{n}^{-2\gamma }{(\text{log}n)}^2\right\}=o\left(s_n^{-1}{n}^{-1/2}\right) \end{gathered}$$

since $b_n=o\left\{s_n^{-1}{n}^{2\gamma -{\alpha }_0-1/2}/{\left(\text{log}n\right)}^2\right\}$ by assumption. This leads to

$${\left(u^T{A}_n^{-1}u\right)}^{-1/2}{u}^T{A}_n^{-1}r=O\left(n^{1/2}\right)\cdot {\Vert u\Vert }_1\cdot {\Vert A_n^{-1}r\Vert }_{\infty }=o_P(1)$$ (24)

since ‖u‖₁ = O(s_n) by assumption.

It remains to consider the term ${\left(u^T{A}_n^{-1}u\right)}^{-1/2}{u}^T{A}_n^{-1}\xi ={\sum }_{i=1}^n{\eta }_i$ with ${\eta }_i={\left(u^T{A}_n^{-1}u\right)}^{-1/2}{u}^T{A}_n^{-1}{z}_i\left[y_i-b^{\prime }\left({\theta }_{0,i}\right)\right]$. Clearly, the n random variables η_i’s are independent with mean 0 and

$$\sum _{i=1}^n\text{var}\left({\eta }_i\right)={\left(u^T{A}_n^{-1}u\right)}^{-1}{u}^T{A}_n^{-1}\left(\varphi A_n\right)A_n^{-1}u=\varphi .$$

It follows from Condition 2 and the Cauchy–Schwarz inequality that

$$\begin{gathered} \sum _{i=1}^{n}E{\left|{\eta }_i\right|}^3=\sum _{i=1}^n{\left|{\left(u^T{A}_n^{-1}u\right)}^{-1/2}{u}^T{A}_n^{-1}{z}_i\right|}^{3}E{\left|y_i-b^{\prime }\left({\theta }_{0,i}\right)\right|}^3 \\ =O(1)\sum _{i=1}^n{\left|{\left(u^T{A}_n^{-1}u\right)}^{-1/2}{u}^T{A}_n^{-1}{z}_i\right|}^3 \\ \le O(1)\sum _{i=1}^n{\Vert {\left(u^T{A}_n^{-1}u\right)}^{-1/2}{u}^T{A}_n^{-1/2}\Vert }_2^3{\Vert A_n^{-1/2}{z}_i\Vert }_2^3 \\ =O(1)\sum _{i=1}^n{\left(z_i^T{A}_n^{-1}{z}_i\right)}^{3/2}=o(1). \end{gathered}$$

Thus an application of Lyapunov’s theorem yields

$${\left(u^T{A}_n^{-1}u\right)}^{-1/2}{u}^T{A}_n^{-1}\xi =\sum _{i=1}^n{\eta }_i\stackrel{\mathcal{D}}{\to }N(0,\varphi ).$$ (25)

By Slutsky’s lemma, we see from (23)–(25) that

$${\left(u^T{A}_n^{-1}u\right)}^{-1/2}\left(u^T\widehat{\beta }-u^T{\mathit{\beta }}_0\right)\stackrel{\mathcal{D}}{\to }N(0,\varphi ),$$

showing the asymptotic normality of any L₁-sparse one-dimensional projection $u^T\widehat{\mathit{\beta }}$ of the MLE $\widehat{\mathit{\beta }}$. This completes the proof of Theorem 1.

B.2. Proof of Theorem 4

The proof is similar to that for Theorem 1. Without loss of generality, we assume that Σ = I_p because under global null, a rotation of X yields standard normal rows. First let ξ = (ξ₁, ⋯ , ξ_p)^T = (X^TX)⁻¹X^T[y − μ₀], where μ₀ = b′ (0)1 with $1={\left(1,\cdots ,1\right)}^T\in {ℝ}^n$ because β₀ = 0. Then y – μ₀ has i.i.d. uniform sub-Gaussian components and is independent of $X=\left(z_1,\cdots ,z_p\right)\in {ℝ}^{n\times p}$. Define event

$$\mathcal{E}=\left\{{\Vert \xi \Vert }_{\infty }\le c_2\sqrt{n^{-1}\text{log}n}\right\}.$$

By Lemma 5, it is seen that $P(\mathcal{E})\ge 1-o\left(p^{-a}\right)$. Furthermore, define the neighborhood

$${\mathcal{N}}_0=\left\{{\Vert \mathit{\beta }\Vert }_{\infty }\le c_3\sqrt{n^{-1}\text{log}n}\right\}$$ (26)

for some c₃ > c₂(b″ (0))⁻¹. We next show that the MLE must fall into the region ${\mathcal{N}}_0$ with probability at least 1 − O(p^−a) following the similar arguments in Theorem 1.

First, we define

$$\mathit{\gamma }(\mathit{\beta })={\left({\gamma }_1(\mathit{\beta }),\cdots ,{\gamma }_p(\mathit{\beta })\right)}^T\equiv X^T\mu (X\mathit{\beta })$$

and

$$\Psi (\mathit{\beta })=\mathit{\gamma }(\mathit{\beta })-\mathit{\gamma }\left({\mathit{\beta }}_0\right)-{\mathbf{X}}^T\left[y-{\mathit{\mu }}_0\right],\mathit{\beta }\in {ℝ}^p.$$

Applying a forth order Taylor expansion of γ(β) around β₀ = 0 with the Lagrange remainder term componentwise leads to

$$\mathit{\gamma }(\mathit{\beta })=\mathit{\gamma }\left({\mathit{\beta }}_0\right)+b^{″}(0)X^{T}X\left(\mathit{\beta }-{\mathit{\beta }}_0\right)+r+s+t,$$

where r = (r₁, ⋯ , r_p)^T, s = (s₁, ⋯ , s_p)^T, t = (t₁, ⋯ , t_p)^T and for each 1 ≤ j ≤ p,

$$r_j=\frac{b^{‴}(0)}{2}\sum _{i=1}^n{x}_{ij}{\left(x_i^T\mathit{\beta }\right)}^2$$ (27)

$$s_j=\frac{b^{(4)}(0)}{6}\sum _{i=1}^n{x}_{ij}{\left(x_i^T\mathit{\beta }\right)}^3$$ (28)

$$t_j=\frac{1}{24}\sum _{i=1}^n{b}^{(5)}\left(x_i^T\widetilde{{\mathit{\beta }}^j}\right)x_{ij}{\left(x_i^T\mathit{\beta }\right)}^4.$$ (29)

With $\widetilde{{\beta }^j}$ some p-dimensional vector lying on the line segment joining β and β₀.

Let us define another vector-valued function

$$\overline{\Psi }(\mathit{\beta })\equiv {\left[b^{″}(0)X^{T}X\right]}^{-1}\Psi (\mathit{\beta })=\mathit{\beta }-{\mathit{\beta }}_0+u,$$ (30)

where u = −(b″ (0))⁻¹ξ + [b″ (0)X^TX]⁻¹(r + s + t). It follows from the above derivation that for any $\mathit{\beta }\in {\mathcal{N}}_0$,

$${\Vert u\Vert }_{\infty }\le {\Vert {\left(b^{″}(0)\right)}^{-1}\xi \Vert }_{\infty }+{\Vert {\left[b^{″}(0)X^{T}X\right]}^{-1}(r+s+t)\Vert }_{\infty }.$$

Now, we bound the terms on the right hand side.

First note that on event $\mathcal{E}$,

$${\Vert {\left(b^{″}(0)\right)}^{-1}\xi \Vert }_{\infty }\le {\left(b^{″}(0)\right)}^{-1}{c}_2\sqrt{n\text{log}n}.$$ (31)

Then, we consider the next term: ${\Vert {\left[b^{″}(0)X^{T}X\right]}^{-1}(r+s+t)\Vert }_{\infty }$. We observe that

$$\begin{gathered} {\Vert {\left[b^{″}(0)X^{T}X\right]}^{-1}(r+s+t)\Vert }_{\infty }\le {\left|b^{″}(0)\right|}^{-1}{\Vert {\left[n^{-1}{X}^{T}X\right]}^{-1}\Vert }_{\infty }{\Vert n^{-1}(r+s+t)\Vert }_{\infty } \\ \le {\left|b^{″}(0)\right|}^{-1}{\Vert {\left[n^{-1}{X}^{T}X\right]}^{-1}\Vert }_{\infty } \\ \cdot \left({\Vert n^{-1}r\Vert }_{\infty }+{\Vert n^{-1}s\Vert }_{\infty }+{\Vert n^{-1}t\Vert }_{\infty }\right). \end{gathered}$$

By Lemma 6, we have that ${\Vert {\left[n^{-1}{X}^{T}X\right]}^{-1}\Vert }_{\infty }\le 1+O\left(p{n}^{-1/2}\right)$. Lemmas 10, 11, 12 assert that

$$\begin{gathered} \left({\Vert n^{-1}r\Vert }_{\infty }+{\Vert n^{-1}s\Vert }_{\infty }+{\Vert n^{-1}t\Vert }_{\infty }\right) \\ =\left\{n^{\alpha -5/6}\text{log}n+n^{3/2\alpha -5/4}{(\text{log}n)}^{3/2}+n^{\alpha -1}{(\text{log}n)}^{1/2}+n^{2\alpha -3/2}{(\text{log}n)}^{3/2}\right\}\sqrt{n^{-1}\text{log}n}. \end{gathered}$$

We combine last two bounds so that we have

$${\Vert {\left[b^{″}(0)X^{T}X\right]}^{-1}(r+s+t)\Vert }_{\infty }=o(\sqrt{n^{-1}\text{log}n})$$ (32)

with probability at least 1 − o(p^−c) when p = O(n^α) with α < 2/3.

Combining equations (31) and (32), we obtain that if p = O(n^α) with α ∈ [0, 2/3), then

$${\Vert u\Vert }_{\infty }\le c_3\sqrt{n^{-1}\text{log}n}.$$

Thus, the MLE must fall into the region ${\mathcal{N}}_0$ following the similar arguments in Theorem 1.

Next, we show the componentwise asymptotic normality of the MLE $\widehat{\mathit{\beta }}$. By equation (30), we have $\widehat{\mathit{\beta }}=-u={\left(b^{″}(0)\right)}^{-1}{\left(X^{T}X\right)}^{-1}{X}^T\left[y-{\mu }_0\right]-{\left[b^{″}(0)X^{T}X\right]}^{-1}(r+s+t)$. So, we can write

$${\widehat{\beta }}_j={\left(b^{″}(0)\right)}^{-1}{n}^{-1}{e}_j^T{X}^T\left[y-{\mu }_0\right]+{\left(b^{″}(0)\right)}^{-1}T-e_j^T{\left[b^{″}(0)X^{T}X\right]}^{-1}(r+s+t)$$ (33)

where $T=e_j^T{\left(X^{T}X\right)}^{-1}{X}^T\left[y-{\mu }_0\right]-n^{-1}{e}_j^T{X}^T\left[y-{\mu }_0\right]$. By Lemma 13 and Equation (32), both n^1/2(b″ (0))⁻¹T and $n^{1/2}{e}_j^T{\left[b^{″}(0)X^{T}X\right]}^{-1}(r+s+t)$ converges to zero in probability. So, it is enough to consider the first summand in (33). Now, we show that $n^{-1/2}{e}_j^T{X}^T\left[y-{\mu }_0\right]$ is asymptotically normal. In fact, we can write $e_j^T{X}^T\left[y-{\mu }_0\right]={\sum }_{i=1}^n{x}_{ij}{y}_i$ where each summand x_ijy_i is independent over i and has variance ϕb″ (0). Moreover, ${\sum }_{i=1}^{n}E{\left|x_{ij}{y}_i\right|}^3=O(n)$ since |x_ij|³ and |y_i|³ are independent and finite mean. So, we apply Lyapunov’s theorem to obtain $b^{″}{\left(0\right)}^{-1/2}{n}^{-1/2}{e}_j^T{X}^T\left[y-{\mu }_0\right]\stackrel{\mathcal{D}}{\to }N\left(0,\varphi \right)$. Finally, we know that $b^{″}(0)n{\left(A_n^{-1}\right)}_{jj}\to 1$ in probability from the remark in Theorem 1. Thus, Slutsky’s lemma yields

$${\left(A_n^{-1}\right)}_{jj}^{-1/2}{\widehat{\beta }}_j\stackrel{\mathcal{D}}{\to }N(0,\varphi ).$$ (34)

This completes the proof of the theorem.

Lemma 5

Assume that the components of y − μ₀ are uniform sub-Gaussians. That is, there exist a positive constant C such that P(|(y − μ₀)_i| > t) ≤ C exp {−Ct²} for all 1 ≤ i ≤ n. Then, it holds that, for some positive constant c₂,

$${\Vert {\left(X^{T}X\right)}^{-1}{X}^T\left(y-{\mu }_0\right)\Vert }_{\infty }\le c_2\sqrt{n^{-1}\text{log}n}.$$

with asymptotic probability 1 − o(p^−a).

Proof We prove the result by conditioning on X. Let E = n⁻¹X^TX − I_p. Then by matrix inversion,

$$\begin{gathered} {\left(n^{-1}{X}^{T}X\right)}^{-1}={\left(I_p+E\right)}^{-1}=I_p-\sum _{k=1}^{\infty }{\left(-1\right)}^{k+1}{\left(E\right)}^k \\ =I_p-E+\sum _{k=2}^{\infty }{\left(-1\right)}^k{\left(E\right)}^k=2I_p-n^{-1}{X}^{T}X+\sum _{k=2}^{\infty }{\left(-1\right)}^k{\left(E\right)}^k. \end{gathered}$$

Thus, it follows that

$$\begin{gathered} {\Vert {\left(X^{T}X\right)}^{-1}{X}^T\left(y-{\mu }_0\right)\Vert }_{\infty } \\ \le {\Vert 2n^{-1}{X}^T\left(y-{\mu }_0\right)\Vert }_{\infty }+{\Vert n^{-2}{X}^{T}X{X}^T\left(y-{\mu }_0\right)\Vert }_{\infty }+{\Vert n^{-1}\sum _{k=2}^{\infty }{(-1)}^k{(E)}^k{X}^T\left(y-{\mu }_0\right)\Vert }_{\infty } \\ ={\eta }_1+{\eta }_2+{\eta }_3. \end{gathered}$$

In the rest of the proof, we will bound η₁, η₂ and η₃.

Part 1: Bound of η₁.

First, it is easy to see that

$$\begin{gathered} {\eta }_1={\Vert 2n^{-1}{X}^T\left(y-{\mu }_0\right)\Vert }_{\infty } \\ =2\underset{1\le j\le p}{\text{max}}\left|n^{-1}\sum _{i=1}^n{x}_{ij}{\left(y-{\mu }_0\right)}_i\right|. \end{gathered}$$

We observe that each summand x_ij(y − μ₀)_i is the product of two subgaussian random variables, and so satisfies P(|x_ij(y − μ₀)_i| > t) ≤ C exp(−Ct) for some constant C > 0 by Lemma 1 in Fan et al. (2016). Moreover, E[x_ij(y − μ₀)_i] = 0 since x_ij and (y − μ₀)_i are independent and have zero mean. Thus, we can use Lemma 9 by setting W_ij = x_ij(y − μ₀)_i and α = 1. So, we get

$${\eta }_1=2\underset{1\le j\le p}{\text{max}}\left|n^{-1}\sum _{i=1}^n{x}_{ij}{\left(y-{\mu }_0\right)}_i\right|\le c_2\sqrt{n^{-1}\text{log}p}$$ (35)

with probability 1 − O(p^−c) for some positive constants c and c₂.

Part 2: Bound of η₂.

Now, we study η₂ = ‖n⁻²X^TXX^T(y − μ₀)‖_∞. Let z_k be the k-th column of X, that is z_k = Xe_k. Direct calculations yield

$$e_k^T{X}^{T}X{X}^T\left(y-{\mu }_0\right)=\sum _{j=1}^p\left(z_k^T{z}_j\right)\left(z_j^T\left(y-{\mu }_0\right)\right)={\Vert z_k\Vert }_2^2{z}_k^T\left(y-{\mu }_0\right)+\sum _{j\ne k}^p\left(z_k^T{z}_j\right)\left(z_j^T\left(y-{\mu }_0\right)\right).$$

Thus, it follows that

$${\Vert X^{T}X{X}^T\left(y-{\mu }_0\right)\Vert }_{\infty }\le {\text{max}}_k\left|{\Vert z_k\Vert }_2^2{z}_k^T\left(y-{\mu }_0\right)\right|+\underset{k}{\text{max}}\left|\sum _{j\ne k}^p\left(z_k^T{z}_j\right)\left(z_j^T\left(y-{\mu }_0\right)\right)\right|.$$ (36)

First, we consier ${\text{max}}_k\left|{\Vert z_k\Vert }_2^2{z}_k^T\left(y-{\mu }_0\right)\right|$. Lemma 14 shows that ${\text{max}}_k{\Vert z_k\Vert }_2^2\le O(n)$ with probability 1 − O(p^−c). We also have ${\text{max}}_k\left|z_k^T\left(y-{\mu }_0\right)\right|=\frac{n}{2}{\eta }_1\le O\left(\sqrt{n\text{log}p}\right)$ by equation (35). It follows that

$$\underset{k}{\text{max}}\left|\Vert z_k{\Vert }_2^2{z}_k^T\left(y-{\mu }_0\right)\right|\le {\text{max}}_k\Vert z_k{\Vert }_2^2{\text{max}}_k\left|z_k^T\left(y-{\mu }_0\right)\right|\le O(n\sqrt{n\text{log}p}).$$ (37)

Next, let $a_j=z_k^T{z}_j/{\Vert z_k\Vert }_2$ and $b_j=z_j^T\left(y-{\mu }_0\right)/{\Vert y-{\mu }_0\Vert }_2$. Then it is easy to see that conditional on z_k and y, a_j ~ N(0, 1), b_j ~ N(0, 1) and $\mathrm{cov}\left(a_j,b_j|z_k,y\right)=z_k^T\left(y-{\mu }_0\right)/\left({\Vert z_k\Vert }_2{\Vert y-{\mu }_0\Vert }_2\right)$. By (E.6) of Lemma 7 in Fan et al. (2016), it can be shown that

$$\begin{gathered} P\left(\frac{1}{p-1}\left|\sum _{j\ne k}^p\left(z_k^T{z}_j\right)\left(z_j^T\left(y-{\mu }_0\right)\right)-z_k^T\left(y-{\mu }_0\right)\right|\ge c{\Vert z_k\Vert }_2{\Vert y-{\mu }_0\Vert }_2\sqrt{p^{-1}\text{log}p}|z_k,y\right) \\ =P\left(\frac{1}{p-1}\left|\sum _{j\ne k}^p{a}_j{b}_j-\frac{z_k^T\left(y-{\mu }_0\right)}{{\Vert z_k\Vert }_2{\Vert y-{\mu }_0\Vert }_2}\right|\ge c\sqrt{p^{-1}\text{log}p}|z_k,y\right)\le c{p}^{-c_1}, \end{gathered}$$

where c₁ is some large positive constant independent of z_k and y. Moreover, we can choose c₁ as large as we want by increasing c. Thus, it follows that

$$P\left(\frac{1}{p-1}\left|\sum _{j\ne k}^p\left(z_k^T{z}_j\right)\left(z_j^T\left(y-{\mu }_0\right)\right)-z_k^T\left(y-{\mu }_0\right)\right|\ge c{\Vert z_k\Vert }_2{\Vert y-{\mu }_0\Vert }_2\sqrt{p^{-1}\text{log}p}\right)\le c{p}^{-c_1}.$$

It follows from probability union bound that

$$P\left(\frac{1}{p-1}{\text{max}}_k\left|\frac{1}{{\Vert z_k\Vert }_2{\Vert y-{\mu }_0\Vert }_2}\sum _{j\ne k}^p\left(z_k^T{z}_j\right)\left(z_j^T\left(y-{\mu }_0\right)\right)-z_k^T\left(y-{\mu }_0\right)\right|\ge c\sqrt{p^{-1}\text{log}p}\right)\le c{p}^{-c_1+1}.$$

Taking c₁ > 1 yields that with probability at least 1 − o(p^−a) for some a > 0,

$$\underset{k}{\text{max}}\left\{\frac{1}{{\Vert z_k\Vert }_2{\Vert y-{\mu }_0\Vert }_2}\left|\frac{1}{p-1}\sum _{j\ne k}^p\left(z_k^T{z}_j\right)\left(z_j^T\left(y-{\mu }_0\right)\right)-z_k^T\left(y-{\mu }_0\right)\right|\right\}\le c\sqrt{p^{-1}\text{log}p}.$$

By Lemma 14, we have ${\text{max}}_k\Vert z_k{\Vert }_2=\sqrt{{\text{max}}_k\Vert z_k{\Vert }_2^2}\le O_p\left(\sqrt{n}\right)$. Therefore, by using the fact that ${\Vert y-{\mu }_0\Vert }_2\le O_p\left(\sqrt{n}\right)$, we have

$$\begin{gathered} \underset{k}{\text{max}}\left|\sum _{j\ne k}^p\left(z_k^T{z}_j\right)\left(z_j^T\left(y-{\mu }_0\right)\right)\right| \\ \le \underset{k}{\text{max}}\left|\sum _{j\ne k}^p\left[\left(z_k^T{z}_j\right)\left(z_j^T\left(y-{\mu }_0\right)\right)\right]-(p-1)z_k^T\left(y-{\mu }_0\right)\right|+(p-1)\underset{k}{\text{max}}\left|z_k^T\left(y-{\mu }_0\right)\right| \\ \le p\underset{k}{\text{max}}{\Vert z_k\Vert }_2{\Vert y-{\mu }_0\Vert }_2\underset{k}{\text{max}}\left\{\frac{1}{{\Vert z_k\Vert }_2{\Vert y-{\mu }_0\Vert }_2}|\frac{1}{p-1}\sum _{j\ne k}^p\left[\left(z_k^T{z}_j\right)\left(z_j^T\left(y-{\mu }_0\right)\right)-z_k^T\left(y-{\mu }_0\right)\right]\right\}+p\underset{k}{\text{max}}\left|z_k^T\left(y-{\mu }_0\right)\right| \\ \le cpn\sqrt{\text{log}p}\sqrt{p^{-1}\text{log}p}+cp\sqrt{n\text{log}p}. \end{gathered}$$ (38)

Combining (36)–(38) yields

$${\eta }_2={\Vert n^{-2}{X}^{T}X{X}^T\left(y-{\mu }_0\right)\Vert }_{\infty }\le c{p}^{1/2}{n}^{-1}\text{log}p=o(\sqrt{n^{-1}\text{log}n}).$$ (39)

Part 3: Bound of η₃.

Finally, we study η₃. We observe that ${\eta }_3\le \Vert \sum _{k=2}^{\infty }{\left(-1\right)}^{k+1}{\left(E\right)}^k{\Vert }_{\infty }\Vert n^{-1}{X}^T\left(y-{\mu }_0\right){\Vert }_{\infty }$. Lemma 7 proves that $\Vert \sum _{k=2}^{\infty }{\left(-1\right)}^{k+1}{\left(E\right)}^k{\Vert }_{\infty }\le O\left(p^{3/2}{n}^{-1}\right)$ while equation (35) shows that ${\Vert n^{-1}{X}^T\left(y-{\mu }_0\right)\Vert }_{\infty }=O\left(\sqrt{n^{-1}\text{log}p}\right)$ with probability 1 − O(p^−c). Putting these facts together, we obtain

$${\eta }_3\le O\left(p^{3/2}{n}^{-1}\sqrt{n^{-1}\text{log}p}\right)=o(\sqrt{n^{-1}\text{log}n})$$ (40)

where we use $p=O\left(n^{{\alpha }_0}\right)$ with α₀ ∈ [0, 2/3).

Combining equations (35), (39), and (40), we obtain that with probability at least 1 − o(p^−a),

$${\Vert {\left(X^{T}X\right)}^{-1}{X}^T\left(y-{\mu }_0\right)\Vert }_{\infty }\le c\sqrt{n^{-1}\text{log}n}.$$

■

Lemma 6

Under the assumptions of Theorem 4, ‖(n⁻¹X^TX)⁻¹‖_∞ ≤ 1 + O(pn^−1/2) with probability 1 − O(p^−c).

Proof Let E = n⁻¹X^TX − I_p. Then, ‖E‖₂ ≤ C(p/n)^1/2 for some constant C with probability 1 − O(p^−c) by Theorem 4.6.1 in Vershynin (2016). Furthermore, by matrix inversion, we get

$${\left(n^{-1}{X}^{T}X\right)}^{-1}={\left(I_p+E\right)}^{-1}=I_p-\sum _{k=1}^{\infty }{\left(-1\right)}^{k+1}{\left(E\right)}^k.$$

Now, we take the norm and use triangle inequalities to get

$$\begin{gathered} {\Vert {\left(n^{-1}{X}^{T}X\right)}^{-1}\Vert }_{\infty }\le {\Vert I_p\Vert }_{\infty }+\sum _{k=1}^{\infty }{\Vert E^k\Vert }_{\infty }\le 1+p^{1/2}\sum _{k=1}^{\infty }{\Vert E^k\Vert }_2 \\ \le 1+p^{1/2}\sum _{k=1}^{\infty }\Vert E{\Vert }_2^k\le 1+C{p}^{1/2}\sum _{k=1}^{\infty }{\left({\left(p/n\right)}^{1/2}\right)}^k \\ \le 1+C{p}^{1/2}{\left(p/n\right)}^{1/2} \end{gathered}$$

where we use the fact that p/n is bounded by a constant less than 1. ■

Lemma 7

In the same setting as Lemma 6, if E = n⁻¹X^TX−I_p then ${\Vert {\sum }_{k=2}^{\infty }{\left(-1\right)}^{k+1}{\left(E\right)}^k\Vert }_{\infty }\le C{p}^{3/2}{n}^{-1}$, with probability 1 − O(p^−c).

Proof Again, we use that ‖E‖₂ ≤ C(p/n)^1/2 for some constant C with probability 1 − O(p^−c). By similar calculations as in Lemma 6, we deduce

$$\begin{gathered} {\Vert \sum _{k=2}^{\infty }{\left(-1\right)}^{k+1}{\left(E\right)}^k\Vert }_{\infty }\le \sum _{k=2}^{\infty }{\Vert {\left(-1\right)}^{k+1}{\left(E\right)}^k\Vert }_{\infty }\le \sum _{k=2}^{\infty }{p}^{1/2}{\Vert {\left(E\right)}^k\Vert }_2 \\ =\sum _{k=2}^{\infty }{p}^{1/2}\Vert E{\Vert }_2^k\le \sum _{k=2}^{\infty }{p}^{1/2}{\left(p/n\right)}^{k/2}\le C{p}^{3/2}{n}^{-1}. \end{gathered}$$

■

Lemma 8

Let W_j be nonnegative random variables for 1 ≤ j ≤ p that are not necessarily independent. If P (W_j > t) ≤ C₁ exp(−C₂a_nt²) for some constants C₁ and C₂ and for some sequence a_n, then for any c > 0, ${\text{max}}_{1\le j\le p}{W}_j\le {\left((c+1)/C_2\right)}^{1/2}{a}_n^{-1/2}{\left(\text{log}p\right)}^{1/2}$ with probability at least 1 − O(p^−c).

Proof Using union bound, we get

$$P\left(\underset{1\le j\le p}{\text{max}}{W}_j>t\right)\le \sum _{1\le j\le p}P\left(W_j>t\right)\le p{C}_1\text{exp}\left(-C_2{a}_n{t}^2\right).$$

Taking $t=a_n^{-1/2}{\left(\text{log}p\right)}^{1/2}{\left((c+1)/C_2\right)}^{1/2}$ concludes the proof since then

$$P\left(\underset{1\le j\le p}{\text{max}}{W}_j>a_n^{-1/2}{\left(\text{log}p\right)}^{1/2}{\left((c+1)/C_2\right)}^{1/2}\right)\le C_1{p}^{-c}.$$

■

Lemma 9

Let W_ij be random variables which are independent over the index i. Assume that there are constants C₁ and C₂ such that $P\left(\left|W_{ij}\right|>t\right)\le C_1\text{exp}\left(-C_2{t}^{\alpha }\right)$ with 0 < α ≤ 1. Then, with probability 1 − O(p^−c),

$$\underset{0\le j\le p}{\text{max}}\left|n^{-1}\sum _{i=1}^n\left(W_{ij}-E{W}_{ij}\right)\right|\le C{n}^{-(1/2)\alpha }{(\text{log}p)}^{1/2},$$

for some positive constants c and C.

Proof We have $P\left(\left|n^{-1}{\sum }_{i=1}^n\left(W_{ij}-E{W}_{ij}\right)\right|>t\right)\le C_3\text{exp}\left(-C_4{n}^{\alpha }{t}^2\right)$ by Lemma 6 of Fan et al. (2016) where C₃ and C₄ are some positive constants which only depend on C₁ and C₂. This probability bound shows that the assumption of Lemma 8 holds with a_n = n^α. Thus, Lemma 8 finishes the proof. ■

Lemma 10

With probability 1 − O(p^−c), the vector r defined in (27) satisfies the bound ${\Vert n^{-1}r\Vert }_{\infty }=O\left(n^{\alpha -5/6}\text{log}n\sqrt{n^{-1}\text{log}n}\right)$.

Proof We begin by observing that both x_ij and $\left(x_i^T\mathit{\beta }/\Vert \mathit{\beta }{\Vert }_2\right)$ are standard normal variables. So, using Lemma 1 of Fan et al. (2016), we have $P\left(x_{ij}{\left(x_i^T\mathit{\beta }/\Vert \mathit{\beta }{\Vert }_2\right)}^2>t\right)\le C\text{exp}\left(-C{t}^{2/3}\right)$ for some constant C which does not depend β. It is easy to see that $x_{ij}{\left(x_i^T\mathit{\beta }\right)}^2$ are independent random variables across i’s with mean 0. By Lemma 9, ${\text{max}}_{1\le j\le p}\left|n^{-1}{\sum }_{i=1}^n{x}_{ij}{\left(\frac{x_i^T\mathit{\beta }}{\Vert \mathit{\beta }{\Vert }_2}\right)}^2\right|$ is of order O(n^−1/3(log p)^1/2). Moreover, $\Vert \mathit{\beta }{\Vert }_2\le p^{1/2}\Vert \mathit{\beta }{\Vert }_{\infty }\le O\left(p^{1/2}\sqrt{n^{-1}\text{log}n}\right)$ when $\mathit{\beta }\in {\mathcal{N}}_0$. Therefore,

$$\begin{gathered} {\Vert n^{-1}r\Vert }_{\infty }=\underset{1\le j\le p}{\text{max}}\left|\frac{b^{(3)}(0)}{2}\Vert \mathit{\beta }{\Vert }_2^2{n}^{-1}\sum _{i=1}^n{x}_{ij}{\left(x_i^T\mathit{\beta }/\Vert \mathit{\beta }{\Vert }_2\right)}^2\right| \\ \le Cp{n}^{-1}\left(\text{log}n\right)n^{-1/3}{\left(\text{log}p\right)}^{1/2}=C{n}^{\alpha -4/3}{\left(\text{log}n\right)}^{3/2} \\ =O\left(n^{\alpha -5/6}\left(\text{log}n\right)\sqrt{n^{-1}\text{log}n}\right), \end{gathered}$$

since p = O(n^α). ■

Lemma 11

With probability 1 − O(p^−c), the vector s defined in (28) satisfies the bound ${\Vert n^{-1}s\Vert }_{\infty }=O\left(\left(n^{3/2\alpha -5/4}{\left(\text{log}n\right)}^{3/2}\right)+\left(n^{\alpha -1}{\left(\text{log}n\right)}^{1/2}\right)\sqrt{n^{-1}\text{log}n}\right)$.

Proof First, observe that for some constant C, $\left|n^{-1}{s}_j\right|\le C\Vert \beta {\Vert }_2^3{n}^{-1}{\sum }_{i=1}^n{x}_{ij}{\left(\frac{x_i^T\mathit{\beta }}{\Vert \mathit{\beta }{\Vert }_2}\right)}^3$. Moreover, the summands $x_{ij}{\left(\frac{x_i^T\mathit{\beta }}{\Vert \mathit{\beta }{\Vert }_2}\right)}^3$ are independent over i and they satisfy the probability bound $P\left(\left|x_{ij}{\left(\frac{x_i^T\mathit{\beta }}{\Vert \mathit{\beta }{\Vert }_2}\right)}^3\right|>t\right)\le C\text{exp}\left(-C{t}^{1/2}\right)$ by Lemma 1 of Fan et al. (2016). Thus, by Lemma 9, we obtain

$$\underset{1\le j\le p}{\text{max}}\left|n^{-1}\sum _{i=1}^n\left(x_{ij}{\left(\frac{x_i^T\mathit{\beta }}{\Vert \mathit{\beta }{\Vert }_2}\right)}^3-E\left[{\left(x_{ij}\frac{x_i^T\mathit{\beta }}{\Vert \mathit{\beta }{\Vert }_2}\right)}^3\right]\right)\right|=O\left(n^{-1/4}{\left(\text{log}p\right)}^{1/2}\right).$$

Now, we calculate the expected value of the summand $x_{ij}{\left(\frac{X_i^T\mathit{\beta }}{\Vert \mathit{\beta }{\Vert }_2}\right)}^3$. We decompose $X_i^T\beta$ as $x_{ij}{\beta }_j+x_{i,-j}^T{\mathit{\beta }}_{-j}$ where x_i,−j and β_−j are the vectors x_i and β whose jth entry is removed. We use the independence of x_i,−j and x_ij and get

$$\begin{gathered} E\left[x_{ij}{\left(\frac{x_i^T\mathit{\beta }}{\Vert \mathit{\beta }{\Vert }_2}\right)}^3\right]=\frac{1}{\Vert \mathit{\beta }{\Vert }_2^3}E\left[x_{ij}{\left(x_{ij}{\beta }_j+x_{i,-j}^T{\mathit{\beta }}_{-j}\right)}^3\right] \\ =\frac{1}{\Vert \mathit{\beta }{\Vert }_2^3}E\left[x_{ij}^4{\beta }_j^3+3x_{ij}^3{\beta }_j^2\left(x_{i,-j}^T{\mathit{\beta }}_{-j}\right)+3x_{ij}^2{\beta }_j{\left(x_{i,-j}^T{\mathit{\beta }}_{-j}\right)}^2+x_{ij}{\left(x_{i,-j}^T{\mathit{\beta }}_{-j}\right)}^3\right] \\ =\frac{1}{\Vert \mathit{\beta }{\Vert }_2^3}\left[3{\beta }_j^3+3{\beta }_j{\Vert {\mathit{\beta }}_{-j}\Vert }_2^2\right] \\ =\frac{3{\beta }_j}{\Vert \beta {\Vert }_2}. \end{gathered}$$

Finally, we can combine the result of Lemma 9 and the expected value of $x_{ij}{\left(\frac{x_i^T\mathit{\beta }}{\Vert \mathit{\beta }{\Vert }_2}\right)}^3$. we bound ‖n⁻¹s‖_∞ as follows

$$\begin{gathered} {\Vert n^{-1}s\Vert }_{\infty }\le C\Vert \beta {\Vert }_2^3\underset{1\le j\le p}{\text{max}}\left|n^{-1}\sum _{i=1}^n{x}_{ij}{\left(\frac{{\text{x}}_i^T\beta }{\Vert \beta {\Vert }_2}\right)}^3\right| \\ \le O\left(\Vert \mathit{\beta }{\Vert }_2^3\left(n^{-1/4}{\left(\text{log}p\right)}^{1/2}+\frac{\Vert \mathit{\beta }{\Vert }_{\infty }}{\Vert \mathit{\beta }{\Vert }_2}\right)\right) \\ \le O\left(\Vert \mathit{\beta }{\Vert }_2^3{n}^{-1/4}{\left(\text{log}p\right)}^{1/2}+\Vert \mathit{\beta }{\Vert }_{\infty }\Vert \mathit{\beta }{\Vert }_2^2\right)). \end{gathered}$$

Since $\mathit{\beta }\in {\mathcal{N}}_0$, we have ‖β‖₂ = O(p^1/2n^−1/2(log p)^1/2) and ‖β‖_∞ = O(n^−1/2(log p)^1/2). Thus, ${\Vert n^{-1}s\Vert }_{\infty }=O(\left(n^{3/2\alpha -5/4}{\left(\text{log}n\right)}^{3/2}+\left(n^{\alpha -1}{\left(\text{log}n\right)}^{1/2}\right)\sqrt{n^{-1}\text{log}n}\right)$ when p = O(n^α). ■

Lemma 12

With probability 1 − O(p^−c), the vector t defined in (29) satisfies the bound ${\Vert n^{-1}t\Vert }_{\infty }=O\left(n^{2\alpha -3/2}{\left(\text{log}n\right)}^{3/2}\sqrt{n^{-1}\text{log}n}\right)$.

Proof The proof is similar to the proof of Lemma 11. Since b⁽⁵⁾(·) is uniformly bounded, $\left|n^{-1}{t}_j\right|\le C\Vert \mathit{\beta }{\Vert }_2^4{n}^{-1}{\sum }_{i=1}^n\left|x_{ij}{\left(\frac{x_i^T\mathit{\beta }}{\Vert \mathit{\beta }{\Vert }_2}\right)}^4\right|$ for some constant C. We focus on the summands $\left|x_{ij}{\left(\frac{x_i^T\mathit{\beta }}{\Vert \mathit{\beta }{\Vert }_2}\right)}^4\right|$ which are independent across i. Moreover, repeated application of Lemma 1 of Fan et al. (2016) yields $P\left(x_{ij}{\left(x_i^T\mathit{\beta }/\Vert \mathit{\beta }{\Vert }_2\right)}^2>t\right)\le C\text{exp}\left(-C{t}^{2/5}\right)$ for some constant C independent of β. We can bound the expected value of the summand by Cauchy-Schwartz: $E\left[\left|x_{ij}{\left(\frac{x_i^T\mathit{\beta }}{\Vert \mathit{\beta }{\Vert }_2}\right)}^4\right|\right]\le {\left(E{x}_{ij}^{2}E\left[{\left(\frac{x_i^T\mathit{\beta }}{\Vert \mathit{\beta }{\Vert }_2}\right)}^8\right]\right)}^{1/2}=\sqrt{105}$. So, by Lemma 9, we get

$$\begin{gathered} {\Vert n^{-1}t\Vert }_{\infty }\le C\Vert \mathit{\beta }{\Vert }_2^4\left(\sqrt{105}+n^{-1/5}{(\text{log}p)}^{1/2}\right) \\ =O\left(\Vert \mathit{\beta }{\Vert }_2^4\right)=O\left(p^2{n}^{-2}{(\text{log}n)}^2\right). \end{gathered}$$

Finally, we can deduce that ${\Vert n^{-1}t\Vert }_{\infty }=O\left(n^{2\alpha -3/2}{\left(\text{log}n\right)}^{3/2}\sqrt{n^{-1}\text{log}n}\right)$ when p = O(n^α). ■

Lemma 13

Let $T=e_j^T{\left(X^{T}X\right)}^{-1}{X}^T\left[y-{\mu }_0\right]-n^{-1}{e}_j^T{X}^T\left[y-{\mu }_0\right]$. Under the assumptions of Theorem 4, we have

$$e_j^T{\left(X^{T}X\right)}^{-1}{X}^T\left[y-{\mu }_0\right]-n^{-1}{e}_j^T{X}^T\left[y-{\mu }_0\right]=o_p\left(n^{-1/2}\right).$$ (41)

Proof Since X and y are independent, expectation of T is clearly zero. Then, we consider the variance of T. To this end, we condition on X. We can calculate the conditional variance of T as follows

$$\begin{gathered} \text{var}[T|X]=\text{var}\left[e_j^T\left({\left(X^{T}X\right)}^{-1}-n^{-1}{I}_p\right)X^T\left(y-{\mu }_0\right)|X\right] \\ =\varphi b^{″}(0)e_j^T\left({\left(X^{T}X\right)}^{-1}-n^{-1}{I}_p\right)X^{T}X\left({\left(X^{T}X\right)}^{-1}-n^{-1}{I}_p\right)e_j \end{gathered}$$

where we use var[y] = ϕb″ (0)I_n. When we define E = n⁻¹X^TX − I_p, simple calculations show that

$$\begin{gathered} \text{Var}[T|X]=\varphi b^{″}(0)n^{-1}{e}_j^T\left({\left(n^{-1}{X}^{T}X\right)}^{-1}-I_p\right)+\left(n^{-1}{X}^{T}X-I_p\right))e_j \\ =\varphi b^{″}(0)n^{-1}{e}_j^T\left(\sum _{k=2}^{\infty }{\left(-1\right)}^k{E}^k\right)e_j. \end{gathered}$$

Now, we can obtain the unconditional variance using the law of total variance.

$$\begin{gathered} \text{var}[T]=E[\text{var}[T|X]]+\text{var}[E[T|X]] \\ =\varphi b^{″}(0)n^{-1}{e}_j^{T}E\left(\sum _{k=2}^{\infty }{\left(-1\right)}^k{E}^k\right)e_j. \end{gathered}$$

Thus, using Lemma 7, we can show that var[T] = o(n⁻¹). Finally, we use Chebyshev’s inequality P(|T| > n^−1/2) ≤ nvar[T] = o(1). So, we conclude that T = o (n^−1/2) ■

Lemma 14

Let x_ij be standard normal random variables for 1 ≤ i ≤ n and 1 ≤ j ≤ p. Then, ${\text{max}}_{1\le j\le p}{\sum }_{i=1}^n{x}_{ij}^2\le n+O\left(n^{1/2}{\left(\text{log}p\right)}^{1/2}\right)$ with probability 1 − O(p^−c) for some positive constant c. Consequently, when log p = O(n^α) for some 0 < α ≤ 1, we have ${\text{max}}_{1\le j\le p}{\sum }_{i=1}^n{x}_{ij}^2=O(n)$, for large enough n with probability 1 – O(p^−c).

Proof Since x_ij is a standard normal variable, $x_{ij}^2$ is subexponential random variable whose mean is 1. So, Lemma 9 entails that

$$\underset{1\le j\le p}{\text{max}}\left|n^{-1}\sum _{i=1}^n\left(x_{ij}^2-1\right)\right|=O\left(n^{-1/2}{(\text{log}p)}^{1/2}\right)$$

with probability 1 − O(p^−c). Thus, simple calculations yields

$$\underset{1\le j\le p}{\text{max}}\sum _{i=1}^n{x}_{ij}^2=\underset{1\le j\le p}{\text{max}}\left|n+\sum _{i=1}^n\left(x_{ij}^2-1\right)\right|\le n+O\left(n^{1/2}{\left(\text{log}p\right)}^{1/2}\right)$$

with probability 1 − O(p^−c). ■

B.3. Proof of Theorem 2

To prove the conclusion in Theorem 2, we use the proof by contradiction. Let us make an assumption (A) that the asymptotic normality (11) in Theorem 1 which has been proved to hold when p = o(n^1/2) continues to hold when p ~ n^α0 for some constant 1/2 < α₀ ≤ 1, where ~ stands for asymptotic order. As shown in Section 3.1, in the case of logistic regression under global null (that is, β₀ = 0) with deterministic rescaled orthonormal design matrix X (in the sense of n⁻¹X^TX = I_p) the limiting distribution in (11) by assumption (A) becomes

$$2^{-1}{n}^{1/2}{\widehat{\beta }}_j\stackrel{\mathcal{D}}{\to }N(0,1),$$ (42)

where $\widehat{\mathit{\beta }}={\left({\widehat{\beta }}_1,\cdots ,{\widehat{\beta }}_p\right)}^T$ is the MLE.

Let us now assume that the rescaled random design matrix n^−1/2X is uniformly distributed on the Stiefel manifold $V_p\left({ℝ}^n\right)$ which can be thought of as the space of all n × p orthonormal matrices. Then it follows from (42) that

$$2^{-1}{n}^{1/2}{\widehat{\beta }}_j\stackrel{\mathcal{D}}{\to }N(0,1)\text{conditional}\text{on}X.$$ (43)

Based on the limiting distribution in (43), we can make two observations. First, it holds that

$$2^{-1}{n}^{1/2}{\widehat{\beta }}_j\stackrel{\mathcal{D}}{\to }N\left(0,1\right)$$ (44)

unconditional on the design matrix X. Second, ${\widehat{\beta }}_j$ is asymptotically independent of the design matrix X, and so is the MLE $\widehat{\mathit{\beta }}$.

Since the distribution of n^−1/2X is assumed to be the Haar measure on the Stiefel manifold $V_p\left({ℝ}^n\right)$, we have

$$n^{-1/2}XQ\stackrel{d}{=}{n}^{-1/2}X,$$ (45)

where Q is any fixed p × p orthogonal matrix and $\stackrel{d}{=}$ stands for equal in distribution. Recall that the MLE $\widehat{\mathit{\beta }}$ solves the score equation (4), which is in turn equivalent to equation

$$Q^T{X}^T[y-\mu (X\mathit{\beta })]=0$$ (46)

since Q is orthogonal. We now use the fact that the model is under global null which entails that the response vector y is independent of the design matrix X. Combining this fact with (45)–(46) yields

$$Q^T\widehat{\mathit{\beta }}\stackrel{d}{=}\widehat{\mathit{\beta }}$$ (47)

by noting that Xβ = (XQ)(Q^T β). Since the distributional identity (47) holds for any fixed p × p orthogonal matrix Q, we conclude that the MLE $\widehat{\mathit{\beta }}$ has a spherical distribution on ${ℝ}^p$. It is a well-known fact that all the marginal characteristic functions of a spherical distribution have the same generator. Such a fact along with (44) entails that

$$2^{-1}{n}^{1/2}\widehat{\mathit{\beta }}\text{is}\text{asymptotically}\text{close}\text{to}N\left(0,I_p\right).$$ (48)

To simplify the exposition, let us now make the asymptotic limit exact and assume that

$$\widehat{\mathit{\beta }}~N\left(0,4n^{-1}{I}_p\right)\text{and}\text{is}\text{independent}\text{of}X.$$ (49)

The remaining analysis focuses on the score equation (4) which is solved exactly by the MLE $\widehat{\mathit{\beta }}$, that is,

$$X^T[y-\mu (X\widehat{\mathit{\beta }})]=0,$$ (50)

which leads to

$$\xi \equiv n^{-1/2}{X}^T[y-\mu (0)]=n^{-1/2}{X}^T[\mu (X\widehat{\mathit{\beta }})-\mu (0)]\equiv \eta .$$ (51)

Let us first consider the random variable ξ defined in (51). Note that 2[y − μ(0)] has independent and identically distributed (i.i.d.) components each taking value 1 or −1 with equal probability 1/2, and is independent of X. Thus since n^−1/2X is uniformly distributed on the Stiefel manifold $V_p\left({ℝ}^n\right)$, it is easy to see that

$$\xi =n^{-1/2}{X}^T[y-\mu (0)]\stackrel{d}{=}{2}^{-1}{n}^{-1/2}{X}^{T}1,$$ (52)

where $1\in {ℝ}^n$ is a vector with all components being one. Using similar arguments as before, we can show that ξ has a spherical distribution on ${ℝ}^p$. Thus the joint distribution of ξ is determined completely by the marginal distribution of ξ. For each 1 ≤ j ≤ p, denote by ξ_j the jth component of ξ = 2⁻¹n^−1/2X^T1 using the distributional representation in (52). Let X = (x₁, ⋯ , x_p) with each $x_j\in {ℝ}^n$. Then we have

$${\xi }_j=2^{-1}{n}^{-1/2}{x}_j^{T}1\stackrel{d}{=}{2}^{-1}\left(n^{1/2}/{\Vert {\tilde{x}}_j\Vert }_2\right)n^{-1/2}{\tilde{x}}_j^{T}1,$$ (53)

where ${\tilde{x}}_j~N\left(0,4^{-1}{I}_n\right)$. It follows from (53) and the concentration phenomenon of Gaussian measures that each ξ_j is asymptotically close to N(0, 4⁻¹) and thus consequently ξ is asymptotically close to N(0, 4⁻¹ I_p). A key fact (i) for the finite-sample distribution of ξ is that the standard deviation of each component ξ_j converges to 1/2 at rate Op(n^−1/2) that does not depend upon the dimensionality p at all.

We now turn our attention to the second term η defined in (51). In view of (49) and the fact that n^−1/2X is uniformly distributed on the Stiefel manifold $V_p\left({ℝ}^n\right)$, we can show that with significant probability,

$${\Vert X\widehat{\mathit{\beta }}\Vert }_{\infty }\le o(1)$$ (54)

for $p~n^{{\alpha }_0}$ with α₀ < 1. The uniform bound in (54) enables us to apply the mean value theorem for the vector-valued function η around β₀ = 0, which results in

$$\begin{gathered} \eta =n^{-1/2}{X}^T[\mu (X\widehat{\mathit{\beta }})-\mu (0)]=4^{-1}{n}^{-1/2}{X}^{T}X\widehat{\mathit{\beta }}+r \\ =4^{-1}{n}^{1/2}\widehat{\mathit{\beta }}+r \end{gathered}$$ (55)

since n^−1/2X is assumed to be orthonormal, where

$$r=n^{-1/2}{X}^T\left\{{\int }_0^1\left[\Sigma (tX\widehat{\mathit{\beta }})-4^{-1}{I}_n\right]dt\right\}X\widehat{\mathit{\beta }}.$$ (56)

Here, the remainder term $r={\left(r_1,\cdots ,r_p\right)}^T\in {ℝ}^p$ is stochastic and each component r_j is generally of order O_p{p^1/2n^−1/2} in light of (49) when the true model may deviate from the global null case of β₀ = 0.

Since our focus in this theorem is the logistic regression model under the global null, we can in fact claim that each component r_j is generally of order O_p{pn⁻¹}, which is a better rate of convergence than the one mentioned above thanks to the assumption of β₀ = 0. To prove this claim, note that the variance function b″(·) is symmetric in $\theta \in ℝ$ and takes the maximum value 1/4 at θ = 0. Thus in view of (54), we can show that with significant probability,

$$4^{-1}{I}_n-\Sigma (tX\widehat{\mathit{\beta }})\ge c\text{diag}\{(tX\widehat{\mathit{\beta }})\circ (tX\widehat{\mathit{\beta }})\}=c{t}^2\text{diag}\{(X\widehat{\mathit{\beta }})\circ (X\widehat{\mathit{\beta }})\}$$ (57)

for all t ∈ [0, 1], where c > 0 is some constant and ≥ stands for the inequality for positive semidefinite matrices. Moreover, it follows from (49) and the fact that n^−1/2X is uniformly distributed on the Stiefel manifold $V_p\left({ℝ}^n\right)$ that with significant probability, all the n components of $X\widehat{\mathit{\beta }}$ are concentrated in the order of p^1/2n^−1/2. This result along with (57) and the fact that n⁻¹X^TX = I_p entails that with significant probability,

$$\begin{gathered} n^{-1/2}{X}^T\left\{{\int }_0^1\left[4^{-1}{I}_n-\Sigma (tX\widehat{\mathit{\beta }})\right]dt\right\}X \\ \ge n^{-1/2}{X}^T\left\{{\int }_0^1{c}_{*}{t}^{2}p{n}^{-1}dt\right\}X \\ =3^{-1}{c}_{*}p{n}^{-3/2}{X}^{T}X=3^{-1}{c}_{*}p{n}^{-1/2}{I}_p, \end{gathered}$$ (58)

where c_* > 0 is some constant. Thus combining (56), (58), and (49) proves the above claim.

We make two important observations about the remainder term r in (55). First, r has a spherical distribution on ${ℝ}^p$. This is because by (55) and (51) it holds that

$$r=\mathit{\eta }-4^{-1}{n}^{1/2}\widehat{\mathit{\beta }}=\xi -4^{-1}{n}^{1/2}\widehat{\mathit{\beta }},$$

which has a spherical distribution on ${ℝ}^p$. Thus the joint distribution of r is determined completely by the marginal distribution of r. Second, for the nonlinear setting of logistic regression model, the appearance of the remainder term r in (55) is due solely to the nonlinearity of the mean function μ(·), and we have shown that each component r_j can indeed achieve the worst-case order pn⁻¹ in probability. For each 1 ≤ j ≤ p, denote by η_j the jth component of η. Then in view of (49) and (55), a key fact (ii) for the finite-sample distribution of η is that the standard deviation of each component η_j converges to 1/2 at rate O_P{pn⁻¹} that generally does depend upon the dimensionality p.

Finally, we are ready to compare the two random variables ξ and η on the two sides of equation (51). Since equation (51) is a distributional identity in ${ℝ}^p$, naturally the square root of the sum of varξ_j’s and the square root of the sum of varη_j’s are expected to converge to the common value 2⁻¹p^1/2 at rates that are asymptotically negligible. However, the former has rate p^1/2O_P(n^−1/2) = O_p{p^1/2n^−1/2}, whereas the latter has rate p^1/2O_P{pn⁻¹} = OP{p^3/2n⁻¹}. A key consequence is that when $p~n^{{\alpha }_0}$ for some constant the former rate is $O_P\left\{n^{-\left(1-{\alpha }_0\right)/2}\right\}=o_P(1)$, while the latter rate becomes $O_P\left\{n^{3{\alpha }_0/2-1}\right\}$ which is now asymptotically diverging or nonvanishing. Such an intrinsic asymptotic difference is, however, prohibited by the distributional identity (51) in ${ℝ}^p$, which results in a contradiction. Therefore, we have now argued that assumption (A) we started with for 2/3 ≤ α₀ < 1 must be false, that is, the asymptotic normality (11) which has been proved to hold when p = o(n^1/2) generally would not continue to hold when $p~n^{{\alpha }_0}$ with constant 2/3 ≤ α₀ ≤ 1. In other words, we have proved the invalidity of the conventional GLM p-values in this regime of diverging dimensionality, which concludes the proof of Theorem 2.

B.4. Proof of Theorem 3

By assumption, X ~ N(0, I_n ⊗ Σ) with covariance matrix Σ nonsingular. Let us first make a useful observation. For the general case of nonsingular covariance matrix Σ, we can introduce a change of variable by letting $\tilde{\mathit{\beta }}={\Sigma }^{1/2}\mathit{\beta }$ and correspondingly $\tilde{X}=X{\Sigma }^{-1/2}$. Clearly, $\tilde{X}~N\left(0,I_n\otimes I_p\right)$ and the MLE for the transformed parameter vector $\tilde{\mathit{\beta }}$ is exactly ${\Sigma }^{1/2}\widehat{\mathit{\beta }}$, where $\widehat{\mathit{\beta }}$ denotes the MLE under the original design matrix X. Thus to show the breakdown point of the conventional asymptotic normality of the MLE, it suffices to focus on the specific case of X ~ N(0, I_n ⊗ I_p).

Hereafter we assume that X ~ N(0, I_n ⊗ I_p) with p = o(n). The rest of the arguments are similar to those in the proof of Theorem 2 in Section B.3 except for some modifications needed for the case of Gaussian design. Specifically, for the case of logistic regression model under global null (that is, β₀ = 0), the limiting distribution in (11) becomes

$$2^{-1}{n}^{1/2}{\widehat{\beta }}_j\stackrel{\mathcal{D}}{\to }N(0,1),$$ (59)

since n⁻¹X^TX → I_p almost surely in spectrum and thus $4^{-1}n{\left(A_n^{-1}\right)}_{jj}\to 1$ in probability as n → ∞. Here, we have used a claim that both the largest and smallest eigenvalues of n⁻¹X^TX converge to 1 almost surely as n → ∞ for the case of p = o(n), which can be shown by using the classical results from random matrix theory (RMT) Geman (1980); Silverstein (1985); Bai (1999).

Note that since X ~ N(0, I_n ⊗ I_p), it holds that

$$n^{-1/2}XQ\stackrel{d}{=}{n}^{-1/2}X,$$ (60)

where Q is any fixed p × p orthogonal matrix and $\stackrel{d}{=}$ stands for equal in distribution. By X ~ N(0, I_n ⊗ I_p), it is also easy to see that

$$\xi =n^{-1/2}{X}^T[y-\mu (0)]\stackrel{d}{=}{2}^{-1}{n}^{-1/2}{X}^{T}1,$$ (61)

where $1\in {ℝ}^n$ is a vector with all components being one. In view of (49) and the assumption of X ~ N(0, I_n ⊗ I_p), we can show that with significant probability,

$${\Vert X\widehat{\mathit{\beta }}\Vert }_{\infty }\le o\left(1\right)$$ (62)

for $p~n^{{\alpha }_0}$ with constant α₀ < 1. It holds further that with significant probability, all the n components of $X\widehat{\mathit{\beta }}$ are concentrated in the order of p^1/2n^−1/2. This result along with (57) and the fact that n⁻¹X^TX → I_p almost surely in spectrum entails that with asymptotic probability one,

$$\begin{gathered} n^{-1/2}{X}^T\left\{{\int }_0^1\left[4^{-1}{I}_n-\Sigma (tX\widehat{\mathit{\beta }})\right]dt\right\}X \\ \ge n^{-1/2}{X}^T\left\{{\int }_0^1{c}_{*}{t}^{2}p{n}^{-1}dt\right\}X \\ =3^{-1}{c}_{*}p{n}^{-3/2}{X}^{T}X\to 3^{-1}{c}_{*}p{n}^{-1/2}{I}_p, \end{gathered}$$ (63)

where c_* > 0 is some constant. This completes the proof of Theorem 3.