IPAD: Stable Interpretable Forecasting with Knockoffs Inference

Abstract

Interpretability and stability are two important features that are desired in many contemporary big data applications arising in statistics, economics, and finance. While the former is enjoyed to some extent by many existing forecasting approaches, the latter in the sense of controlling the fraction of wrongly discovered features which can enhance greatly the interpretability is still largely underdeveloped. To this end, in this paper we exploit the general framework of model-X knockoffs introduced recently in Candès, Fan, Janson and Lv (2018), which is nonconventional for reproducible large-scale inference in that the framework is completely free of the use of p-values for significance testing, and suggest a new method of intertwined probabilistic factors decoupling (IPAD) for stable interpretable forecasting with knockoffs inference in high-dimensional models. The recipe of the method is constructing the knockoff variables by assuming a latent factor model that is exploited widely in economics and finance for the association structure of covariates. Our method and work are distinct from the existing literature in that we estimate the covariate distribution from data instead of assuming that it is known when constructing the knockoff variables, our procedure does not require any sample splitting, we provide theoretical justifications on the asymptotic false discovery rate control, and the theory for the power analysis is also established. Several simulation examples and the real data analysis further demonstrate that the newly suggested method has appealing finite-sample performance with desired interpretability and stability compared to some popularly used forecasting methods.

1. Introduction

Forecasting is a fundamental problem that arises in statistics, economics, and finance. With the availability of big data, many machine learning algorithms such as the Lasso and random forest can be resorted to for such a purpose by exploring a large pool of potential features. Many of these existing procedures provide a certain measure of feature importance which can then be utilized to judge the relative importance of selected features for the goal of interpretability. Yet the issue of stability in the sense of controlling the fraction of wrongly discovered features is still largely underdeveloped. As argued in [20] in the econometric settings, it is difficult to obtain interpretability and stability simultaneously even in simple Lasso forecasting. A natural question is how to ensure both interpretability and stability for flexible forecasting.

Naturally stability is related to statistical inference. The recent years have witnessed a growing body of work on high-dimensional inference in the statistics and econometrics literature; see, for example, [42], [38], [43], [16], [18], [17], [36], and [29]. Most existing work on high-dimensional inference for interpretable models has focused primarily on the aspects of post-selection inference known as selective inference and debiasing for regularization and machine learning methods. In real applications, one is often interested in conducting global inference relative to the full model as opposed to local inference conditional on the selected model. Moreover, many statistical inferences are based on p-values from significance testing. However, oftentimes obtaining valid p-values even for the Lasso in relatively complicated high-dimensional nonlinear models also remains largely unresolved, not to mention for the case of more complicated model fitting procedures such as random forest. Indeed high-dimensional inference is intrinsically challenging even in the parametric settings [27].

The desired property of stability for interpretable forecasting in this paper concentrates on global inference by controlling precisely the fraction of wrongly discovered features in high-dimensional models, which is also known as reproducible large-scale inference. Such a problem involves testing the joint significance of a large number of features simultaneously, which is known widely as the problem of multiple testing in statistical inference. For this problem, the null hypothesis for each feature states that the feature is unimportant in the joint model which can be understood as the property that this individual feature and the response are independent conditional on all the remaining features, while the corresponding alternative hypothesis states the opposite. Conventionally p-values from the hypothesis testing are used to decide whether or not to reject each null hypothesis with a significance level to control the probability of false discovery in a single hypothesis test, meaning rejecting the null hypothesis when it is true. When performing multiple hyothesis tests, the probability of making at least one false discovery which is known as the family-wise error rate can be inflated compared to that for the case of a single hypothesis test. The work on controlling such an error rate for multiple testing dates back to [13], where a simple, useful idea is lowering the significance level for each individual test as the target level divided by the total number of tests to be performed. The Bonferroni correction procedure is, however, well known to be conservative with relatively low power. Later on, [30] proposed a stepdown procedure which is less conservative than the Bonferroni procedure. More recently, [35] suggested a procedure in which the critical values of individual tests are constructed sequentially.

A more powerful and extremely popular approach to multiple testing is the Benjamini–Hochberg (BH) procedure for controlling the false discovery rate (FDR) which was originated in [9], where the FDR is defined as the expectation of the fraction of falsely rejected null hypotheses known as the false discovery proportion. Given the p-values from the multiple hypothesis tests, this procedure sorts the p-values from low to high and chooses a simple, intuitive cutoff point, which can be viewed as an adaptive extension of the Bonferroni correction for multiple comparisons, of the p-values for rejecting the null hypotheses. The BH procedure was shown to be capable of controlling the FDR at the desired level for independent test statistics in [9] and for positive regression dependency among the test statistics in [10], where it was shown that a simple modification of the procedure can control the FDR under other forms of dependency but such a modification is generally conservative. There is a huge literature on the theory, applications, and various extensions of the original BH procedure for FDR control; see, for instance, [8], [24], [7], and [19].

The aforementioned econometric and statistical inference methods including the BH-type procedures for FDR control are all rooted on the availability and validity of computable p-values for evaluating variable importance. As mentioned before, such a prerequisite can become a luxury that is largely unclear how to obtain in high dimensions even for the case of Lasso in general nonlinear models and random forest. In contrast, [4] proposed a novel procedure named the knockoff filter for FDR control that bypasses the use of p-values in Gaussian linear model with deterministic design matrix, where the dimensionality is no larger than the sample size, and [5] generalized the method to high-dimensional linear models as a two-step procedure based on sample splitting, where a feature screening approach is used to reduce the dimensionality to below sample size (see, e.g., [23] and [25]) and then the knockoff filter is applied to the set of selected features after the screening step for selective inference. The key ingredient of the knockoff filter is constructing the so-called knockoff variables in a geometrical way that mimic perfectly the correlation structure among the original covariates and can be used as control variables to evaluate the importance of original variables. Recently, [15] extended the work of [4] by introducing the framework of model-X knockoffs for FDR control in general high-dimensional nonlinear models. A crucial distinction is that the knockoff variables are constructed in a probabilistic fashion such that the joint dependency structure of the original variables and their knockoff copies is invariant to the swapping of any set of original variables and their knockoff counterparts, which enables us to go beyond linear models and handle high dimensionality. As a result, model-X knockoffs enjoys exact finite-sample FDR control at the target level. However, a major assumption in [15] is that the joint distribution of all the covariates needs to be known for the valid FDR control.

Motivated by applications in economics and finance, in this paper we model the association structure of the covariates using the latent factor model, which reduces effectively the dimensionality and enables reliable estimation of the unknown joint distribution of all the covariates. By taking into account the latent factor model structure, we first estimate the association structure of covariates and then construct empirical knockoffs matrix using the estimated dependency structure. Our empirical knockoffs matrix can be regarded as an approximation to the oracle knockoffs matrix in [15] that requires the knowledge of the true covariate distribution. Exploiting the general framework of model-X knockoffs in [15], we suggest the new method of intertwined probabilistic factors decoupling (IPAD) for stable interpretable forecasting with knockoffs inference in high-dimensional models. The innovations of our method and work are fourfold. First, we estimate the covariate distribution from data instead of assuming that it is known when constructing the knockoff variables. Second, our procedure does not require any sample splitting and is thus more practical when the sample size is limited. Third, we provide theoretical justifications on the asymptotic false discovery rate control when the estimated dependency structure is employed. Fourth, the theory for power analysis is also established which reveals that there can be asymptotically no power loss in applying the knockoffs procedure compared to the underlying variable selection method. Therefore, FDR control by knockoffs can be a pure gain. Compared to earlier work, an additional challenge of our study is that knowing the true underlying distribution does not lead to the most efficient construction of the oracle knockoffs matrix due to the presence of latent factors.

The rest of the paper is organized as follows. Section 2 introduces the model setting and presents the new IPAD procedure. We establish the asymptotic properties of IPAD in Section 3. Sections 4 and 5 present several simulation and real data examples to showcase the finite-sample performance and the advantages of our newly suggested procedure compared to some popularly used ones. We discuss some implications and extensions of our work in Section 6. The proofs of the main results are relegated to the Appendix. Additional technical details and numerical results are provided in the Supplementary Material.

2. Intertwined probabilistic factors decoupling

To facilitate the technical presentation, we will introduce the model setting for the high-dimensional FDR control problem in Section 2.1 and present the new IPAD procedure in Section 2.2.

2.1. Model setting

Consider the high-dimensional linear regression model

$$\mathbf{y}=\mathbf{X}\mathit{\beta }+\mathit{\epsilon },$$ (1)

where y ∈ ℝⁿ is the response vector, X ∈ ℝ^n×p is the random matrix of a large number of potential regressors, β = (β₁, ⋯, β_p)′ ∈ ℝ^p is the regression coefficient vector, ε ∈ ℝⁿ is the vector of model errors, and n and p denote the sample size and dimensionality, respectively. Here without loss of generality, we assume that both the response and the covariates are centered with mean zero and thus there is no intercept. Motivated by many applications in economics and finance, we further assume that the design matrix X follows the exact factor model

$$\mathbf{X}={\mathbf{F}}^0{\mathbf{\Lambda }}^{0^{\prime }}+\mathbf{E}={\mathbf{C}}^0+\mathbf{E},$$ (2)

where ${\mathbf{F}}^0={\left({\mathbf{f}}_1^0,\cdots ,{\mathbf{f}}_n^0\right)}^{\prime }\in {ℝ}^{n\times r}$ is a random matrix of latent factors, ${\mathbf{\Lambda }}^0={\left({\mathit{\lambda }}_1^0,\cdots ,{\mathit{\lambda }}_p^0\right)}^{\prime }\in {ℝ}^{p\times r}$ is a matrix of deterministic factor loadings, and error term E ∈ ℝ^n×p captures the remaining variation that cannot be explained by these latent factors. We assume that the number of factors r is fixed but unknown and the components of E are independent and identically distributed (i.i.d.) from some parametric distribution with cumulative distribution function G(·;η⁰), where η⁰ ∈ ℝ^m is a finite-dimensional unknown parameter vector. For technical simplicity, models (1) and (2) are assumed to have no endogeneity and satisfy that F⁰ has i.i.d. rows and is independent of E.

In this paper, we focus on the high-dimensional scenario when the dimensionality p can be much larger than sample size n. Therefore, to ensure model identifiability we impose the sparsity assumption that the true regression coefficient vector β has only a small portion of nonzeros; specifically, β takes nonzero values only on some (unknown) index set ${\mathcal{S}}^0\subset \{1,\cdots ,p\}$ and β_j = 0 for all $j\in {\mathcal{S}}^1:=\{1,\cdots ,p\}\backslash {\mathcal{S}}^0$. Denote by $s=\left|{\mathcal{S}}^0\right|$ the size of ${\mathcal{S}}^0$. We assume that s = o(n) throughout the paper.

We are interested in identifying the index set ${\mathcal{S}}^0$ with a theoretically guaranteed error rate. To be more precise, we try to select variables in ${\mathcal{S}}^0$ while keeping the false discovery rate (FDR) under some prespecified desired level q ∈ (0,1), where the FDR is defined as

$$\text{FDR}:=\mathbb{E}[\text{FDP}]\text{with}\text{FDP}:=\frac{\left|\widehat{\mathcal{S}}\cap {\mathcal{S}}^1\right|}{\left|\widehat{\mathcal{S}}\right|\vee 1}.$$ (3)

Here the FDP stands for the false discovery proportion and $\widehat{\mathcal{S}}$ represents the set of variables selected by some procedure using observed data (X, y). A slightly modified version of FDR is defined as

$$\text{mFDR}:=\mathbb{E}\left[\frac{\left|\widehat{\mathcal{S}}\cap {\mathcal{S}}^1\right|}{\left|\widehat{\mathcal{S}}\right|+q^{-1}}\right].$$ (4)

Clearly, FDR is more conservative than mFDR in that the latter is always under control if the former is.

It is easy to see that FDR is a measurement of type I error for variable selection. The other important aspect of variable selection is power, which is defined as

$$\text{Power}:=\mathbb{E}\left[\frac{\left|\widehat{\mathcal{S}}\cap {\mathcal{S}}^0\right|}{\left|{\mathcal{S}}^0\right|}\right]=\mathbb{E}\left[\frac{\Vert \widehat{\mathcal{S}}\cap {\mathcal{S}}^0|}{s}\right].$$ (5)

It is well known that FDR and power are two sides of the same coin. We aim at developing a variable selection procedure with theoretically guaranteed FDR control and meanwhile achieving high power.

2.2. IPAD

The key ingredient of the model-X knockoffs framework introduced originally in [15] is the construction of the so-called model-X knockoff variables defined as follows.

Definition 1 (Model-X knockoff variables [15])

For a set of random variables x = (X₁,⋯, X_p), a new set of random variables $\tilde{\mathbf{x}}=\left({\tilde{X}}_1,\cdots ,{\tilde{X}}_p\right)$ is called a set of model-X knockoff variables if it satisfies the following properties:

1. For any subset $\mathcal{S}\subset \{1,\cdots ,p\}$, we have ${[\mathbf{x},\tilde{\mathbf{x}}]}_{\text{swap}(\mathcal{S})}\stackrel{d}{=}[\mathbf{x},\tilde{\mathbf{x}}]$, where $\stackrel{d}{=}$ denotes equal in distribution and the vector ${[\mathbf{x},\tilde{\mathbf{x}}]}_{\text{swap}(\mathcal{S})}$ is obtained by swapping X_j and ${\tilde{X}}_j$ for each $j\in \mathcal{S}$.

2. Conditional on x, the knockoffs vector $\tilde{\mathbf{x}}$ is independent of response Y.

See Section B of Supplementary Material for a brief review of the model-X knockoffs framework. In theory, if the distribution of C₀ and the value of η₀ are known, the SCIP algorithm proposed in [15] can be used to construct the knockoff variables. However, the computational cost can be high depending on the exact distributions. Instead we introduce a more efficient and practically implementable approach for constructing the knockoff variables below.

We start with introducing the knockoff generating function – for each given augmented parameter vector θ = vec(vec C, η), define

$$\tilde{\mathbf{X}}(\mathit{\theta })=\mathbf{C}+{\mathbf{E}}_{\mathit{\eta }},$$ (6)

where E_η is a matrix composed of i.i.d. random samples from distribution G(·;η). To gain some insights, let us first consider the ideal situation where the the factor model structure (2) is fully available; that is, we know the realization C⁰ and the true distribution G(·;η⁰) for the error matrix E. In such case, the oracle (ideal) knockoffs matrix $\tilde{\mathbf{X}}({\mathit{\theta }}^0)$ can be constructed as

$$\tilde{\mathbf{X}}\left({\mathit{\theta }}^0\right)={\mathbf{C}}^0+{\mathbf{E}}_{{\mathit{\eta }}^0},$$ (7)

where ${\mathbf{E}}_{{\mathit{\eta }}^0}$ is an i.i.d. copy of E. Note that ${\mathbf{E}}_{{\mathit{\eta }}^0}$ itself is not a function of η⁰, but we slightly abuse the notation to emphasize the dependence of the distribution function on parameter η⁰. In practice, θ₀ is unknown and needs to be estimated. Letting $\widehat{\mathit{\theta }}$ denote an estimator (obtained using data X) of θ⁰, we name $\tilde{\mathbf{X}}(\widehat{\mathit{\theta }})$ as the empirical knockoffs matrix:

$$\tilde{\mathbf{X}}(\widehat{\mathit{\theta }})=\widehat{\mathbf{C}}+{\mathbf{E}}_{\widehat{\mathit{\eta }}},$$ (8)

where $\widehat{\mathbf{C}}$ is an empirical estimate of C⁰ and ${\mathbf{E}}_{\widehat{\mathit{\eta }}}\in {ℝ}^{n\times p}$ is composed of i.i.d. random variables from the plug-in estimate of the distribution function, $G(\cdot ;\widehat{\mathit{\eta }})$, and is independent of (X, y) conditional on $\widehat{\mathit{\eta }}$. The following proposition justifies the validity of the oracle knockoffs matrix.

Proposition 1

Under model setting (2), the oracle knockoffs matrix defined in (7) satisfies Definition 1.

However, the empirical knockoffs matrix given in (8) generally does not satisfy the exchangeability property because of the dependence of $\widehat{\mathit{\theta }}$ on the training data X. Although the oracle knockoffs matrix is generally unavailable, it plays an important role in our theoretical developments as a proxy of the empirical knockoffs matrix. We remark that in the construction above, we slightly misuse the concept and call C⁰ a parameter. This is because although C⁰ is a random matrix, for the construction of valid knockoff variables it is the particular realization C⁰ leading to the observed data matrix X that matters. In other words, a valid construction of knockoff variables requires the knowledge of the specific realization C⁰ instead of the distribution of C⁰. To understand this, consider the scenario where the underlying parameter η⁰ and the exact distribution of C⁰ are fully available. If we independently generate random variables from this known distribution and form a new data matrix X₁, because of the independence between X₁ and X, the exchangeability assumption in Definition 1 will be violated and thus X₁ cannot be a valid knockoffs matrix. On the other hand, as long as we know the realization C⁰ and parameter η⁰, a valid knockoffs matrix $\tilde{\mathbf{X}}({\mathit{\theta }}^0)$ can be constructed using (7) regardless of whether the exact distribution of C⁰ is available or not.

In practice, however, θ⁰ is unavailable and consequently, $\tilde{\mathbf{X}}({\mathit{\theta }}^0)$ is inaccessible. To over-come this difficulty, we next introduce our new method IPAD. With the aid of empirical knockoffs matrix, we suggest the following IPAD procedure for FDR control with knockoffs inference.

Procedure 1 (IPAD)

1. (Estimation of parameters) Estimate the unknown parameters in θ⁰ using the design matrix X. Denote by $\widehat{\mathit{\theta }}=(\widehat{\mathbf{C}},\widehat{\mathit{\eta }})$ the resulting estimated parameter vector.

2. (Construction of empirical knockoffs matrix) Construct the empirical knockoffs matrix (8) by applying the knockoff generating function in (6) to the estimated parameter $\widehat{\mathit{\theta }}$.

3. (Application of knockoffs inference) Calculate knockoff statistics $W_j(\widehat{\mathit{\theta }})$ using data $([\mathbf{X},\tilde{\mathbf{X}}(\widehat{\mathit{\theta }})],\mathbf{y})$ and then construct $\widehat{\mathcal{S}}$ by applying knockoffs inference to $W_j(\widehat{\mathit{\theta }})$.

Intuitively, the accuracy of the estimator $\widehat{\mathit{\theta }}$ in Step 1 will affect the performance of our IPAD procedure. In fact, as shown later in our Theorem 1 in Section 3, the consistency rate of $\widehat{\mathit{\theta }}$ is indeed reflected in the asymptotic FDR control of the IPAD procedure. There are various ways to construct estimator $\widehat{\mathit{\theta }}$. A natural and popularly used one is the principal component (PC) estimator $(\widehat{\mathbf{F}},\widehat{\mathbf{\Lambda }})$ studied in [2]. Specifically, we first estimate the number of factors r, denoted as $\widehat{r}$, using some method such as the information criterion in [3] or the approach in [1], and then set $\widehat{\mathbf{C}}=\widehat{\mathbf{F}}{\widehat{\mathbf{\Lambda }}}^{\prime }$, where $\widehat{\mathbf{F}}$ is T^1/2 times the eigenvectors corresponding to the top $\widehat{r}$ largest eigenvalues of XX′, and $\widehat{\mathbf{\Lambda }}={\mathbf{X}}^{\prime }\widehat{\mathbf{F}}/T$. As for the estimation of η⁰, existing methods such as the method of moments can be used based on the residual matrix $\widehat{\mathbf{E}}=\left({\widehat{e}}_{ij}\right)$. As a concrete example, consider the case where E has i.i.d. $\mathcal{N}\left(0,{\sigma }^2\right)$ entries. Then the unknown population parameter is η⁰ = σ² and can be estimated naturally as ${(np)}^{-1}{\sum }_{i=1}^n{\sum }_{j=1}^p{\widehat{e}}_{ij}^2$.

In Step 3, various methods can be used to construct knockoff statistics. For the illustration purpose, we use the Lasso coefficient difference (LCD) statistic as in [15]. Specifically, with y the response vector and $([\mathbf{X},\tilde{\mathbf{X}}(\widehat{\mathit{\theta }})])$ the augmented design matrix we consider the variable selection procedure Lasso [39] which solves the following optimization problem

$${\widehat{\mathit{\beta }}}^{\mathrm{aug}}(\widehat{\mathit{\theta }};\lambda )=\arg \underset{\mathbf{b}\in {ℝ}^{2p}}{\min }\left\{\Vert \mathbf{y}-[\mathbf{X},\tilde{\mathbf{X}}(\widehat{\mathit{\theta }})]\mathbf{b}{\Vert }_2^2+\lambda \Vert \mathbf{b}{\Vert }_1\right\},$$ (9)

where λ ≥ 0 is the regularization parameter and ∥ · ∥_m with m ≥ 1 denotes the vector ℓ_m-norm. Then for each variable x_j, the knockoff statistic can be constructed as

$$W_j(\widehat{\mathit{\theta }};\lambda )=\left|{\widehat{\beta }}_j^{\text{aug}}(\widehat{\mathit{\theta }};\lambda )\right|-\left|{\widehat{\beta }}_{p+j}^{\text{aug}}(\widehat{\mathit{\theta }};\lambda )\right|,$$ (10)

where ${\widehat{\beta }}_{\ell }^{\text{aug}}(\widehat{\mathit{\theta }};\lambda )$ is the ℓth component of the Lasso regression coefficient vector ${\widehat{\mathit{\beta }}}^{\text{aug}}(\widehat{\mathit{\theta }};\lambda )$. It is seen that intuitively the LCD knockoff statistics evaluate the relative importance of the jth original variable by comparing its Lasso coefficient ${\widehat{\beta }}_j^{\text{aug}}(\widehat{\mathit{\theta }};\lambda )$ with that of its knockoff copy ${\widehat{\beta }}_{j+p}^{\text{aug}}(\widehat{\mathit{\theta }};\lambda )$. In the ideal case when the oracle knockoffs matrix $\tilde{\mathbf{X}}({\mathit{\theta }}^0)$ is used instead of $\tilde{\mathbf{X}}(\widehat{\mathit{\theta }})$ in (9), it is easy to verify that the LCD is a valid construction of knockoff statistics and satisfies the sign-flip property in (A.2). Consequently, the general theory in [15] can be applied to show that the FDR is controlled in finite sample. We next show that even with the empirical knockoffs matrix employed in (9), the FDR can still be asymptotically controlled with delicate technical analyses.

3. Asymptotic properties of IPAD

We now provide theoretical justifications for our IPAD procedure suggested in Section 2 with the LCD knockoff statistics $W_j(\widehat{\mathit{\theta }};\lambda )=w_j([\mathbf{X},\tilde{\mathbf{X}}(\widehat{\mathit{\theta }})],\mathbf{y};\lambda )$ defined in (10). We will first present some technical conditions in Section 3.1, then prove in Section 3.2 that the FDR is asymptotically under control at desired target level q, and finally in Section 3.3 show that asymptotically IPAD has no power loss compared to the Lasso under some regularity conditions.

3.1. Technical conditions

We first introduce some notation and definitions which will be used later on. We use $X\sim \mathrm{subG}\left(C_x^2\right)$ to denote that X is a sub-Gaussian random variable with variance proxy $C_x^2>0$ if $\mathbb{E}[X]=0$ and its tail probability satisfies $\mathbb{P}(|X|>u)\le 2\exp \left(u^2/C_x^2\right)$ for each u ≥ 0. In all technical assumptions below, we use M > 1 to denote a large enough generic constant. Throughout the paper, for any vector v = (v_i) let us denote by ∥v∥₁, ∥v∥₂, and ∥v∥_max the ℓ₁-norm, ℓ₂-norm, and max-norm defined as $\Vert \mathbf{v}{\Vert }_1={\sum }_i\left|v_i\right|$, $\Vert \mathbf{v}{\Vert }_2={\left({\sum }_i{v}_i^2\right)}^{1/2}$ and ∥v∥_max = max_i |v_i|, respectively. For any matrix M = (m_ij), we denote by ∥M∥_F, ∥M∥₁, ∥M∥₂, and ∥M∥_max the Frobenius norm, entrywise ℓ₁-norm, spectral norm, and entrywise ℓ_∞-norm defined as ∥M∥_F = ∥vec(M)∥₂, ∥M∥₁ = ∥vec(M)∥₁, ∥M∥₂ = sup_v≠0 ∥Mv∥₂/∥v∥₂, and ∥M∥_max = ∥vec(M)∥_max, respectively, where vec(M) represents the vectorization of M. For a symmetric matrix M, vech(M) stands for the vectorization of the lower triangular part.

Condition 1 (Regression errors)

The model error vector ε has i.i.d. components from $\mathrm{subG}\left(C_{\epsilon }^2\right)$.

Condition 2 (Latent factors)

The rows of F⁰ consist of mean zero i.i.d. random vectors ${\mathbf{f}}_i^0\in {ℝ}^r$ such that ∥F⁰∥_max ≤ M almost surely (a.s.) and ${\Vert {\mathbf{\Sigma }}_f\Vert }_2+{\Vert {\mathbf{\Sigma }}_f^{-1}\Vert }_2\le M$, where ${\mathbf{\Sigma }}_f:=\mathbb{E}\left[{\mathbf{f}}_i^0{\mathbf{f}}_i^{0\prime }\right]$.

Condition 3 (Factor loadings)

The rows of Λ⁰ consist of deterministic vectors ${\mathit{\lambda }}_j^{\theta }\in {ℝ}^r$ such that ∥Λ⁰∥_max ≤ M and ${\Vert p^{-1}{\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{\Lambda }}^0\Vert }_2+{\Vert {\left(p^{-1}{\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{\Lambda }}^0\right)}^{-1}\Vert }_2\le M$.

Condition 4 (Factor errors)

The entries of matrix ${\mathbf{E}}_{{\mathit{\eta }}^0}$ are i.i.d. copies of $e_{{\mathit{\eta }}^0}\sim \mathrm{subG}\left(C_e^2\right)$ with continuous distribution function G(·;η⁰). For each 1 ≤ ℓ ≤ m, the ℓth element of η⁰ is specified as ${\eta }_{\ell }^0=h_{\ell }\left(\mathbb{E}\left[e_{{\mathit{\eta }}^0}\right],\cdots ,\mathbb{E}\left[e_{{\mathit{\eta }}^0}^m\right]\right)$ with h_ℓ : ℝ^m → ℝ some local Lipschitz continuous function in the sense that

$$\left|h_{\ell }\left(t_1,\cdots ,t_m\right)-h_{\ell }\left(\mathbb{E}\left[e_{{\mathit{\eta }}^0}\right],\cdots ,\mathbb{E}\left[e_{{\mathit{\eta }}^0}^m\right]\right)\right|\le M\underset{k\in \{1,\cdots ,m\}}{\max }\left|t_k-\mathbb{E}\left[e_{{\mathit{\eta }}^0}^k\right]\right|$$

for each $t_k\in \{t:|t-\mathbb{E}[e_{{\mathit{\eta }}^0}^k]|\le M{c}_{np}\}$ and 1 ≤ k ≤ m, where c_np := (p⁻¹ logn)^1/2 + (n⁻¹ log p)^1/2. Moreover, there exists some stochastic process (e_η)_η such that

1. for each $\mathit{\eta }\in \left\{\mathit{\eta }\in {ℝ}^m:{\Vert \mathit{\eta }-{\mathit{\eta }}^0\Vert }_{\max }\le M{c}_{np}\right\}$, the entries of E_η in (6) have identical distribution to e_η,

2. for some sub-Gaussian random variable $Z\sim \mathrm{subG}\left(c_e^2\right)$ with some positive constant c_e,

$$\underset{\mathit{\eta }:{\Vert \mathit{\eta }-{\mathit{\eta }}^0\Vert }_{\max }\le M{c}_{np}}{\sup }\left|e_{\mathit{\eta }}-e_{{\mathit{\eta }}^0}\right|\le M^{1/2}{c}_{np}^{1/2}|Z|.$$ (11)

Condition 5 (Eigenseparation)

The r eigenvalues of $p^{-1}{\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{\Lambda }}^0{\mathbf{\Sigma }}_f$ are distinct for all p.

The number of factors r is assumed to be known for developing the theory with simplification, but in practice it can be estimated consistently using methods such as information criteria [3] and test statistics [1]. The sub-Gaussian assumptions in Conditions 1 and 4 can be replaced with some other tail conditions as long as similar concentration inequalities hold. Condition 3 is standard in the analysis of factor models. Stochastic loadings can be assumed in Condition 3 with some appropriate distributional assumption, such as sub-Gaussianity, at the cost of much more tedious technical arguments. The boundedness of the eigenvalues of Σ_f in Condition 2 is standard while the i.i.d. assumption and boundedness of ${\mathbf{f}}_i^0$ are stronger compared to the existing literature (e.g., [3] and [2]). However, these conditions are imposed mostly for technical simplicity. In fact, the boundedness condition on ${\mathbf{f}}_i^0$ can be replaced with (unbounded) sub-Gaussian or other heavier-tail assumption whenever concentration inequalities are available at the cost of slower convergence rates and stronger sample size requirement. Our theory on FDR control is based on that in [15], which applies only to the case of i.i.d. rows of design matrix X. This is the main reason for imposing the i.i.d. assumption on ε_i and f_i in Conditions 1 and 2. However, we conjecture that similar results can also hold in the presence of some sufficiently weak serial dependence in ε_i and f_i. Condition 4 introduces a sub-Gaussian process e_η with respect to η. The norm in (11) can be replaced with any other norm since η is finite dimensional. In the specific case when the components of E have Gaussian distribution such that η is a scalar parameter representing variance, by the reflection principle for the Wiener process ([12], p.511), e_η can be constructed as a Wiener process and the inequality (11) can be satisfied. For more information on sub-Gaussian processes, see, e.g., [41]. To understand why we need Condition 5, note from the proof of Lemma 3 that the PC estimator $(\widehat{\mathbf{F}},\widehat{\mathbf{\Lambda }})$ is only consistent for (F⁰H, Λ⁰H⁻¹), where $\mathbf{H}=\left({\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{\Lambda }}^0/p\right)({\mathbf{F}}^{0^{\prime }}\widehat{\mathbf{F}}/n){\mathbf{V}}^{-1}$ with V an r × r diagonal matrix of r largest eigenvalues of XX′/(np). Condition 5 guarantees that ${\widehat{\mathbf{F}}}^{\prime }{\mathbf{F}}^0/n$ is asymptotically unique and invertible, which have been proved by [2], and the fact is used in the proof of Lemma 6 in Appendix. This ultimately ensures that C⁰ can be estimated well, which in turn guarantees that η⁰ can be estimated accurately.

Recall that in the IPAD procedure, we first obtain the augmented Lasso estimator ${\widehat{\mathit{\beta }}}^{\mathrm{aug}}(\theta ;\lambda )\in {ℝ}^{2p}$ by regressing y on $[\mathbf{X},\tilde{\mathbf{X}}(\theta )]$. Denote by ${\mathcal{A}}^{\text{aug}}(\mathit{\theta };\lambda )=\mathrm{supp}({\widehat{\mathit{\beta }}}^{\text{aug}}(\mathit{\theta };\lambda ))\subset \{1,\cdots ,2p\}$ the active set of the augmented Lasso regression coefficient vector. Throughout this section, we content ourselves with sparse estimates satisfying

$$\left|{\mathcal{A}}^{\text{aug}}(\mathit{\theta };\lambda )\right|\le k/2$$ (12)

for some positive integer k which may diverge with n at an order to be specified later; see, e.g., [28] and [32] for a similar constraint and justifications therein. This can always be achieved since users have the freedom to choose the size of the Lasso model.

3.2. FDR control

To develop the theory for IPAD, we consider the PC estimator $\widehat{\mathbf{C}}$ for the realization C⁰ summarized in Section 2.2. The estimator $\widehat{\mathit{\eta }}={\left({\widehat{\eta }}_1,\cdots ,{\widehat{\eta }}_m\right)}^{\prime }$ is constructed as ${\widehat{\eta }}_{\ell }=h_{\ell }({\mathbb{E}}_{np}\widehat{e},\cdots {\mathbb{E}}_{np}{\widehat{e}}^m)$ with h_ℓ, 1 ≤ ℓ ≤ m, introduced in Condition 4 and ${\mathbb{E}}_{np}{\widehat{e}}^k={(np)}^{-1}{\sum }_{1\le i\le n,1\le j\le p}{\widehat{e}}_{ij}^k$ the empirical moments of ${\widehat{e}}_{ij}$. Throughout our theoretical analysis, we consider the regularization parameter fixed at λ = C₀n^−1/2 log p with C₀ some large enough constant for all the Lasso procedures. Therefore, we will drop the dependence of various quantities on λ whenever there is no confusion. For example, we will write ${\mathcal{A}}^{\text{aug}}(\theta ;\lambda )$ and ${\widehat{\mathit{\beta }}}^{\mathrm{aug}}(\theta ;\lambda )$ as ${\mathcal{A}}^{\text{aug}}(\theta )$ and ${\widehat{\mathit{\beta }}}^{\mathrm{aug}}(\theta )$, respectively.

Denote by $\mathbf{U}(\theta ):=n^{-1}{[\mathbf{X},\tilde{\mathbf{X}}(\theta )]}^{\prime }[\mathbf{X},\tilde{\mathbf{X}}(\theta )]$ and $\mathbf{v}(\theta ):=n^{-1}{[\mathbf{X},\tilde{\mathbf{X}}(\theta )]}^{\prime }\mathbf{y}$ and define T(θ) := vec(vech U(θ), v(θ)) ∈ ℝ^P with P := p(2p + 3). The following lemma states that the statistic T(θ) plays a crucial role in our procedure.

Lemma 1

The set of variables $\widehat{\mathcal{S}}$ selected by Procedure 1 depends only on T(θ).

For any given θ, define the active set $\mathcal{A}\ast (\theta ):={\mathcal{A}}_1^{\text{aug}}(\theta )\cup {\mathcal{A}}_2^{\text{aug}}(\theta )\subset \{1,\cdots ,p\}$, where ${\mathcal{A}}_1^{\text{aug}}(\theta ):=\left\{j:j\in \{1,\cdots ,p\}\cap {\mathcal{A}}^{\text{aug}}(\theta )\right\}$ and ${\mathcal{A}}_2^{\text{aug}}(\theta ):=\{j-p:j\in \{p+1,\cdots ,2p\}\cap {\mathcal{A}}^{\mathrm{aug}}(\theta )\}$. That is, $\mathcal{A}\ast (\theta )$ is equal to the support of knockoff statistics (W₁(θ), ⋯, W_p(θ))′ if there are no ties on the magnitudes of the augmented Lasso coefficient vector ${\widehat{\mathit{\beta }}}^{\text{aug}}(\theta )$.

We next focus on the low-dimensional structure of T(θ) inherited from the augmented Lasso because it will be made clear that this is the key to controlling the FDR without sample splitting. For any subset $\mathcal{A}\subset \{1,\cdots ,p\}$, define a lower-dimensional expression of the vector as ${\mathbf{T}}_{\mathcal{A}}(\theta ):=\mathrm{vec}\left(\mathrm{vech}{\mathbf{U}}_{\mathcal{A}}(\theta ),{\mathbf{v}}_{\mathcal{A}}(\theta )\right)$ with ${\mathbf{U}}_{\mathcal{A}}(\theta )$ the principle submatrix of U(θ) formed by columns and rows in $\mathcal{A}$ and ${\mathbf{v}}_{\mathcal{A}}(\theta )$ the subvector of v(θ) formed by components in $\mathcal{A}$. Then it is easy to see that ${\mathbf{U}}_{\mathcal{A}}(\theta )=n^{-1}{\left[{\mathbf{X}}_{\mathcal{A}},{\tilde{\mathbf{X}}}_{\mathcal{A}}(\theta )\right]}^{\prime }\left[{\mathbf{X}}_{\mathcal{A}},{\tilde{\mathbf{X}}}_{\mathcal{A}}(\theta )\right]$ and $v_{\mathcal{A}}(\theta )=n^{-1}{\left[X_{\mathcal{A}},{\tilde{X}}_{\mathcal{A}}(\theta )\right]}^{\prime }y$. Motivated by Lemma 1, we define a family of mappings indexed by $\mathcal{A}$ that describes the selection algorithm of Procedure 1 with given data set $\left(\left[{\mathbf{X}}_{\mathcal{A}},{\tilde{\mathbf{X}}}_{\mathcal{A}}(\theta )\right],\mathbf{y}\right)$ that forms ${\mathbf{T}}_{\mathcal{A}}(\theta )$. Formally, define a mapping $S_{\mathcal{A}}:{ℝ}^{|\mathcal{A}|(2|\mathcal{A}|+3)}\to 2^{\mathcal{A}}$ as ${\text{t}}_{\mathcal{A}}\mapsto S_{\mathcal{A}}\left({\text{t}}_{\mathcal{A}}\right)$ for given ${\mathbf{T}}_{\mathcal{A}}(\theta )={\mathbf{t}}_{\mathcal{A}}$, where $2^{\mathcal{A}}$ refers to the power set of $\mathcal{A}$. That is, $S_{\mathcal{A}}\left({\mathbf{t}}_{\mathcal{A}}\right)$ represents the outcome of first restricting ourselves to the smaller set of variables $\mathcal{A}$ and then applying IPAD to ${\mathbf{T}}_{\mathcal{A}}(\theta )={\mathbf{t}}_{\mathcal{A}}$ to further select variables from set $\mathcal{A}$.

Lemma 2

Under Conditions 1–4, for any subset $\mathcal{A}\supset \mathcal{A}\ast (\theta )$ we have $S_{\{1,\cdots ,p\}}(\mathbf{T}(\theta ))=S_{\mathcal{A}}\left({\mathbf{T}}_{\mathcal{A}}(\theta )\right)$.

When restricting on set $\mathcal{A}$, we can apply Procedure 1 to a lower-dimensional data set $\left(\left[{\mathbf{X}}_{\mathcal{A}},{\tilde{\mathbf{X}}}_{\mathcal{A}}(\theta )\right],\mathbf{y}\right)$ that forms ${\mathbf{T}}_{\mathcal{A}}(\theta )$ to further select variables from $\mathcal{A}$. The previous two lemmas ensure that this gives us a subset of $\mathcal{A}$ that is identical to S_{1,⋯,p}(T(θ)). Note that the lower-dimensional problem based on ${\mathbf{T}}_{\mathcal{A}}(\theta )$ can be easier compared to the original one. We also would like to emphasize that the dimensionality reduction to a smaller model $\mathcal{A}$ is only for assisting the theoretical analysis and our Procedure 1 does not need any knowledge of such set $\mathcal{A}$.

It is convenient to define ${\mathbf{t}}_0=\mathbb{E}\mathbf{T}\left({\mathit{\theta }}^0\right)\in {ℝ}^P$. Denote by

$$\mathbb{I}:=\left\{\mathbf{t}\in {ℝ}^P:{\Vert \mathbf{t}-{\mathbf{t}}_0\Vert }_{\max }\le a_{np}:=C_1\left(k^{1/2}+s^{3/2}\right){\tilde{c}}_{np}\right\},$$ (13)

where C₁ is some positive constant and ${\tilde{c}}_{np}=p^{-1/2}\log n+n^{-1/2}\log p$. For any subset $\mathcal{A}\subset \{1,\cdots ,p\}$, let ${\mathbb{I}}_{\mathcal{A}}$ be the subspace of $\mathbb{I}$ when taking out the coordinates corresponding to $\mathbb{E}{\mathbf{T}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)$. Thus ${\mathbb{I}}_{\mathcal{A}}\subset {ℝ}^{|\mathcal{A}|(2|\mathcal{A}|+3)}$. In addition to Conditions 1–5, we need an assumption on the algorithmic stability of Procedure 1.

Condition 6 (Algorithmic stability)

For any subset $\mathcal{A}\subset \{1,\cdots ,p\}$ that satisfies $|\mathcal{A}|\le k\le n\wedge p$, there exists a positive sequence ρ_np → 0 as n ∧ p → ∞ such that

$$\underset{\left|\mathcal{A}\right|\le k}{\sup }\underset{{\mathbf{t}}_1,{\mathbf{t}}_2\in {\mathbb{I}}_{\mathcal{A}}}{\sup }\frac{\left|S_{\mathcal{A}}\left({\mathbf{t}}_2\right)\mathrm{\Delta }{S}_{\mathcal{A}}\left({\mathbf{t}}_1\right)\right|}{\left|S_{\mathcal{A}}\left({\mathbf{t}}_1\right)\right|\wedge \left|S_{\mathcal{A}}\left({\mathbf{t}}_2\right)\right|}=O\left({\rho }_{np}\right),$$

where Δ stands for the symmetric difference between two sets.

Intuitively the above condition assumes that the knockoffs procedure is stable with respect to a small perturbation to the input t in any lower-dimensional subspace ${\mathbb{I}}_{\mathcal{A}}$. Under these regularity conditions, the asymptotic FDR control of our IPAD procedure can be established.

Theorem 1 (Robust FDR control)

Assume that Conditions 1–6 hold. Fix an arbitrary positive constant ν. If (s, k, n, p) satisfies s ∨ k ≤ n ∧ p, c_np ≤ c/[r²M²C(ν + 2)]^1/2, and $\left(k^{1/2}+s^{3/2}\right){\tilde{c}}_{np}\to 0$ as n ∧ p → ∞ with c and C some positive constants defined in Lemma 7 in Appendix, then the set of variables $\widehat{\mathcal{S}}$ obtained by Procedure 1 (IPAD) with the LCD knockoff statistics controls the FDR (3) to be no larger than q + O (ρ_np + n^−ν + p^−ν).

Recall that by definition, the FDR is a function of $\mathbf{T}(\widehat{\mathit{\theta }})$ and can be written as $\mathbb{E}\mathrm{FDP}(\mathbf{T}(\widehat{\mathit{\theta }}))$ while the FDR computed with the oracle knockoffs, $\mathbb{E}\mathrm{FDP}\left(\mathbf{T}\left({\theta }^0\right)\right)$, is perfectly controlled to be no larger than q. This observation motivates us to first establish asymptotic equivalence of $\mathbf{T}(\widehat{\mathit{\theta }})$ and T(θ⁰) with large probability. Then a natural idea is to show that $\mathbb{E}\mathrm{FDP}(\mathbf{T}(\widehat{\mathit{\theta }}))$ converges to $\mathbb{E}\mathrm{FDP}\left(\mathbf{T}\left({\theta }^0\right)\right)$ in probability, which turns out to be highly nontrivial because of the discontinuity of FDP(·) (the convergence would be straightforward via the Portmanteau lemma if FDP(·) were continuous). Condition 6 above provides a remedy to this issue by imposing the algorithmic stability assumption.

3.3. Power analysis

We have established the asymptotic FDR control for our IPAD procedure in Section 3.2. We now look at the other side of the coin – the power (5). Recall that in IPAD, we apply the knockoffs inference procedure to the knockoff statistics LCD, which are constructed using the augmented Lasso in (9). Therefore the final set of variables selected by IPAD is a subset of variables picked by the augmented Lasso. For this reason, the power of IPAD is always upper bounded by that of Lasso. We will show in this section that there is in fact no power loss relative to the augmented Lasso in the asymptotic sense.

Condition 7 (Signal strength I)

For any subset $\mathcal{A}\subset {\mathcal{S}}^0$ that satisfies $|\mathcal{A}|/s>1-\gamma$ for some γ ∈ (0, 1], it holds that ${\Vert {\beta }_{\mathcal{A}}\Vert }_1>b_{np}s{n}^{-1/2}\log p$ for some positive sequence b_np → ∞.

Condition 8 (Signal strength II)

There exists some constant C₂ ∈ (2(qs)⁻¹,1) such that $\left|{\mathcal{S}}_2\right|\ge C_{2}s$ with ${\mathcal{S}}_2=\left\{j:\left|{\beta }_j\right|\gg {(s/n)}^{1/2}\log p\right\}$.

Condition 7 requires that the overall signal is not too weak, but is weaker than the conventional beta-min condition min ${\min }_{j\in {\mathcal{S}}^0}\left|{\beta }_j\right|\gg n^{-1/2}\log p$. Under Condition 8, we can show that $|\widehat{\mathcal{S}}|\ge C_{2}s$ with probability at least 1 − O(p^−ν + n^−ν) using similar techniques to those of Lemma 6 in [26]. The intuition is that given s → ∞, for a variable selection procedure to have high power it should select at least a reasonably large number of variables. The result $|\widehat{\mathcal{S}}|\ge C_{2}s$ will be used to derive the asymptotic order of threshold T, which is in turn crucial to establish the theorem below on power.

Theorem 2 (Power guarantee)

Assume that Conditions 1–5 and 7–8 hold. Fix an arbitrary positive constant ν. If (s,k,n,p) satisfies 2s ≤ k ≤ n ∧ p, c_np ≤ c/(r²M²C(ν + 2))^1/2, and $s{k}^{1/2}{\tilde{c}}_{np}\to 0$ as n ∧ p → ∞ with c and C some positive constants defined in Lemma 7, then both the Lasso procedure based on (X, y) and our IPAD procedure (Procedure 1) have power bounded from below by γ − o(1) as n ∧ p → ∞. In particular, if γ = 1 IPAD has no power loss compared to Lasso asymptotically.

4. Simulation studies

We have shown in Section 3 that IPAD can asymptotically control the FDR in high-dimensional setting and there can be no power loss in applying the procedure. We next move on to numerically investigate the finite-sample performance of IPAD using synthetic data sets. We will compare IPAD with the knockoff filter in [4] (BCKnockoff) and the high-dimensional knockoff filter in [5] (HD-BCKnockoff). In what follows, we will first explain in detail the model setups and simulation settings, then discuss the implementation of the aforementioned methods, and finally summarize the comparison results.

4.1. Simulation designs and settings

In all simulations, the design matrix X ∈ ℝ^n×p is generated from the factor model

$$\mathbf{X}={\mathbf{F}}^0{\left({\mathbf{\Lambda }}^0\right)}^{\prime }+\sqrt{r\theta }\mathbf{E}={\mathbf{C}}^0+\sqrt{r\theta }\mathbf{E},$$ (14)

where ${\mathbf{F}}^0={\left({\mathbf{f}}_1^0,\cdots ,{\mathbf{f}}_n^0\right)}^{\prime }\in {ℝ}^{n\times r}$ is the matrix of latent factors, ${\mathbf{\Lambda }}^0={\left({\lambda }_1^0,\cdots ,{\lambda }_p^0\right)}^{\prime }\in {ℝ}^{p\times r}$ is the matrix of factor loadings, E ∈ ℝ^n×p is the matrix of model errors, and θ is a constant controlling the signal-to-noise ratio. The term $\sqrt{r}$ is used to single out the effect of the number of factors in calculating the signal-to-noise ratio in factor model (14). We then rescale each column of X to have ℓ₂-norm one and simulate the response vector y = (y₁,⋯, y_n)⁰ from the following model

$$y_i=f\left({\mathbf{x}}_i\right)+\sqrt{c}{\epsilon }_i,i=1,\cdots ,n,$$ (15)

where f : ℝ^p → ℝ is the link function which can be linear or nonlinear, c > 0 is a constant controlling the signal-to-noise ratio, and ε = (ε₁,⋯, ε_n)^′ is the vector of model error. We next explain the four different designs of our simulation studies.

4.1.1. Design 1: linear model with normal factor design matrix

The elements of F⁰, Λ⁰, E, and ε are drawn independently from $\mathcal{N}(0,1)$. The link function takes a linear form, that is, $\mathbf{y}=\mathbf{X}\beta +\sqrt{c}\epsilon$, where the coefficient vector β = (β₁,⋯, β_p)′ ∈ ℝp is generated by first choosing s random locations for the true signals and then setting β_j at each location to be either A or −A randomly with A some positive value. The remaining p − s components of β are set to zero.

4.1.2. Design 2: linear model with fat-tail factor matrix and serial dependence

The elements of E are generated as

$$e_{ij}=\left(\frac{\nu -2}{{\chi }_{\nu ,j}^2}\right)u_{ij},$$ (16)

where $u_{ij}\sim i.i.d.\mathcal{N}(0,1)$ for all i = 1,⋯, n and j = 1,⋯, p, and ${\chi }_{\nu ,j}^2,j=1,\cdots ,p$ are i.i.d. random variables from chi-square distribution with ν = 8 degrees of freedom. The rest of the design is the same as in Design 1. It is worth mentioning that in this case, the entries of matrix E have fat-tail distribution with serial dependence in each column because of the common factor ${\chi }_{\nu ,j}^2$. This design is used to check the robustness of IPAD method with respect to the serial dependence and the fat-tail distribution of E.

4.1.3. Design 3: linear model with misspecified design matrix

To evaluate the robustness of IPAD procedure to the misspecification of the factor model structure (14), we set Λ = 0, rθ = 1 and simulate the rows of matrix E independently from $\mathcal{N}(\mathbf{0},\mathbf{\Sigma })$ with Σ = (σ_ij), σ_ij = ρ^|i−j| for ≤ i,j ≤ p. The remaining design is the same as in Design 1. It is seen that our assumption on the independence of the entries of E is violated. This design is used to test the robustness of IPAD to misspecification of the factor model structure of X.

4.1.4. Design 4: nonlinear model with normal factor design matrix

Our last design is used to evaluate the performance of IPAD method when the link function f is nonlinear. To be more specific, we assume the following nonlinear model between the response and covariates

$$\mathbf{y}=\sin (\mathbf{X}\beta )\exp (\mathbf{X}\beta )+\sqrt{c}\epsilon ,$$

where the coefficient vector β, design matrix X, and model error ε are generated similarly as in Design 1.

4.1.5. Simulation settings

The target FDR level is set to be q = 0.2 in all simulations. For Design 1 and Design 2, we set n = 2000, p = 2000, A = 4, s = 50, c = 0.2, r = 3, and θ = 1. In order to evaluate the sensitivity of our method to the dimensionality p and the model sparsity s, we also explore the settings of p = 1000,3000 and s = 100,150. In Design 3, we set r = 0 and ρ = 0,0.5. In Design 4, since the model is nonlinear, we use the nonparametric method of random forest [14] to fit the model and consider lower-dimensional settings of p = 50, 250, and 500. We also decrease the number of observations to n = 1000 and number of true variables to s = 10. Moreover, we set θ = 1, 2 and c = 0.1, 0.2, 0.3 to test the effects of signal-to-noise ratio on the performance of IPAD procedure in Design 4. The implementation details for the estimation procedure of IPAD are provided in Section E.1 of Supplementary Material.

4.2. Simulation results

For each method, we use 100 simulated data sets to calculate its empirical FDR and power, which are the average FDP and TDP (true discovery proportion as in (5)) over 100 repetitions, respectively. Two different thresholds, knockoff and knockoff+ (T₁ and T₂ in Result 1, respectively), are used in the knockoffs inference implementation. It is worth mentioning that as shown in [15] and summarized in Result 1, knockoff+ controls FDR (3) exactly while knockoff controls only the modified FDR (4).

Tables 1 and 2 summarize the results from Designs 1 and 2, respectively. As shown in Table 1, all approaches can control empirical FDR at the target level (q = 0.2) and knockoff+, which is more conservative, reduces power negligibly. It is worth mentioning that even for Design 2, in which the design matrix X is drawn from fat-tail distribution with serial dependence, we still have FDR under control with decent level of power. This suggests that the no serial correlation assumption in our theoretical analysis could just be technical. Compared to the results by BCKnockoff and HD-BCKnockoff, we see that using the extra information from the factor structure in constructing knockoff variables can help with both FDR and power. Table 2 also shows the effects of model sparsity on the performance of various approaches. It can be seen that when the number of true signals is increased from 50 to 150, the FDR is still under control and the empirical power of IPAD remains steady.

Table 1:

Simulation results for Designs 1 and 2 of Section 4.1 with different values of dimensionality p

	Design 1					Design 2
	FDR	Power	FDR+	Power+	R2	FDR	Power	FDR+	Power+	R2
	p = 1000
IPAD	0.195	0.991	0.180	0.990	0.659	0.199	0.961	0.180	0.960	0.652
BCKnockoff	0.207	0.942	0.192	0.938	0.659	0.172	0.887	0.152	0.885	0.653
	p = 2000
IPAD	0.194	0.979	0.179	0.979	0.649	0.199	0.935	0.183	0.933	0.656
HD-BCKnockoff	0.142	0.706	0.127	0.691	0.649	0.136	0.607	0.113	0.581	0.644
	p = 3000
IPAD	0.191	0.964	0.176	0.963	0.652	0.188	0.913	0.171	0.911	0.658
HD-BCKnockoff	0.172	0.668	0.149	0.658	0.652	0.125	0.559	0.099	0.524	0.651

Note that FDR₊ and Power₊ are the values of FDR and Power corresponding to the knockoff₊ threshold T₂.

Table 2:

Simulation results for Designs 1 and 2 of Section 4.1 with different sparsity level s

	Design 1					Design 2
	FDR	Power	FDR+	Power+	R2	FDR	Power	FDR+	Power+	R2
	s = 50
IPAD	0.194	0.979	0.179	0.979	0.649	0.199	0.935	0.183	0.933	0.656
HD-BCKnockoff	0.142	0.706	0.127	0.691	0.649	0.136	0.607	0.113	0.581	0.644
	s = 100
IPAD	0.191	0.978	0.183	0.977	0.783	0.181	0.937	0.174	0.936	0.789
HD-BCKnockoff	0.152	0.703	0.140	0.698	0.787	0.106	0.583	0.097	0.573	0.778
	s = 150
IPAD	0.183	0.973	0.178	0.972	0.842	0.188	0.935	0.182	0.935	0.848
HD-BCKnockoff	0.139	0.660	0.130	0.654	0.858	0.115	0.578	0.106	0.570	0.843

Table 3 is devoted to the case of Design 3, where the rows of matrix X are generated independently from multivariate normal distribution with AR(1) correlation structure. This is a setting where the factor model structure in X is misspecified. Since BCknockoff and HD-BCknockoff make no use of the factor structure in generating knockoff variables, in both low- and high-dimensional examples both methods control FDR exactly at the target level. IPAD based methods have empirical FDR slightly over the target level, which may be caused by the misspecification of the factor structure. On the other hand, IPAD based approaches have much higher empirical power than comparison methods.

Table 3:

Simulation results for Design 3 of Section 4.1

	ρ = 0					ρ = 0:5
	FDR	Power	FDR+	Power+	R2	FDR	Power	FDR+	Power+	R2
	p = 1000
IPAD	0.204	0.995	0.189	0.995	0.444	0.226	0.984	0.216	0.984	0.446
BCKnockoff	0.188	0.919	0.172	0.917	0.444	0.137	0.827	0.117	0.821	0.445
	p = 2000
IPAD	0.203	0.993	0.189	0.993	0.447	0.220	0.982	0.202	0.980	0.445
HD-BCKnockoff	0.151	0.630	0.126	0.603	0.449	0.115	0.522	0.090	0.467	0.442
	p = 3000
IPAD	0.225	0.988	0.205	0.987	0.445	0.219	0.979	0.206	0.978	0.443
HD-BCKnockoff	0.150	0.589	0.126	0.560	0.446	0.092	0.439	0.064	0.381	0.447

Table 4 corresponds to Design 4 in which response y is related to X nonlinearly. Since BCKnockoff and HD-BCKnockoff are designed for linear models, only the results from IPAD method are reported. It can be seen form Table 4 that IPAD approach can control FDR with reasonably high power even in the nonlinear setting. We also observe that in nonlinear setting, the power of IPAD deteriorates faster as dimensionality p increases compared to the linear setting due to the use of the fully nonparametric approach for estimation.

Table 4:

Simulation results for Design 4 of Section 4.1

	θ = 1					θ = 2
	FDR	Power	FDR+	Power+	R2	FDR	Power	FDR+	Power+	R2
	p = 50
c = 0.1	0.109	0.839	0.081	0.720	0.707	0.110	0.943	0.061	0.858	0.707
c = 0.2	0.137	0.847	0.068	0.726	0.547	0.097	0.920	0.061	0.837	0.547
c = 0.3	0.137	0.765	0.091	0.582	0.451	0.123	0.907	0.076	0.774	0.451
	p = 250
c = 0.1	0.189	0.740	0.104	0.504	0.702	0.174	0.876	0.139	0.788	0.702
c = 0.2	0.218	0.666	0.131	0.522	0.552	0.209	0.831	0.118	0.660	0.552
c = 0.3	0.200	0.569	0.101	0.361	0.451	0.224	0.766	0.141	0.599	0.451
	p = 500
c = 0.1	0.243	0.661	0.169	0.497	0.702	0.223	0.831	0.173	0.740	0.702
c = 0.2	0.204	0.507	0.111	0.266	0.543	0.216	0.749	0.126	0.594	0.543
c = 0.3	0.247	0.478	0.128	0.299	0.451	0.241	0.691	0.156	0.550	0.451

5. Empirical analysis

Our simulation results in Section 4 suggest that IPAD is a powerful approach with asymptotic FDR control. We further examine the application of IPAD to the quarterly data on 109 macroeconomic variables from the third quarter of year 1960 (1960Q3) to the fourth quarter of year 2008 (2008Q4) in the United States discussed in [37]. These variables are transformed by taking logarithms and/or differencing following [37]. Our real data analysis consists of two parts. In the first part, we focus on the performance of IPAD method in terms of empirical FDR and power. To save space, the numerical results for the real data based simulation study are presented in Section E.2 of of Supplementary Material. In the second part, the forecasting performance of IPAD method will be evaluated.

We now apply the IPAD approach to the real economic data set for forecasting. One-step ahead prediction is conducted using rolling window of size 120. More specifically, one of the 109 variables is chosen as the response and the remaining 108 variables are treated as predictors. For each quarter between 1990Q3 and 2008Q4, we use the previous 120 periods for model fitting and then one-step ahead prediction is conducted based on the fitted model. We compare IPAD with the competing methods of autoregression of order one (AR(1)), factor augmented AR(1) (FAR), and Lasso, where each method is implemented in a same way as IPAD for one-step ahead prediction; see Section E.3 of Supplementary Material for the implementation details of all the methods.

The number of factors $\widehat{r}$ is chosen by the PC_p1 criterion in [3]. For the Lasso and IPAD, the regularization parameter λ is selected with the tenfold cross-validation. Table 5 shows the root mean-squared prediction error (RMSE) of these methods. As can be seen, the RMSE of IPAD is very close to those of comparison methods. To statistically compare the relative prediction accuracy of IPAD versus other approaches, we have used the Diebold–Mariano test [21], where the square of one-step ahead prediction error is used as the loss function. Table 6 reports the test results. The results indicate that one-step ahead prediction accuracy of IPAD is comparable to other approaches.

Table 5:

Root mean-squared error of one-period ahead forecast of various macroeconomic variables

	AR	FAR	Lasso	IPAD
RGDP	2.245	1.929	2.070	2.106
CPI-ALL	1.526	1.552	1.579	1.571
Imports	7.549	5.871	6.595	6.993
IP: cons dble	9.683	8.353	8.424	9.175
Emp: TTU	1.112	0.989	1.167	1.100
U: mean duration	0.573	0.487	0.502	0.494
HStarts: South	0.074	0.071	0.076	0.074
NAPM new ordrs	4.800	4.378	4.659	4.673
PCED-NDUR-ENERGY	31.927	32.121	33.546	32.164
Emp. Hours	2.102	1.899	2.080	1.944
FedFunds	0.421	0.396	0.406	0.392
Cons credit	2.573	2.537	2.648	2.580
EX rate: Canada	10.132	10.139	10.122	10.113
DJIA	23.117	23.997	24.585	23.398
Consumer expect	6.496	6.888	6.681	6.661

Table 6:

Diebold–Mariano test for comparing prediction accuracy of IPAD against other procedures

	IPAD vs. AR	IPAD vs. FAR	IPAD vs. Lasso
RGDP	−0.780	1.160	0.462
CPI-ALL	0.521	0.394	−0.218
Imports	−0.976	2:631**	1.464
IP: cons dble	−1.026	1.567	2:487*
Emp: TTU	−0.140	1.692	−1.845
U: mean duration	−3:383***	0.672	−0.505
HStarts: South	0.096	0.821	−0.766
NAPM new ordrs	−0.517	1.814	0.076
PCED-NDUR-ENERGY	0.753	0.049	−1.759
Emp. Hours	−1.200	0.297	−2:063*
FedFunds	−0.971	−0.134	−0.625
Cons credit	0.207	0.359	−0.661
EX rate: Canada	−0.466	−0.138	−0.037
DJIA	0.585	−0.959	−1.428
Consumer expect	1.212	−1.038	−0.277

It is worth mentioning that one main advantage of IPAD is its interpretability and stability. Using IPAD for forecasting, we not only enjoy the same level of accuracy as other methods but also obtain the information on variable importance with stability. Recall that for each one-step ahead prediction, we apply IPAD 100 times and obtain 100 sets of selected variables. Thus we can calculate the selection frequency of each variable in each one-step ahead prediction. Figure 1 depicts the frequencies of top five selected variables in predicting real GDP growth before and after year 2000, where the variable importance is ranked according to the aggregated frequencies over the entire time period before or after 2000. We have experimented with different cutoff years around year 2000, and the top five ranked variables stay the same so only the results corresponding to cutoff year 2000 are reported. Changes in index of help wanted advertising in newspapers, percentages of changes in real personal consumption of services, and percentage of changes in real gross private domestic investment in residential sector were the top three important variables in predicting real GDP growth during the whole period. It is interesting to see that percentage of changes in residential price index was among top five important variables in predicting GDP growth during the 90s, and then starting from year 2000 it was replaced by changes in index of consumer expectations about stability of economy. Moreover, it is also seen that the percentage of changes in industrial production of fuels was of great importance for predicting real GDP growth during some periods but not the others.

Figure 1:

Frequencies of top selected variables in predicting real GDP growth. The set of selected variables are index of help-wanted advertising in newspapers (Help wanted indx), real personal consumption expenditures - services (Cons-Serv), real gross private domestic investment - residential (Res.Inv), residential price index (PFI-RES), industrial production index - fuels (IP:fuels), and University of Michigan index of consumer expectations (Consumer expect).

As a comparison, it is very difficult to interpret the results of FAR. As for Lasso based method, there is no theoretical guarantee on FDR control and in addition, Lasso usually gives us models with much larger size. For instance, in predicting real GDP growth, IPAD on average selects 5.42 macroeconomic variables while Lasso on average selects 13.32 variables. To summarize, our real data analysis indicates that IPAD is an applicable approach for controlling FDR with competitive prediction power and high interpretability and stability.

6. Discussions

We have suggested in this paper a new procedure IPAD for feature selection in high-dimensional linear models that achieves asymptotic FDR control while retaining high power. Our model setting involves a latent factor model that is motivated by applications in economics and finance. Our method falls into the general model-X knockoffs framework in [15], but allows the unknown covariate distribution for the knockoff variable construction. With the LCD knockoff statistics, we have shown that the FDR of IPAD can be asymptotically under control while the power can be asymptotically the same as that of Lasso. Our simulation study and empirical analysis also suggest that IPAD has highly competitively performance compared to many widely used forecasting methods such as Lasso and FAR, but with much higher interpretability and stability.

Our work has focused on the scenario of static models. It would be interesting to extend the IPAD procedure to high-dimensional dynamic models with time series data. It is also interesting to consider nonlinear models and more flexible machine learning methods for forecasting as well as more refined factor model structures on the covariates for the knockoffs inference with IPAD, and develop theoretical guarantees for the IPAD framework in these more general model settings. These extensions are beyond the scope of the current paper and are interesting topics for future research.

A. Proofs of main results

We provide the proofs of Theorems 1–2 in this appendix. The proofs of Proposition 1 and Lemmas 1–2 and additional technical details are included in the Supplementary Material.

To ease the technical presentation, let us introduce some notation. We denote by ≲. the inequality up to some positive constant factor. Restricting the columns of X and $\tilde{\mathbf{X}}(\widehat{\mathit{\theta }})$ to the variables in index set $\mathcal{A}$ such that $|\mathcal{A}|\le k$, we obtain the n × k submatrices ${\mathbf{X}}_{\mathcal{A}}$ and ${\tilde{\mathbf{X}}}_{\mathcal{A}}(\widehat{\mathit{\theta }})$, respectively. Moreover, we define ${\mathbf{T}}_{\mathcal{A}}(\widehat{\mathit{\theta }}):=\mathrm{vec}\left(\mathrm{vech}{\mathbf{U}}_{\mathcal{A}}(\widehat{\mathit{\theta }}),{\mathbf{v}}_{\mathcal{A}}(\widehat{\mathit{\theta }})\right)\in {ℝ}^{k(2k+3)}$ with ${\mathbf{U}}_{\mathcal{A}}(\widehat{\mathit{\theta }})$ the principle submatrix of $\mathbf{U}(\widehat{\mathit{\theta }})$ formed by columns and rows in set $\mathcal{A}$, and ${\mathbf{v}}_{\mathcal{A}}(\widehat{\mathit{\theta }})$ the subvector of $\mathbf{v}(\widehat{\mathit{\theta }})$ formed by components in set $\mathcal{A}$. Then it is easy to see that ${\mathbf{U}}_{\mathcal{A}}(\widehat{\mathit{\theta }})=n^{-1}{\left[{\mathbf{X}}_{\mathcal{A}},{\tilde{\mathbf{X}}}_{\mathcal{A}}(\widehat{\mathit{\theta }})\right]}^{\prime }\left[{\mathbf{X}}_{\mathcal{A}},{\tilde{\mathbf{X}}}_{\mathcal{A}}(\widehat{\mathit{\theta }})\right]$ and ${\mathbf{v}}_{\mathcal{A}}(\widehat{\mathit{\theta }})=n^{-1}{\left[{\mathbf{X}}_{\mathcal{A}},{\tilde{\mathbf{X}}}_{\mathcal{A}}(\widehat{\mathit{\theta }})\right]}^{\prime }\mathbf{y}$. For the oracle factor loading matrix Λ⁰, with a slight abuse of notation we use ${\mathbf{\Lambda }}_{\mathcal{A}}^0$ to denote the row restricted to the variables in $\mathcal{A}$ for notational convenience. Recall that ν > 0 is a fixed positive number, c_np = (p⁻¹ log n)^1/2 +(n⁻¹ log p)^1/2, and ${\tilde{c}}_{np}=p^{-1/2}\log n+n^{-1/2}\log p$. We define π_np = n^−ν + p^−ν. Since λ is fixed at C₀n^−1/2 log p, in all the proofs we will drop the dependence of various quantities on λ whenever there is no confusion.

A.1. Proof of Theorem 1

Recall that for a given θ, $\mathcal{A}\ast \left({\theta }^0\right)$ is the support of knockoff statistics (W₁(θ),⋯, W_p(θ))^′. Define set $\widehat{\mathcal{A}}(\widehat{\mathit{\theta }}):=\mathcal{A}\ast (\widehat{\mathit{\theta }})\cup \mathcal{A}\ast \left({\theta }^0\right)$. It follows from (12) that the cardinality of $\widehat{\mathcal{A}}(\widehat{\mathit{\theta }})$ is bounded by k. Hereafter we write $\widehat{\mathcal{A}}(\widehat{\mathit{\theta }})$ as $\widehat{\mathcal{A}}$ for notational simplicity.

By Lemmas 1–2 and the definition of the FDP, we know that $S_{\{1,\cdots ,p\}}(\mathbf{T}(\widehat{\mathit{\theta }}))=S_{\widehat{\mathcal{A}}}\left({\mathbf{T}}_{\widehat{\mathcal{A}}}(\widehat{\mathit{\theta }})\right)$ and thus the resulting FDR’s are the same. Therefore, we can restrict ourselves to the smaller model $\widehat{\mathcal{A}}$ when studying the FDR of IPAD. The same arguments as above also hold for the oracle knockoffs; that is, the FDR of IPAD applied to T(θ⁰) is the same as that applied to ${\mathbf{T}}_{\widehat{\mathcal{A}}}({\theta }^0)$. Note that all the FDR’s we discuss here are with respect to the full model {1,⋯, p}. For this reason, in what follows we will abuse the notation and use ${\text{FDR}}_{\widehat{\mathcal{A}}}\left({\mathbf{T}}_{\widehat{\mathcal{A}}}(\theta )\right)$ and ${\text{FDR}}_{\widehat{\mathcal{A}}}\left({\mathbf{T}}_{\widehat{\mathcal{A}}}({\theta }^0)\right)$ to denote the FDR of IPAD based on ${\mathbf{T}}_{\widehat{\mathcal{A}}}(\theta )$ and ${\mathbf{T}}_{\widehat{\mathcal{A}}}({\theta }^0)$, respectively. We want to emphasize that although we put a subscript $\widehat{\mathcal{A}}$ in FDR’s, their values are still deterministic as argued above. Summarizing the facts, we obtain

$$\begin{gathered} {\mathrm{FDR}}_{\widehat{\mathcal{A}}}\left({\mathbf{T}}_{\widehat{\mathcal{A}}}(\widehat{\mathit{\theta }})\right)={\mathrm{FDR}}_{\{1,\cdots ,p\}}(\mathbf{T}(\widehat{\mathit{\theta }})), \\ {\mathrm{FDR}}_{\widehat{\mathcal{A}}}\left({\mathbf{T}}_{\widehat{\mathcal{A}}}\left({\mathit{\theta }}^0\right)\right)={\mathrm{FDR}}_{\{1,\cdots ,p\}}\left(\mathbf{T}\left({\mathit{\theta }}^0\right)\right). \end{gathered}$$

Meanwhile, by construction $\tilde{\mathbf{X}}\left({\mathit{\theta }}^0\right)$ satisfies the two properties in Definition 1 and is a valid model-X knockoffs matrix. Therefore, for any value of the regularization parameter, the LCD statistics W_j(θ⁰) based on $\left(\left[\mathbf{X},\tilde{\mathbf{X}}\left({\mathit{\theta }}^0\right)\right],\mathbf{y}\right)$ together with Result 1 ensure the exact FDR control at some target level q ∈ (0, 1). Summarizing this, we obtain that the FDR of IPAD applied to T(θ⁰) is controlled at target level q.

Combining the arguments in the previous two paragraphs, we deduce

$${\mathrm{FDR}}_{\widehat{\mathcal{A}}}\left({\mathbf{T}}_{\widehat{\mathcal{A}}}\left({\mathit{\theta }}^0\right)\right)={\mathrm{FDR}}_{\{1,\cdots ,p\}}\left(\mathbf{T}\left({\mathit{\theta }}^0\right)\right)\le q.$$

Thus the desired results follow automatically if we can prove that ${\text{FDR}}_{\widehat{\mathcal{A}}}\left({\mathbf{T}}_{\widehat{\mathcal{A}}}(\widehat{\mathit{\theta }})\right)$ is asymptotically close to ${\mathrm{FDR}}_{\widehat{\mathcal{A}}}\left({\mathbf{T}}_{\widehat{\mathcal{A}}}\left({\mathit{\theta }}^0\right)\right)$. We next proceed to prove it.

Recall the definitions of $\mathbb{I}$ and ${\mathbb{I}}_{\mathcal{A}}$ as in (13). Define the event

$${\mathcal{E}}_{np}=\left\{{\mathbf{T}}_{\widehat{\mathcal{A}}}(\widehat{\mathit{\theta }})\in {\mathbb{I}}_{\widehat{\mathcal{A}}}\right\}\cap \left\{{\mathbf{T}}_{\widehat{\mathcal{A}}}\left({\mathit{\theta }}^0\right)\in {\mathbb{I}}_{\widehat{\mathcal{A}}}\right\}.$$

Lemma 3 in Section C.4 establishes $\widehat{\mathit{\theta }}\in {\mathbf{\Theta }}_{np}$ with probability at least 1 − O(π_np) and θ⁰ ∈ Θ_np. Hence, Lemma 4 in Section C.5 guarantees that

$$\mathbb{P}\left({\mathcal{E}}_{np}^c\right)\le 2\mathbb{P}\left(\underset{\left|\right|A|\le k,\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\mathbf{T}}_{\mathcal{A}}(\mathit{\theta })-\mathbb{E}\left[{\mathbf{T}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)\right]\Vert }_{\max }>a_{np}\right)=O\left({\pi }_{np}\right),$$ (17)

where $a_{np}=C_1\left(k^{1/2}+s^{3/2}\right){\tilde{c}}_{np}$ for some constant C₁ > 0.

For a given deterministic set $\mathcal{A}\subset \{1,\cdots ,p\}$, let ${\text{FDP}}_{\mathcal{A}}(\cdot )$ be the FDP function corresponding to ${\text{FDP}}_{\mathcal{A}}(\cdot )$. By the definition of FDP function, we have for any t₁, ${\mathbf{t}}_2\in {ℝ}^{|\mathcal{A}|(2|\mathcal{A}|+3)}$,

$$\begin{gathered} {\text{FDP}}_{\mathcal{A}}\left({\mathbf{t}}_2\right)-{\text{FDP}}_{\mathcal{A}}\left({\mathbf{t}}_1\right)=\frac{\left|{\mathcal{S}}^1\cap S_A\left({\mathbf{t}}_2\right)\right|}{\left|S_{\mathcal{A}}\left({\mathbf{t}}_2\right)\right|}-\frac{\left|{\mathcal{S}}^1\cap S_{\mathcal{A}}\left({\mathbf{t}}_1\right)\right|}{\left|S_{\mathcal{A}}\left({\mathbf{t}}_1\right)\right|} \\ =\frac{\left|{\mathcal{S}}^1\cap S_{\mathcal{A}}\left({\mathbf{t}}_2\right)\right|\cdot \left(\left|S_{\mathcal{A}}\left({\mathbf{t}}_1\right)\right|-\left|S_{\mathcal{A}}\left({\mathbf{t}}_2\right)\right|\right)}{\left|S_{\mathcal{A}}\left({\mathbf{t}}_1\right)\right|\cdot \left|S_{\mathcal{A}}\left({\mathbf{t}}_2\right)\right|}+\frac{\left|{\mathcal{S}}^1\cap S_{\mathcal{A}}\left({\mathbf{t}}_2\right)\right|-\left|{\mathcal{S}}^1\cap S_{\mathcal{A}}\left({\mathbf{t}}_1\right)\right|}{\left|S_{\mathcal{A}}\left({\mathbf{t}}_1\right)\right|}. \end{gathered}$$

Further, note that

$$\begin{gathered} \left|{\mathcal{S}}^1\cap S_{\mathcal{A}}\left({\mathbf{t}}_2\right)\right|/\left|S_{\mathcal{A}}\left({\mathbf{t}}_2\right)\right|\le 1,\Vert S_{\mathcal{A}}\left({\mathbf{t}}_2\right)|-|S_{\mathcal{A}}\left({\mathbf{t}}_1\right)\Vert \le \left|S_{\mathcal{A}}\left({\mathbf{t}}_2\right)\mathrm{\Delta }{S}_{\mathcal{A}}\left({\mathbf{t}}_1\right)\right|, \\ \Vert {\mathcal{S}}^1\cap S_A\left({\mathbf{t}}_2\right)|-|{\mathcal{S}}^1\cap S_A\left({\mathbf{t}}_1\right)\Vert \le |\left\{S_A\left({\mathbf{t}}_2\right)\mathrm{\Delta }{S}_A\left({\mathbf{t}}_1\right)\right\}\cap {\mathcal{S}}^1|. \end{gathered}$$

Combining the results above yields

$$\begin{gathered} \left|{\text{FDP}}_{\mathcal{A}}\left({\mathbf{t}}_1\right)-{\text{FDP}}_{\mathcal{A}}\left({\mathbf{t}}_2\right)\right| \\ \le \frac{\left|\right|S_{\mathcal{A}}\left({\mathbf{t}}_1\right)|-|S_{\mathcal{A}}\left({\mathbf{t}}_2\right)\left|\right|}{\left|S_{\mathcal{A}}\left({\mathbf{t}}_1\right)\right|}+\frac{\left|\left\{S_{\mathcal{A}}\left({\mathbf{t}}_2\right)\mathrm{\Delta }{S}_{\mathcal{A}}\left({\mathbf{t}}_1\right)\right\}\cap {\mathcal{S}}^1\right|}{\left|S_{\mathcal{A}}\left({\mathbf{t}}_1\right)\right|}\le 2\frac{\left|S_{\mathcal{A}}\left({\mathbf{t}}_2\right)\mathrm{\Delta }{S}_{\mathcal{A}}\left({\mathbf{t}}_1\right)\right|}{\left|S_{\mathcal{A}}\left({\mathbf{t}}_1\right)\right|}. \end{gathered}$$

Similarly we have

$$\left|{\text{FDP}}_{\mathcal{A}}\left({\mathbf{t}}_1\right)-{\text{FDP}}_{\mathcal{A}}\left({\mathbf{t}}_2\right)\right|\le 2\frac{\left|S_{\mathcal{A}}\left({\mathbf{t}}_2\right)\mathrm{\Delta }{S}_{\mathcal{A}}\left({\mathbf{t}}_1\right)\right|}{\left|S_{\mathcal{A}}\left({\mathbf{t}}_2\right)\right|}.$$

Thus it holds that

$$\underset{\left|\mathcal{A}\right|\le k}{\sup }\underset{{\mathbf{t}}_1,{\mathbf{t}}_2\in {\mathbb{I}}_A}{\sup }\left|{\mathrm{FDP}}_{\mathcal{A}}\left({\mathbf{t}}_1\right)-{\mathrm{FDP}}_{\mathcal{A}}\left({\mathbf{t}}_2\right)\right|\le \underset{\left|\mathcal{A}\right|\le k}{\sup }\underset{{\mathbf{t}}_1,{\mathbf{t}}_2\in {\mathbb{I}}_{\mathcal{A}}}{\sup }\frac{\left|S_{\mathcal{A}}\left({\mathbf{t}}_2\right)\mathrm{\Delta }{S}_{\mathcal{A}}\left({\mathbf{t}}_1\right)\right|}{\left|S_{\mathcal{A}}\left({\mathbf{t}}_1\right)\right|\wedge \left|S_{\mathcal{A}}\left({\mathbf{t}}_2\right)\right|}=O\left({\rho }_{np}\right),$$ (18)

where the last two steps are due to Condition 6. Therefore, (17) and (18) together with the fact that FDP(·) ∈ [0, 1] entail that

$$\begin{gathered} \left|{\mathrm{FDR}}_{\widehat{\mathcal{A}}}\left({\mathbf{T}}_{\widehat{\mathcal{A}}}(\widehat{\mathit{\theta }})\right)-{\mathrm{FDR}}_{\widehat{\mathcal{A}}}\left({\mathbf{T}}_{\widehat{\mathcal{A}}}\left({\mathit{\theta }}^0\right)\right)\right|=\left|\mathbb{E}{\mathrm{FDP}}_{\widehat{\mathcal{A}}}\left({\mathbf{T}}_{\widehat{\mathcal{A}}}(\widehat{\mathit{\theta }})\right)-\mathbb{E}{\mathrm{FDP}}_{\widehat{\mathcal{A}}}\left({\mathbf{T}}_{\widehat{\mathcal{A}}}\left({\mathit{\theta }}^0\right)\right)\right| \\ \le \mathbb{E}\left|{\mathrm{FDP}}_{\widehat{\mathcal{A}}}\left({\mathbf{T}}_{\widehat{\mathcal{A}}}(\widehat{\mathit{\theta }})\right)-{\mathrm{FDP}}_{\widehat{\mathcal{A}}}\left({\mathbf{T}}_{\widehat{\mathcal{A}}}\left({\mathit{\theta }}^0\right)\right)\right| \\ \le \mathbb{E}\left[\left|{\mathrm{FDP}}_{\widehat{\mathcal{A}}}\left({\mathbf{T}}_{\widehat{\mathcal{A}}}(\widehat{\mathit{\theta }})\right)-{\mathrm{FDP}}_{\widehat{\mathcal{A}}}\left({\mathbf{T}}_{\widehat{\mathcal{A}}}\left({\mathit{\theta }}^0\right)\right)\right||{\mathcal{E}}_{np}\right]\mathbb{P}\left({\mathcal{E}}_{np}\right)+2\mathbb{P}\left({\mathcal{E}}_{np}^c\right) \\ \le \underset{\left|\mathcal{A}\right|\le k}{\sup }\underset{{\mathbf{t}}_1,{\mathbf{t}}_2\in {\mathbb{I}}_{\mathcal{A}}}{\sup }\left|{\text{FDP}}_{\mathcal{A}}\left({\mathbf{t}}_1\right)-{\mathrm{FDP}}_{\mathcal{A}}\left({\mathbf{t}}_2\right)\right|+O\left({\pi }_{np}\right)=O\left({\rho }_{np}\right)+O\left({\pi }_{np}\right). \end{gathered}$$

This completes the proof of Theorem 1.

A.2. Proof of Theorem 2

By the definition of the LCD statistics, we construct the augmented Lasso estimator for each θ ∈ Θ_np, which is defined as

$${\widehat{\mathit{\beta }}}^{\text{aug}}(\mathit{\theta })=\arg \underset{\mathbf{b}\in {ℝ}^{2p}}{\min }\Vert \mathbf{y}-[\mathbf{X},\tilde{\mathbf{X}}(\mathit{\theta })]\mathbf{b}{\Vert }_2^2+\lambda \Vert \mathbf{b}{\Vert }_1.$$ (19)

The Lasso estimator of regressing y on only X is also given by

$$\widehat{\mathit{\beta }}=\arg \underset{\mathbf{b}\in {ℝ}^p}{\min }\Vert \mathbf{y}-\mathbf{X}b{\Vert }_2^2+\lambda \Vert \mathbf{b}{\Vert }_1,$$ (20)

where λ = O(n^−1/2 log p). According to the true model ${\mathcal{S}}^0$, the underlying true parameter vector corresponding to ${\widehat{\mathit{\beta }}}^{\text{aug}}(\mathit{\theta })$ should be given by β^aug := (β′, 0′)′ ∈ ℝ^2p with $\beta ={\left({\beta }_{{\mathcal{S}}^0}^{\prime },{\mathbf{0}}^{\prime }\right)}^{\prime }\in {ℝ}^p$ and $\left|{\mathcal{S}}^0\right|=s$ for any θ ∈ Θ_np. By Lemma 5 in Section C.6, with probability at least 1 − O(π_np) the Lasso estimators satisfy

$$\underset{\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\widehat{\mathit{\beta }}}^{\text{aug}}(\mathit{\theta })-{\mathit{\beta }}^{\text{aug}}\Vert }_1=O(s\lambda ),\Vert \widehat{\mathit{\beta }}-\mathit{\beta }{\Vert }_1=O(s\lambda ),$$

where λ = O(n^−1/2 log p).

We now prove that under Condition 7, the power of the augmented Lasso (19) is bounded from below by γ ∈ [0, 1]; that is,

$$\mathbb{E}\left|{\widehat{\mathcal{S}}}_{\text{auglasso}}\cap {\mathcal{S}}^0\right|/s\ge \gamma ,$$ (21)

where ${\widehat{\mathcal{S}}}_{\text{auglasso}}=\{j:{\widehat{\mathit{\beta }}}_j^{\text{aug}}(\mathit{\theta })\ne 0\}$. To this end, we first show that with asymptotic probability one,

$$\left|{\widehat{\mathcal{S}}}_{\text{auglasso}}^c\cap {\mathcal{S}}^0\right|/s|\le 1-\gamma .$$ (22)

The key is to use proof by contradiction. Suppose $\left|{\widehat{\mathcal{S}}}_{\text{auglasso}}^c\cap {\mathcal{S}}^0\right|/s>1-\gamma$. Then we can see that

$$\begin{gathered} \underset{\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\widehat{\mathit{\beta }}}^{\text{aug}}(\mathit{\theta })-{\beta }^{\text{aug}}\Vert }_1\ge \underset{\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\widehat{\mathit{\beta }}}_{{\widehat{\mathcal{S}}}_{\text{auglass}}^{\text{c}}}^{\mathrm{aug}}(\mathit{\theta })-{\beta }_{{\widehat{\mathcal{S}}}_{\text{auglass}}^{\text{c}}}^{\mathrm{aug}}\Vert }_1 \\ ={\Vert {\beta }_{{\widehat{\mathcal{S}}}_{\text{auglass}}^{\text{c}}}^{\mathrm{aug}}\Vert }_1\ge {\Vert {\beta }_{{\widehat{\mathcal{S}}}_{\text{auglass}}^{\text{c}}\cap {\mathcal{S}}^0}^{\mathrm{aug}}\Vert }_1>b_{np}s{n}^{-1/2}\log p, \end{gathered}$$

where the last step is by Condition 7. However, by Lemma 5 with probability at least 1−O(π_np), the left hand side above is bounded from above by O(sλ) with λ = O(n^−1/2 log p). These two results contradict with each other since b_np → ∞. Hence (22) is proved. Therefore, the result in (21) follows immediately since $\left|{\widehat{\mathcal{S}}}_{\text{auglasso}}\cap {\mathcal{S}}^0\right|=s-\left|{\widehat{\mathcal{S}}}_{\text{auglasso}}^c\cap {\mathcal{S}}^0\right|$ and

$$\begin{gathered} \mathbb{E}\left|{\widehat{\mathcal{S}}}_{\text{auglasso}}\cap {\mathcal{S}}^0\right|/s\ge \gamma \mathbb{P}\left(\left|{\widehat{\mathcal{S}}}_{\text{auglasso}}\cap {\mathcal{S}}^0\right|/s>\gamma \right) \\ =\gamma \mathbb{P}\left(\left|{\widehat{\mathcal{S}}}_{\text{auglasso}}^c\cap {\mathcal{S}}^0\right|/s\le 1-\gamma \right)=\gamma \left(1-O\left({\pi }_{np}\right)\right). \end{gathered}$$

Let ${\widehat{\mathcal{S}}}_{\text{lasso}}=\left\{j:{\widehat{\mathit{\beta }}}_j\ne 0\right\}$. Using the same argument, we can show that the power of the Lasso (20) is also bounded from below by γ(1−O(π_np)) under Condition 7. That is, we have

$$\mathbb{E}\left|{\widehat{\mathcal{S}}}_{\text{lasso}}\cap {\mathcal{S}}^0\right|/s\ge \gamma \left(1-O\left({\pi }_{np}\right)\right).$$

Next we show that our knockoffs procedure has at least the same power as the augmented Lasso and hence the Lasso itself. Namely, we prove

$$\mathbb{E}\left|{\widehat{\mathcal{S}}}_{\text{lasso}}\cap {\mathcal{S}}^0\right|/s\ge \gamma$$ (23)

with threshold T₂. Note that the same argument is still valid for T₁. Let |W₍₁₎| ≥ ⋯ ≥ |W_(p)| and define j^∗ as $\left|W_{\left(j^{*}\right)}\right|=T_2$. Then by the definition of T₂, it holds that $-T_2<W_{j^{*}+1}\le 0$. Here we have assumed that there are no ties on the magnitudes of W_j’s which should be a reasonable assumption considering the continuity of the Lasso solution. As in the proof of Theorem 3 in [26], it is sufficient to consider the following two cases.

Case 1

Consider the case of $-T_2<W_{j^{*}+1}<0$. In this case, from the definition of threshold T₂ we have

$$\frac{2+\left|\left\{j:W_{(j)}\le -T_2\right\}\right|}{\left|\left\{j:W_{(j)}\ge T_2\right\}\right|}>q.$$

Using the same argument as in Lemma 6 of [26] together with Lemma 5, we can prove from Condition 8 that $|\widehat{\mathcal{S}}|\ge C_{2}s$ with probability at least 1 − O(π_np). This leads to |{j : W_(j) ≤ −T₂}| > C₂qs−2 with the same probability. Now from the same argument as in A.5 of [26], we can obtain T₂ = O(λ). On the other hand, Lemma 5 and some algebra establish that

$$\begin{gathered} O(s\lambda )={\Vert {\widehat{\mathit{\beta }}}^{\text{aug}}(\widehat{\mathit{\theta }})-{\mathit{\beta }}^{\text{aug}}\Vert }_1=\sum _{j=1}^p\left|{\widehat{\beta }}_j^{\text{aug}}(\widehat{\mathit{\theta }})-{\beta }_j\right|+\sum _{j=1}^p\left|{\widehat{\beta }}_{j+p}^{\text{aug}}(\widehat{\mathit{\theta }})\right| \\ =\sum _{j\in {\widehat{\mathcal{S}}}^c\cap {\mathcal{S}}^0}\left|{\widehat{\beta }}_j^{\text{aug}}(\widehat{\mathit{\theta }})-{\beta }_j\right|+\sum _{j\in {\mathcal{S}}^1}\left|{\widehat{\beta }}_j^{\text{aug}}(\widehat{\mathit{\theta }})\right|+\sum _{j\in {\widehat{\mathcal{S}}}^c\cap S^0}\left|{\widehat{\beta }}_j^{\text{aug}}(\widehat{\mathit{\theta }})-{\beta }_j\right|+\sum _{j=1}^p\left|{\widehat{\beta }}_{j+p}^{\text{aug}}(\widehat{\mathit{\theta }})\right|. \end{gathered}$$ (24)

We then consider the lower bound of the last term in (24). For any $j\in {\widehat{S}}^c$, it holds that $\left|{\widehat{\beta }}_{j+p}^{\text{aug}}(\widehat{\mathit{\theta }})\right|>\left|{\widehat{\beta }}_j^{\text{aug}}(\widehat{\mathit{\theta }})\right|-T_2$. Hence we obtain

$$\begin{gathered} \sum _{j=1}^p\left|{\widehat{\beta }}_{j+p}^{\text{aug}}(\widehat{\mathit{\theta }})\right|\ge \sum _{j\in \widehat{\mathcal{S}}\cap {\mathcal{S}}^0}\left|{\widehat{\beta }}_{j+p}^{\text{aug}}(\widehat{\mathit{\theta }})\right|+\sum _{j\in {\widehat{\mathcal{S}}}^{\text{c}}\cap {\mathcal{S}}^0}\left|{\widehat{\beta }}_{j+p}^{\text{aug}}(\widehat{\mathit{\theta }})\right| \\ \ge \sum _{j\in \widehat{\mathcal{S}}\cap S^0}\left|{\widehat{\beta }}_{j+p}^{\mathrm{aug}}(\widehat{\mathit{\theta }})\right|+\sum _{j\in {\widehat{\mathcal{S}}}^c\cap {\mathcal{S}}^0}\left|{\widehat{\beta }}_j^{\mathrm{aug}}(\widehat{\mathit{\theta }})\right|-T_2\left|{\widehat{\mathcal{S}}}^c\cap {\mathcal{S}}^0\right|. \end{gathered}$$ (25)

Plugging (25) into (24) and applying the triangle inequality yield

$$\begin{gathered} O(s\lambda )\ge \sum _{j\in \widehat{\mathcal{S}}\cap {\mathcal{S}}^0}\left|{\widehat{\beta }}_{j+p}^{\mathrm{aug}}(\widehat{\mathit{\theta }})-{\beta }_j\right|+\sum _{j\in {\mathcal{S}}^1}\left|{\widehat{\beta }}_j^{\text{aug}}(\widehat{\mathit{\theta }})\right|+\sum _{j\in {\widehat{\mathcal{S}}}^{\text{c}}\cap {\mathcal{S}}^0}\left|{\widehat{\beta }}_j^{\text{aug}}(\widehat{\mathit{\theta }})-{\beta }_j\right| \\ +\sum _{j\in \widehat{\mathcal{S}}\cap {\mathcal{S}}^0}\left|{\widehat{\beta }}_{j+p}^{\mathrm{aug}}(\widehat{\mathit{\theta }})\right|+\sum _{j\in {\widehat{\mathcal{S}}}^c\cap {\mathcal{S}}^0}\left|{\widehat{\beta }}_j^{\mathrm{aug}}(\widehat{\mathit{\theta }})\right|-T_2\left|{\widehat{\mathcal{S}}}^c\cap {\mathcal{S}}^0\right| \\ \ge \sum _{j\in \widehat{\mathcal{S}}\cap {\mathcal{S}}^0}\left|{\widehat{\beta }}_j^{\mathrm{aug}}(\widehat{\mathit{\theta }})-{\beta }_j\right|+\sum _{j\in {\mathcal{S}}^1}\left|{\widehat{\beta }}_j^{\text{aug}}(\widehat{\mathit{\theta }})\right| \\ +\sum _{j\in {\widehat{\mathcal{S}}}^c\cap {\mathcal{S}}^0}\left|{\beta }_j\right|+\sum _{j\in \widehat{\mathcal{S}}\cap S^0}\left|{\widehat{\beta }}_{j+p}^{\text{aug}}(\widehat{\mathit{\theta }})\right|-T_2\left|{\widehat{\mathcal{S}}}^c\cap {\mathcal{S}}^0\right| \\ \ge \sum _{j\in {\widehat{\mathcal{S}}}^c\cap S^0}\left|{\beta }_j\right|-T_2\left|{\widehat{\mathcal{S}}}^c\cap {\mathcal{S}}^0\right|={\Vert {\beta }_{{\widehat{\mathcal{S}}}^c\cap {\mathcal{S}}^0}\Vert }_1-T_2\left|{\widehat{\mathcal{S}}}^c\cap {\mathcal{S}}^0\right|. \end{gathered}$$

Since $T_2\left|{\widehat{\mathcal{S}}}^c\cap {\mathcal{S}}^0\right|=O(s\lambda )$ for λ = O(n^−1/2 log p) due to the discussion above, we obtain

$${\Vert {\mathit{\beta }}_{{\widehat{\mathcal{S}}}^c\cap {\mathcal{S}}^0}\Vert }_1=O\left(s{n}^{-1/2}\log p\right).$$ (26)

Suppose $\left|{\widehat{\mathcal{S}}}^c\cap {\mathcal{S}}^0\right|/s>1-\gamma$. Then Condition 7 gives ${\Vert {\beta }_{{\widehat{\mathcal{S}}}^c\cap {\mathcal{S}}^0}\Vert }_1>b_{np}s{n}^{-1/2}\log p$ for some positive diverging sequence b_np; this contradicts with (26). Thus we obtain $\left|{\widehat{\mathcal{S}}}^c\cap {\mathcal{S}}^0\right|/s\le 1-\gamma$ with asymptotic probability one, which leads to (23) by taking expectation.

Case 2

Consider the case of $W_{\left(j^{*}+1\right)}=0$. In this case, by the definition of threshold T₂

$$\frac{1+\left|\left\{j:W_{(j)}<0\right\}\right|}{\left|\left\{j:W_{(j)}>0\right\}\right|}\le q.$$ (27)

If |{j : W_(j) < 0}| > C₃s for some constant C₃ > 0, then from the same argument as in A.5 of [26], we can obtain T₂ = O(λ), and the rest of the proof is the same as in Case 1. On the other hand, if |{j : W_(j) < 0}| ≤ o(s) we have

$$\left|\left\{j:W_{(j)}\ne 0\right\}\cap {\mathcal{S}}^0\right|=\left|\left\{j:W_{(j)}>0\right\}\cap {\mathcal{S}}^0\right|+\left|\left\{j:W_{(j)}<0\right\}\cap {\mathcal{S}}^0\right|\le \left|\widehat{\mathcal{S}}\cap {\mathcal{S}}^0\right|+o(s).$$

Now note that $|\{j:W_{(j)}\ne 0\}|\ge |\{j:|{\widehat{\beta }}_j^{\text{aug}}|\ne 0,j=1,\cdots ,p\}|$. Then we can see that with asymptotic probability one,

$$\begin{gathered} \left|\left\{j:W_{(j)}\ne 0\right\}\cap {\mathcal{S}}^0\right|\ge \left|\left\{j:{\widehat{\beta }}_j^{\text{aug}}\ne 0,j=1,\cdots ,p\right\}\cap {\mathcal{S}}^0\right| \\ =\left|{\widehat{\mathcal{S}}}_{\text{auglasso}}\cap {\mathcal{S}}^0\right|\ge \gamma s(1-o(1)). \end{gathered}$$

Consequently, we obtain $\left|\widehat{\mathcal{S}}\cap {\mathcal{S}}^0\right|/s\ge \gamma (1-o(1))$, which leads to (23) by taking expectation. Combining these two cases concludes the proof of Theorem 2.

B. Review of model-X knockoffs framework

The key idea of the model-X knockoffs framework is to construct the so-called model-X knockoff variables, which concept was introduced originally in [15] and whose definition is stated formally as follows for completeness.

Definition 1 (Model-X knockoff variables [15])

For a set of random variables x = (X₁,⋯, X_p), a new set of random variables $\tilde{\mathbf{x}}=\left({\tilde{X}}_1,\cdots ,{\tilde{X}}_p\right)$ is called a set of model-X knockoff variables if it satisfies the following properties:

1. For any subset $\mathcal{S}\subset \{1,\cdots ,p\}$, we have ${[\mathbf{x},\tilde{\mathbf{x}}]}_{\text{swap}(\mathcal{S})}\stackrel{d}{=}[\mathbf{x},\tilde{\mathbf{x}}]$, where $\stackrel{d}{=}$ denotes equal in distribution and the vector ${[\mathbf{x},\tilde{\mathbf{x}}]}_{\text{swap}(\mathcal{S})}$ is obtained by swapping X_j and ${\tilde{X}}_j$ for each $j\in \mathcal{S}$.

2. Conditional on x, the knockoffs vector $\tilde{\mathbf{x}}$ is independent of response Y.

An important consequence is that the null regressors $\left\{X_j:j\in {\mathcal{S}}^1\right\}$ can be swapped with their knockoffs without changing the joint distribution of the original variables x, their knockoffs $\tilde{\mathbf{x}}$, and response Y. That is, we can obtain for any $\mathcal{S}\subset {\mathcal{S}}^1$,

$$\left({[\mathbf{x},\tilde{\mathbf{x}}]}_{\text{swap}(\mathcal{S})},Y\right)\stackrel{d}{=}([\mathbf{x},\tilde{\mathbf{x}}],Y).$$ (A.1)

Such a property is known as the exchangeability property using the terminology in [15]. For more details, see Lemma 3.2 therein. Following [15], one can obtain a knockoffs matrix $\tilde{\mathbf{X}}\in {ℝ}^{n\times p}$ given observed design matrix X.

Using the augmented design matrix $[\mathbf{X},\tilde{\mathbf{X}}]$ and response vector y constructed by stacking the n observations, [15] suggested constructing knockoff statistics $W_j=w_j([\mathbf{X},\tilde{\mathbf{X}}],\mathbf{y})$, j ∈ {1,⋯, p}, for measuring the importance of the jth variable, where w_j is some function that satisfies the property that swapping x_j ∈ ℝⁿ with its corresponding knockoff variable ${\tilde{\mathbf{x}}}_j\in {ℝ}^n$ changes the sign of W_j; that is,

$$w_j({[\mathbf{X},\tilde{\mathbf{X}}]}_{\text{Swap}(\mathcal{S})},\mathbf{y})=\{\begin{array}{ll}{w}_j([\mathbf{X},\tilde{\mathbf{X}}],\mathbf{y}),& j\notin \mathcal{S},\\ -w_j([\mathbf{X},\tilde{\mathbf{X}}],\mathbf{y}),& j\in \mathcal{S}.\end{array}$$ (A.2)

The knockoff statistics constructed above $W_j=-w_j([\mathbf{X},\tilde{\mathbf{X}}],\mathbf{y})$ satisfy the so-called sign-flip property; that is, conditional on |W_j|’s the signs of the null W_j’s with $j\notin {\mathcal{S}}^0$ are i.i.d. coin flips (with equal chance 1/2). For the examples on valid constructions of knockoff statistics, see [15].

Let t > 0 be a fixed threshold and define $\widehat{\mathcal{S}}=\left\{j:W_j\ge t\right\}$ as the set of discovered variables. Then intuitively, the sign-flip property entails

$$\left|\widehat{\mathcal{S}}\cap {\mathcal{S}}^1\right|\stackrel{d}{=}\left|\left\{j:W_j\le -t\right\}\cap {\mathcal{S}}^1\right|\le \left|\left\{j:W_j\le -t\right\}\right|.$$

Therefore, the FDP function can be estimated (conservatively) as

$$\text{FDP}=\frac{\left|\widehat{\mathcal{S}}\cap {\mathcal{S}}^1\right|}{\left|\widehat{\mathcal{S}}\right|\vee 1}\le \frac{\left|\left\{j:W_j\le -t\right\}\right|}{\left|\widehat{\mathcal{S}}\right|\vee 1}=:\widehat{\text{FDP}}$$

for each t. In light of this observation, [15] proposed to choose the threshold by resorting to the above $\widehat{\text{FDP}}$. Their results are summarized formally as follows.

Result 1 ([15])

Let q ∈ (0,1) denote the target FDR level. Assume that we choose a threshold T₁ > 0 such that

$$T_1=\min \left\{t>0:\frac{\left|\left\{j:W_j\le -t\right\}\right|}{\left|\left\{j:W_j\ge t\right\}\right|\vee 1}\le q\right\}$$

or T₁ = +∞ if the set is empty. Then the procedure selecting the variables $\widehat{\mathcal{S}}=\left\{j:W_j\ge T_1\right\}$ controls the mFDR in (4) to no larger than q. Moreover, assume that we choose a slightly more conservative threshold T₂ > 0 such that

$$T_2=\min \left\{t>0:\frac{1+\left|\left\{j:W_j\le -t\right\}\right|}{\left|\left\{j:W_j\ge t\right\}\right|\vee 1}\le q\right\}$$

or T₂ = +∞ if the set is empty. Then the procedure selecting the variables $\widehat{\mathcal{S}}=\left\{j:W_j\ge T_2\right\}$ controls the FDR in (3) to no larger than q.

It is worth noting that Result 1 was derived under the assumption that the joint distribution of the p covariates is known. In our model setting (1) and (2), however there exist unknown parameters that need to be estimated from data. In such case, it is natural to construct the knockoff variables and knockoff statistics with estimated distribution of the p covariates. Such a plug-in principle usually leads to breakdown of the exchangeability property in Definition 1, preventing us from using directly Result 1. To address this challenging issue, we will introduce our new method in the next section and provide detailed theoretical analysis for it.

It is also worth mentioning that recently, [6] provided an elegant new line of theory which ensures FDR control of model-X knockoffs procedure under the approximate exchangeability assumption, which is weaker than the exact exchangeability condition required in Definition 1. However, the conditions they need on estimation error of the joint distribution of x is difficult to be satisfied in high dimensions. [26] investigated the robustness of model-X knockoffs procedure with respect to unknown covariate distribution when covariates x follow a joint Gaussian distribution. Their procedure needs data splitting and their proofs rely heavily on the Gaussian distribution assumption, and thus their development may not be suitable for economic data with limited sample size and heavy-tailed distribution. For these reasons, our results complement substantially those in [15], [26], and [6].

C. Proofs of Proposition 1 and some key lemmas

C.1. Proof of Proposition 1

Observe that the second property of Definition 1 holds naturally since $\tilde{\mathbf{X}}\left({\mathit{\theta }}^0\right)$ is constructed without using the information of y. Thus it remains to verify the first property of Definition 1. Since F⁰ and ${\mathbf{E}}_{{\eta }^0}$ have i.i.d. rows, let us consider the case of a single observation and show that ${\left[\text{x},\tilde{\mathbf{x}}\left({\mathit{\theta }}^0\right)\right]}_{\text{swap}(\mathcal{S})}\stackrel{d}{=}\left[\text{x},\tilde{\mathbf{x}}\left({\mathit{\theta }}^0\right)\right]$ for any subset $\mathcal{S}\subset \{1,\cdots ,p\}$. By Proposition 2 of [15], it suffices to consider the case of $\mathcal{S}=\left\{j\right\}$ for an arbitrary j ∈ {1,⋯, p}. It follows from the definition of model (2) and the construction of $\tilde{\mathbf{x}}\left({\mathit{\theta }}^0\right)$ that

$$\begin{gathered} {\left[\mathrm{x},\tilde{x}\left({\mathit{\theta }}^0\right)\right]}_{\mathrm{swap}(\{j\})}={\left[{\mathbf{c}}^0+\mathbf{e},{\mathbf{c}}^0+{\mathbf{e}}_{\eta }0\right]}_{\mathrm{swap}(\{j\})} \\ =\left[{\mathbf{c}}^0+{\tilde{\mathbf{e}}}^{(j)},{\mathbf{c}}^0+{\tilde{\mathbf{e}}}_{{\eta }^0}^{(j)}\right], \end{gathered}$$ (A.3)

where ${\tilde{\mathbf{e}}}^{(j)}$ and ${\tilde{\mathbf{e}}}_{{\eta }^0}^{(j)}$ are defined such that ${\left[\mathbf{e},{\mathbf{e}}_{{\eta }^0}\right]}_{\text{swap}}(\{j\})=\left[{\tilde{\mathbf{e}}}^{(j)},{\tilde{\mathbf{e}}}_{{\eta }^0}^{(j)}\right]$. Since model (2) assumes that e has i.i.d. components and ${\mathbf{e}}_{{\eta }^0}$ is an independent copy of e, it holds that

$$\left[{\tilde{\mathbf{e}}}^{(j)},{\tilde{\mathbf{e}}}_{{\eta }^0}^{(j)}\right]\stackrel{d}{=}\left[\mathbf{e},{\mathbf{e}}_{{\eta }^0}\right].$$ (A.4)

Therefore, in view of (A.3) and (A.4) and the independence between $\left(\mathbf{e},{\mathbf{e}}_{{\eta }^0}\right)$ and c⁰, we have

$$\begin{gathered} {\left[\mathbf{x},\tilde{\mathbf{x}}\left({\mathit{\theta }}^0\right)\right]}_{\text{swap}(\{j\})}\stackrel{d}{=}\left[{\mathbf{c}}^0+\mathbf{e},{\mathbf{c}}^0+{\mathbf{e}}_{{\eta }^0}\right] \\ =\left[\mathbf{x},\tilde{\mathbf{x}}\left({\mathit{\theta }}^0\right)\right], \end{gathered}$$

which completes the proof of Proposition 1.

C.2. Proof of Lemma 1

For λ fixed at C₀n^−1/2 log p and each given θ, $W_j(\mathit{\theta })=w_j([\mathbf{X},\tilde{\mathbf{X}}(\mathit{\theta })],\mathbf{y})$ depends only on ${\widehat{\beta }}^{aug}(\mathit{\theta })$ by the LCD construction. Moreover, the Lasso solution ${\widehat{\beta }}^{aug}(\mathit{\theta })$ satisfies the Karush–Kuhn–Tucker (KKT) conditions:

$$\mathbf{v}(\mathit{\theta })-\mathbf{U}(\mathit{\theta }){\widehat{\beta }}^{aug}(\mathit{\theta })=n^{-1}\lambda \mathbf{z},$$ (A.5)

$$\text{where}\mathbf{z}={\left(z_1,\cdots ,z_{2p}\right)}^T\text{with}{z}_j\in \{\begin{array}{ll}\left\{\mathrm{sgn}\left({\widehat{\beta }}_j\right)\right\}& \text{if}{\widehat{\beta }}_j\ne 0,\\ [-1,1]& \text{if}{\widehat{\beta }}_j=0,\end{array}\text{for}j=1,\cdots ,2p.$$ (A.6)

This means that ${\widehat{\beta }}^{aug}(\mathit{\theta })$ depends on the data $([\mathbf{X},\tilde{\mathbf{X}}(\mathit{\theta })],\mathbf{y})$ only through U(θ) and v(θ). Thus using notation T(θ) = vec(vechU(θ),v(θ)) with the fact that U(θ) is symmetric, we can reparametrize $w_j([\mathbf{X},\tilde{\mathbf{X}}(\mathit{\theta })],\mathbf{y})$ as w_j(T(θ)) with a slight abuse of notation. Furthermore, note that the thresholds T₁ and T₂ are both completely determined by w_j(T(θ)). Consequently, by the construction of $\widehat{\mathcal{S}}$ we can see that $\widehat{\mathcal{S}}$ depends only on T(θ), which completes the proof of Lemma 1.

C.3. Proof of Lemma 2

We continue to use the same λ and θ as in Lemma 1 and its proof. Recall that $S_{\mathcal{A}}\left({\mathbf{t}}_{\mathcal{A}}\right)$ represents the outcome of first restricting ourselves to the smaller set of variables $\mathcal{A}$ and then applying IPAD to ${\mathbf{T}}_{\mathcal{A}}(\mathit{\theta })={\mathbf{t}}_{\mathcal{A}}$ to further select variables from $\mathcal{A}$. Also recall that $\mathcal{A}\ast (\mathit{\theta })$ is the support of knockoff statistics W_j(θ). Thus the knockoff threshold T₁ or T₂ depends only on W_j(θ) with $j\in \mathcal{A}\ast (\mathit{\theta })$.

On the other hand, when we restrict ourselves to $\mathcal{A}\supset \mathcal{A}\ast (\mathit{\theta })$ we solve the following KKT conditions with respect to $\tilde{\beta }:={\left({\tilde{\beta }}_1,\cdots ,{\tilde{\beta }}_{2|\mathcal{A}|}\right)}^T\in {ℝ}^{2|\mathcal{A}|}$ to get the Lasso solution:

$$\tilde{\beta }={\left({\mathbf{U}}_{\mathcal{A}}{(\mathit{\theta })}^{\prime }{\mathbf{U}}_{\mathcal{A}}(\mathit{\theta })\right)}^{-1}\left({\mathbf{v}}_{\mathcal{A}}(\mathit{\theta })-n^{-1}\lambda \tilde{\mathbf{z}}\right),$$ (A.7)

$$\text{where}\tilde{\mathbf{z}}={\left({\tilde{z}}_1,\cdots ,{\tilde{z}}_{2\left|\mathcal{A}\right|}\right)}^T\text{with}{\tilde{z}}_j\in \{\begin{array}{ll}\left\{\mathrm{sgn}\left({\tilde{\beta }}_j\right)\right\}& \text{if}{\tilde{\beta }}_j\ne 0,\\ [-1,1]& \text{if}{\tilde{\beta }}_j=0,\end{array}\text{for}j=1,\cdots ,2|\mathcal{A}|.$$ (A.8)

Since λ is always fixed at the same value C₀n^−1/2 log p, it is seen that the solution to the above KKT conditions is identical to ${\widehat{\beta }}_{\mathcal{A}\mathcal{A}}^{\mathrm{aug}}(\mathit{\theta })$, where the latter denotes the subvector of ${\widehat{\beta }}^{aug}(\mathit{\theta })$ formed by stacking ${\widehat{\beta }}_{j1}^{\mathrm{aug}}(\mathit{\theta })$, $j_1\in \mathcal{A}$ and ${\widehat{\beta }}_{p+j2}^{\mathrm{aug}}(\mathit{\theta })$, $j_2\in \mathcal{A}$ all together. Therefore, the Lasso solution to (A.7)–(A.8) and the Lasso solution to (A.5)–(A.6) have the identical support (when viewed in the original 2p-dimensional space) and in addition, identical values on the support. This guarantees that S_{1,⋯,p}(T(θ)) and $S_{\mathcal{A}}\left({\mathbf{T}}_{\mathcal{A}}(\mathit{\theta })\right)$ are identical and thus concludes the proof of Lemma 2.

C.4. Lemma 3 and its proof

Lemma 3

Assume that Conditions 2–5 hold. Then with probability at least 1−O(π_np), the estimator $\widehat{\mathit{\theta }}={(\mathrm{vec}{(\widehat{\mathbf{C}})}^{\prime },{\widehat{\mathit{\eta }}}^{\prime })}^{\prime }$ lies in the shrinking set given by

$${\mathbf{\Theta }}_{np}=\{\mathit{\theta }={\left(\mathrm{vec}{(\mathbf{C})}^{\prime },{\eta }^{\prime }\right)}^{\prime }:{\Vert \mathbf{C}-{\mathbf{C}}^0\Vert }_{\max }+{\Vert \eta -{\eta }^0\Vert }_{\max }\le O\left(c_{np}\right)\},$$

where c_np = (n⁻¹ log p)^1/2 + (p⁻¹ log n)1/2 and π_np = p^−ν + n^−ν.

Proof. We divide the proof into two parts. We prove the bound for ${\Vert \widehat{\mathbf{C}}-{\mathbf{C}}^0\Vert }_{\max }$ in Part 1 and then for ${\Vert \widehat{\mathit{\eta }}-{\eta }^0\Vert }_{\max }$ in Part 2.

Part 1

Note that ${\Vert \widehat{\mathbf{C}}-{\mathbf{C}}^0\Vert }_{\max }={\max }_{i,j}\left|{\widehat{c}}_{ij}-c_{ij}^0\right|$, where the maximum is taken over i ∈ {1,⋯, n} and j ∈ {1,⋯, p}. We write ${\mathbf{f}}_i^{*}={\mathbf{H}}^{\prime }{\mathbf{f}}_i^0$ and ${\lambda }_j^{*}={\mathbf{H}}^{-1}{\lambda }_j^0$ with rotation matrix H defined in Lemma 6 in Section D.1. From the definition of c_ij, it holds that

$${\widehat{c}}_{ij}-c_{ij}^0={\left({\widehat{\mathbf{f}}}_i-{\mathbf{f}}_i^{*}\right)}^{\prime }{\lambda }_j^{*}+{\widehat{\mathbf{f}}}_i^{\prime }\left({\widehat{\mathit{\lambda }}}_j-{\lambda }_j^{*}\right).$$

From Lemma 6, we can assume ‖H‖₂ + ‖H⁻¹‖₂ + ‖V‖₂ + ‖V⁻¹‖₂ ≲ 1, which occurs with probability at least 1 − O(p^−ν). We also have ${\max }_{i\in \{1,\cdots ,n\}}{\Vert {\widehat{\mathbf{f}}}_i\Vert }_2^2\lesssim 1$ a.s. by the assumed restriction ${\widehat{\mathbf{F}}}^{\prime }\widehat{\mathbf{F}}/n={\mathbf{I}}_r$ as mentioned on p.213 of [3]. Hence, the triangle and Cauchy–Schwarz inequalities with Conditions 2 and 3 give

$$\begin{gathered} \underset{i,j}{\max }\left|{\widehat{c}}_{ij}-c_{ij}\right|\le \underset{i}{\max }{\Vert {\widehat{\mathbf{f}}}_i-{\mathbf{f}}_i^{*}\Vert }_2\underset{j}{\max }{\Vert {\mathit{\lambda }}_j^{*}\Vert }_2+\underset{i}{\max }{\Vert {\widehat{\mathbf{f}}}_i\Vert }_2\underset{j}{\max }{\Vert {\widehat{\mathit{\lambda }}}_j-{\mathit{\lambda }}_j^{*}\Vert }_2 \\ \lesssim \underset{i}{\max }{\Vert {\widehat{\mathbf{f}}}_i-{\mathbf{f}}_i^{*}\Vert }_2+\underset{j}{\max }{\Vert {\widehat{\mathit{\lambda }}}_j-{\mathit{\lambda }}_j^{*}\Vert }_2. \end{gathered}$$ (A.9)

Then it is sufficient to derive upper bounds for max_i k^ˆf_i−f_i^∗k₂ and ${\max }_j{\Vert {\widehat{\mathit{\lambda }}}_j-{\mathit{\lambda }}_j^{*}\Vert }_2$ that hold with high probability. Using the decomposition of A.1 in [2] along with taking maximum over i, ℓ ∈ {1,⋯, n}, we can deduce

$$\begin{gathered} \underset{i}{\max }{\Vert {\widehat{\mathbf{f}}}_i-{\mathbf{f}}_i^{*}\Vert }_2 \\ \le {\Vert {\mathbf{V}}^{-1}\Vert }_2\underset{i}{\max }(\left({\sigma }_e^2/n\right){\Vert {\widehat{\mathbf{f}}}_i\Vert }_2+n^{-1}\sum _{\ell =1}^n{\Vert {\widehat{\mathbf{f}}}_{\ell }\Vert }_2\left|p^{-1}\sum _{j=1}^p\left(e_{\ell j}{e}_{ij}-\mathbb{E}\left[e_{\ell j}{e}_{ij}\right]\right)\right| \\ +n^{-1}\sum _{\ell =1}^n{\Vert {\widehat{\mathbf{f}}}_{\ell }{\mathbf{f}}_{\ell }^{0^{\prime }}\Vert }_2{\Vert p^{-1}\sum _{j=1}^p{\mathit{\lambda }}_j^0{e}_{ij}\Vert }_2+n^{-1}\sum _{\ell =1}^n{\Vert {\widehat{\mathbf{f}}}_{\ell }{\mathbf{f}}_i^{0^{\prime }}\Vert }_2{\Vert p^{-1}\sum _{j=1}^p{\lambda }_j^0{e}_{\ell j}\Vert }_2) \\ \lesssim O\left(n^{-1}\right)+\underset{i,\ell }{\max }\left|p^{-1}\sum _{j=1}^p\left(e_{\ell j}{e}_{ij}-\mathbb{E}\left[e_{\ell j}{e}_{ij}\right]\right)\right|+\underset{i}{\max }{\Vert p^{-1}\sum _{j=1}^p{\lambda }_j^0{e}_{ij}\Vert }_2 \\ \lesssim O\left(n^{-1}\right)+R_1+R_2, \end{gathered}$$ (A.10)

where we have used the boundedness of ${\Vert {\widehat{\mathbf{f}}}_{\ell }\Vert }_2$ discussed above and ${\Vert {\mathbf{f}}_{\ell }^0\Vert }_2\le r^{1/2}{\Vert {\mathbf{f}}_{\ell }^0\Vert }_{\max }\lesssim 1$ in Condition 2 for the second inequality, and defined $R_1={\max }_{i,\ell }\left|p^{-1}{\sum }_{j=1}^p\left(e_{\ell j}{e}_{ij}-\mathbb{E}\left[e_{\ell j}{e}_{ij}\right]\right)\right|$ and $R_2={\max }_{i,k}\left|p^{-1}{\sum }_{j=1}^p{\lambda }_{jk}^0{e}_{ij}\right|$. Similarly, the expression on p.165 of [2] with taking maximum over i ∈ {1,⋯, n} and j ∈ {1,⋯, p} leads to

$$\begin{gathered} \underset{j}{\max }{\Vert {\widehat{\mathit{\lambda }}}_j-{\mathit{\lambda }}_j^{*}\Vert }_2 \\ \le \Vert \mathbf{H}{\Vert }_2\underset{j}{\max }{\Vert n^{-1}\sum _{i=1}^n{\mathbf{f}}_i^0{e}_{ij}\Vert }_2+{\Vert n^{-1}\sum _{i=1}^n{\widehat{\mathbf{f}}}_i{\left({\widehat{\mathbf{f}}}_i-{\mathbf{f}}_i^{*}\right)}^{\prime }\Vert }_2{\Vert {\mathbf{H}}^{-1}\Vert }_2\underset{j}{\max }{\Vert {\mathit{\lambda }}_j^0\Vert }_2 \\ +\underset{j}{\max }{\Vert n^{-1}\sum _{i=1}^n\left({\widehat{\mathbf{f}}}_i-{\mathbf{f}}_i^{*}\right)e_{ij}\Vert }_2 \\ \lesssim \underset{j}{\max }{\Vert n^{-1}\sum _{i=1}^n{\mathbf{f}}_i^0{e}_{ij}\Vert }_2+\underset{i}{\max }{\Vert {\widehat{\mathbf{f}}}_i-{\mathbf{f}}_i^{*}\Vert }_2+\underset{i}{\max }{\Vert {\widehat{\mathbf{f}}}_i-{\mathbf{f}}_i^{*}\Vert }_2\underset{j}{\max }{\left(n^{-1}\sum _{i=1}^n{e}_{ij}^2\right)}^{1/2} \\ =R_3+\underset{i}{\max }{\Vert {\widehat{\mathbf{f}}}_i-{\mathbf{f}}_i^{*}\Vert }_2\left(1+R_4\right), \end{gathered}$$ (A.11)

where $R_3={\max }_{j,k}{\left|n^{-1}{\sum }_{i=1}^n{f}_{ik}^0{e}_{ij}\right|}_2$ and $R_4={\max }_j{\left(n^{-1}{\sum }_{i=1}^n{e}_{ij}^2\right)}^{1/2}$, and the Cauchy–Schwarz inequality has been used to obtain the second inequality. To evaluate R₄, we note that

$$R_4^2\le \underset{j}{\max }\mathbb{E}{e}_{ij}^2+\underset{j}{\max }\left|n^{-1}\sum _{i=1}^n(e_{ij}^2-\mathbb{E}{e}_{ij}^2)\right|.$$

The first term is bounded by $2C_e^2$. For the second term, Lemma 7(a) in Section D.2 with p replaced by n and the union bound give

$$\begin{gathered} \mathbb{P}\left(\underset{j}{\max }\left|n^{-1}\sum _{i=1}^n\left(e_{ij}^2-\mathbb{E}{e}_{ij}^2\right)\right|>u\right)\le p\underset{j}{\max }\mathbb{P}\left(\left|n^{-1}\sum _{i=1}^n\left(e_{ij}^2-\mathbb{E}{e}_{ij}^2\right)\right|>u\right) \\ \le 2p\exp \left(-n{u}^2/C\right) \end{gathered}$$

for all 0 ≤ u ≤ c. Thus putting u = (C(ν + 1)n⁻¹ log p)^1/2 and using condition c_np ≤ c/(r²M²C(ν + 2))^1/2, we obtain $R_4^2=O(1)+O\left({\left(n^{-1}\log p\right)}^{1/2}\right)=O(1)$ with probability at least 1 − O(p^−ν). This together with the observation from (A.9)–(A.11) yields

$$\begin{gathered} \underset{i,j}{\max }\left|{\widehat{c}}_{ij}-c_{ij}^0\right|\lesssim R_3+\left\{R_1+R_2+O\left(n^{-1}\right)\right\}\left(1+R_4\right) \\ \lesssim R_1+R_2+R_3+O\left(n^{-1}\right). \end{gathered}$$

Hence the convergence rate of ${\max }_{i,j}\left|{\widehat{c}}_{ij}-c_{ij}^0\right|$ is determined by the slowest term out of R₁, R₂, R₃, and O(n⁻¹). We evaluate these terms by Lemma 7 in Section D.2 and the union bound with condition c_np ≤ c/(r²M²C(ν + 2))^1/2 as above. First for R₁, Lemma 7(a) by letting u₁ = (C(ν + 2)p⁻¹ log n)^1/2 results in

$$\mathbb{P}\left(R_1>u_1\right)\le 2n^2\exp \left\{-p(\nu +2)p^{-1}\log n\right\}=O\left(n^{-\nu }\right).$$

Next for R₂, Lemma 7(c) with u₂ = (2(ν + 1)p⁻¹ logn)^1/2 gives

$$\mathbb{P}\left(R_2>u_2\right)\le 2rn\exp \left\{-p(\nu +1)p^{-1}\log n\right\}=O\left(n^{-\nu }\right).$$

Finally for R₃, Lemma 7(b) with putting u₃ = (C(ν + 1)n⁻¹ log p)^1/2 leads to

$$\mathbb{P}\left(R_3>u_3\right)\le 2rp\exp \left\{-n(\nu +1)n^{-1}\log p\right\}=O\left(p^{-\nu }\right).$$

Consequently, we obtain the first result ${\Vert \widehat{\mathbf{C}}-{\mathbf{C}}^0\Vert }_{\max }=O\left(c_{np}\right)$, which holds with probability at least 1 − O(π_np).

Part 2

Next we derive the convergence rate of $\widehat{\mathit{\eta }}$. It is sufficient to prove only the case when η⁰ is a scalar (so that we write ${\eta }^0={\eta }_1^0$) since dimensionality m is fixed and ${\eta }_k^0\text{'}\mathrm{s}$ share the identical property thanks to Condition 4. Recall notation ${\mathbb{E}}_{np}{e}^k={(np)}^{-1}{\sum }_{i,j}{e}_{ij}^k$. Letting ${\delta }_{ij}=c_{ij}^0-{\widehat{c}}_{ij}$, we have ${\widehat{e}}_{ij}=x_{ij}-{\widehat{c}}_{ij}=e_{ij}+{\delta }_{ij}$. For an arbitrary fixed k ∈ {1,⋯, m}, the binomial expansion entails

$$\begin{gathered} \left|{\mathbb{E}}_{np}{\widehat{e}}^k-\mathbb{E}{e}^k\right|=\left|{\mathbb{E}}_{np}{(e+\delta )}^k-\mathbb{E}{e}^k\right| \\ =\left|{\mathbb{E}}_{np}\left(e^k-\mathbb{E}{e}^k\right)+{\mathbb{E}}_{np}\sum _{\ell =0}^{k-1}\left(\begin{array}{l}k\\ \ell \end{array}\right)e^{\ell }{\delta }^{k-\ell }\right| \\ \le \left|{\mathbb{E}}_{np}\left(e^k-\mathbb{E}{e}^k\right)\right|+\sum _{\ell =0}^{k-1}\left(\begin{array}{l}k\\ \ell \end{array}\right)\underset{i,j}{\max }{\left|{\delta }_{ij}\right|}^{k-\ell }{\mathbb{E}}_{np}|e{|}^{\ell } \\ \lesssim \left|{\mathbb{E}}_{np}\left(e^k-\mathbb{E}{e}^k\right)\right|+O\left(\underset{i,j}{\max }\left|{\delta }_{ij}\right|\right)\sum _{\ell =0}^{k-1}{\mathbb{E}}_{np}|e{|}^{\ell }. \end{gathered}$$ (A.12)

For all k ∈ {1,⋯, m}, the strong law of large numbers with Theorem 2.5.7 in [22] entails $\left|{\mathbb{E}}_{np}{e}^k-\mathbb{E}{e}^k\right|=o\left({(np)}^{-1/2}\log (np)\right)$ a.s. under Condition 4. Furthermore, the second term of (A.12) is O(c_np) with probability at least 1 − O(π_np) from Part 1 and the same law of large numbers. Consequently, we obtain

$$\left|{\mathbb{E}}_{np}{\widehat{e}}^k-\mathbb{E}{e}^k\right|\lesssim c_{np}.$$

Therefore by the construction of ${\widehat{\mathit{\eta }}}_1$ and local Lipschitz continuity of h₁ in Condition 4, we see that

$$\begin{gathered} \left|{\widehat{\mathit{\eta }}}_1-{\eta }_1^0\right|=\left|h_1\left({\mathbb{E}}_{np}\widehat{e},\cdots ,{\mathbb{E}}_{np}{\widehat{e}}^m\right)-h_1\left(\mathbb{E}e,\cdots ,\mathbb{E}{e}^m\right)\right| \\ \lesssim \underset{k\in \{1,\cdots ,m\}}{\max }\left|{\mathbb{E}}_{np}{\widehat{e}}^k-\mathbb{E}{e}^k\right| \end{gathered}$$

with probability at least 1 − O(π_np). This completes the proof of Lemma 3.

C.5. Lemma 4 and its proof

Lemma 4

Assume that Conditions 1–4 hold. Then with probability at least 1−O(π_np), the following statements hold

$$\underset{\left|\mathcal{A}\right|\le k,\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\mathbf{U}}_{\mathcal{A}}(\mathit{\theta })-\mathbb{E}\left[{\mathbf{U}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)\right]\Vert }_{\max }=O\left(k^{1/2}{\tilde{c}}_{np}\right),$$

$$\underset{\left|\mathcal{A}\right|\le k,\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\mathbf{v}}_{\mathcal{A}}(\mathit{\theta })-\mathbb{E}\left[{\mathbf{v}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)\right]\Vert }_{\max }=O\left(s^{3/2}{\tilde{c}}_{np}\right),$$

where Θ_np was defined in Lemma 3 and ${\tilde{c}}_{np}=n^{-1/2}\log p+p^{-1/2}\log n$. Consequently, we have

$$\underset{\left|\mathcal{A}\right|\le k,\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\mathbf{T}}_{\mathcal{A}}(\mathit{\theta })-\mathbb{E}\left[{\mathbf{T}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)\right]\Vert }_{\max }=O\left(\left(k^{1/2}+s^{3/2}\right){\tilde{c}}_{np}\right).$$

Proof. To complete the proof of (a), we verify the following

$$(a-i)\underset{\left|\mathcal{A}\right|\le k,\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\mathbf{U}}_{\mathcal{A}}(\mathit{\theta })-{\mathbf{U}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)\Vert }_{\max }\lesssim k^{1/2}{\tilde{c}}_{np},$$

$$(a-ii){\Vert \mathbf{U}\left({\mathit{\theta }}^0\right)-\mathbb{E}\left[\mathbf{U}\left({\mathit{\theta }}^0\right)\right]\Vert }_{\max }\lesssim {\left(n^{-1}\log p\right)}^{1/2}.$$

From (a–i) and (a–ii), we can conclude that

$$\begin{gathered} \underset{\left|\mathcal{A}\right|\le k,\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\mathbf{U}}_{\mathcal{A}}(\mathit{\theta })-\mathbb{E}\left[{\mathbf{U}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)\right]\Vert }_{\max } \\ \le \underset{\left|\mathcal{A}\right|\le k,\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\mathbf{U}}_{\mathcal{A}}(\mathit{\theta })-{\mathbf{U}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)\Vert }_{\max }+\underset{\left|\mathcal{A}\right|\le k}{\sup }{\Vert {\mathbf{U}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)-\mathbb{E}\left[{\mathbf{U}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)\right]\Vert }_{\max } \\ \le \underset{\left|\mathcal{A}\right|\le k,\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\mathbf{U}}_{\mathcal{A}}(\mathit{\theta })-{\mathbf{U}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)\Vert }_{\max }+{\Vert \mathbf{U}\left({\mathit{\theta }}^0\right)-\mathbb{E}\left[\mathbf{U}\left({\mathit{\theta }}^0\right)\right]\Vert }_{\max } \\ \lesssim k^{1/2}{\tilde{c}}_{np}, \end{gathered}$$

which yields result (a).

We begin with showing (a–i); this is the uniform extension of Lemma 8(a) in Section D.3 over $|\mathcal{A}|\le k$. In fact, the proof is almost the same, with the only difference that bound (A.23) should be replaced with the bound derived in Lemma 9(c); that is,

$$\underset{\left|\mathcal{A}\right|\le k}{\max }{\Vert n^{-1/2}{\mathbf{E}}_{\mathcal{A}}\Vert }_2\lesssim 1\vee {\left(k{n}^{-1}\log p\right)}^{1/2},$$ (A.13)

which holds with probability at least 1 − O(p^−ν). Notice that (kn⁻¹ log p)^1/2 ≤ log^1/2 p. Therefore, even if we use (A.13) instead of (A.23) in the proof of Lemma 8(a) we can still derive the same convergence rate $k^{1/2}{\tilde{c}}_{np}$ as in Lemma 8(a), and hence (a–i) holds with probability at least 1 − O(π_np).

For (a–ii), we see that

$$\begin{gathered} {\Vert \mathbf{U}\left({\mathit{\theta }}^0\right)-\mathbb{E}\left[\mathbf{U}\left({\mathit{\theta }}^0\right)\right]\Vert }_{\max }\le {\Vert n^{-1}{\mathbf{X}}^{\prime }\mathbf{X}-\mathbb{E}\left[n^{-1}{\mathbf{X}}^{\prime }\mathbf{X}\right]\Vert }_{\max } \\ +{\Vert n^{-1}\tilde{\mathbf{X}}{\left({\mathit{\theta }}^0\right)}^{\prime }\tilde{\mathbf{X}}\left({\mathit{\theta }}^0\right)-\mathbb{E}\left[n^{-1}\tilde{\mathbf{X}}{\left({\mathit{\theta }}^0\right)}^{\prime }\tilde{\mathbf{X}}\left({\mathit{\theta }}^0\right)\right]\Vert }_{\text{max}} \\ +2{\Vert n^{-1}{\mathbf{X}}^{\prime }\tilde{\mathbf{X}}\left({\mathit{\theta }}^0\right)-\mathbb{E}\left[n^{-1}{\mathbf{X}}^{\prime }\tilde{\mathbf{X}}\left({\mathit{\theta }}^0\right)\right]\Vert }_{\max }=:W_1+W_2+2W_3. \end{gathered}$$ (A.14)

We derive the bounds for each of these terms. First, W₁ is bounded as

$$\begin{gathered} W_1\le {\Vert n^{-1}{\mathbf{C}}^{0^{\prime }}{\mathbf{C}}^0-\mathbb{E}\left[n^{-1}{\mathbf{C}}^{0^{\prime }}{\mathbf{C}}^0\right]\Vert }_{\max }+{\Vert n^{-1}{\mathbf{E}}^{\prime }\mathbf{E}-\mathbb{E}{n}^{-1}{\mathbf{E}}^{\prime }\mathbf{E}\Vert }_{\max }+2{\Vert n^{-1}{\mathbf{E}}^{\prime }{\mathbf{C}}^0\Vert }_{\max } \\ =:W_{1,1}+W_{1,2}+W_{1,3}. \end{gathered}$$

Under Condition 3, we deduce

$$\begin{gathered} W_{1,1}=\underset{j,\ell \in \{1,\cdots ,p\}}{\max }\left|\sum _{k,m=1}^r{\lambda }_{jk}^0{\lambda }_{\ell m}^0{n}^{-1}\sum _{i=1}^n\left(f_{ik}^0{f}_{im}^0-\mathbb{E}{f}_{ik}^0{f}_{im}^0\right)\right| \\ \le r{M}^2\underset{j,\ell \in \{1,\cdots ,p\}}{\max }\left|n^{-1}\sum _{i=1}^n\left(f_{ik}^0{f}_{im}^0-\mathbb{E}{f}_{ik}^0{f}_{im}^0\right)\right|. \end{gathered}$$

From Lemma 7(d) with Condition 2 and the union bound, we have

$$\begin{gathered} \mathbb{P}\left(\underset{j,\ell \in \{1,\cdots ,p\}}{\max }\left|n^{-1}\sum _{i=1}^n\left(f_{ik}^0{f}_{im}^0-\mathbb{E}{f}_{ik}^0{f}_{im}^0\right)\right|>u\right) \\ \le p^2\underset{j,\ell \in \{1,\cdots ,p\}}{\max }\mathbb{P}\left(\left|n^{-1}\sum _{i=1}^n\left(f_{ik}^0{f}_{im}^0-\mathbb{E}{f}_{ik}^0{f}_{im}^0\right)\right|>u\right)\le 2p^2\exp \left(-n{u}^2/C\right). \end{gathered}$$

Hence, letting u = (C(ν + 2)n⁻¹ log p)^1/2 above yields the bound W_1,1 ≲ (n⁻¹ log p)^1/2 with probability at least 1−O(p^−ν). Next for W_1,2, we can find from Lemma 7(a) with p replaced by n and the union bound that

$$\begin{gathered} \mathbb{P}\left({\Vert n^{-1}{\mathbf{E}}^{\prime }\mathbf{E}-\mathbb{E}{n}^{-1}{\mathbf{E}}^{\prime }\mathbf{E}\Vert }_{\max }>u\right)\le p^2\underset{j,\ell }{\max }\mathbb{P}\left(\left|n^{-1}\sum _{i=1}^n\left(e_{ij}{e}_{i\ell }-\mathbb{E}{e}_{ij}{e}_{i\ell }\right)\right|>u\right) \\ \le 2p^2\exp \left(-n{u}^2/C\right). \end{gathered}$$

Letting u = (C(ν + 2)n⁻¹ log p)^1/2 and using n⁻¹ log p ≤ c²/(C(ν + 2)), we obtain W_1,2 ≲ (n⁻¹ log p)^1/2 with probability at least 1 − O(p^−ν). Next for W_1,3, the union bound gives

$$\begin{gathered} \mathbb{P}\left({\Vert n^{-1}{\mathbf{E}}^{\prime }{\mathbf{F}}^0{\mathbf{\Lambda }}^{0^{\prime }}\Vert }_{\max }>u\right)=\mathbb{P}\left(\underset{j,\ell \in \{1,\cdots ,p\}}{\max }\left|n^{-1}\sum _{k=1}^r\sum _{i=1}^n{e}_{ij}{f}_{ik}^0{\lambda }_{\ell k}^0\right|>u\right) \\ \le \mathbb{P}\left(r\underset{j,\ell \in \{1,\cdots ,p\}}{\max }\underset{k\in \{1,\cdots ,r\}}{\max }\left|n^{-1}\sum _{i=1}^n{e}_{ij}{f}_{ik}^0\right|\left|{\lambda }_{\ell k}^0\right|>u\right) \\ \le rp\underset{k\in \{1,\cdots ,r\}}{\max }\underset{j\in \{1,\cdots ,p\}}{\max }\mathbb{P}\left(\left|n^{-1}\sum _{i=1}^n{e}_{ij}{f}_{ik}^0\right|>u/(rM)\right). \end{gathered}$$

Lemma 7(b) states that for all 0 ≤ u/(rM) ≤ c/(rM) it holds that

$$\mathbb{P}\left(\left|n^{-1}\sum _{i=1}^n{e}_{ij}{f}_{ik}^0\right|>u/(rM)\right)\le 2\exp \left\{-n{u}^2/\left(C{r}^2{M}^2\right)\right\}.$$

Therefore, if we put u = rM(C(ν + 1)n⁻¹ log p)^1/2 using n⁻¹ log p ≤ c²/(r²M²C(ν + 1)), the upper bound of the probability is further bounded by 2rp^−ν. Thus we obtain W₁₃ ≲ (n⁻¹ log p)^1/2 with probability at least 1 − O(p^−ν). Consequently, the bound of W₁ is

$$W_1\le W_{1,1}+W_{1,2}+W_{1,3}\lesssim {\left(n^{-1}\log p\right)}^{1/2}$$

with probability at least 1 − O(p^−ν). Note that we have the same result for W₂ since it has the same distribution as W₁. Finally, W₃ is bounded as

$$\begin{gathered} W_3\le {\Vert n^{-1}{\mathbf{C}}^{0^{\prime }}{\mathbf{C}}^0-\mathbb{E}\left[n^{-1}{\mathbf{C}}^{0^{\prime }}{\mathbf{C}}^0\right]\Vert }_{\max }+{\Vert n^{-1}{\mathbf{E}}^{\prime }{\mathbf{E}}_{{\eta }^0}\Vert }_{\max } \\ +{\Vert n^{-1}{\mathbf{E}}^{\prime }{\mathbf{C}}^0\Vert }_{\max }+{\Vert n^{-1}{\mathbf{E}}_{{\eta }^0}^{\prime }{\mathbf{C}}^0\Vert }_{\max }=:W_{1,1}+W_{3,1}+W_{1,3}+W_{3,2}. \end{gathered}$$

The upper bound of W_3,1 turns out to be O((n⁻¹ log p)^1/2) that holds with probability at least 1−O(p^−ν). We check this claim. Using the union bound and the inequality of Lemma 7(a) with p replaced by n and putting u = (C(ν + 2)n⁻¹ log p)^1/2 yield

$$\mathbb{P}\left({\Vert n^{-1}{\mathbf{E}}^{\prime }{\mathbf{E}}_{{\eta }^0}\Vert }_{\max }>u\right)\le p^2\underset{j,\ell }{\max }\mathbb{P}\left(\left|n^{-1}\sum _{i=1}^n\left(e_{ij}{e}_{{\eta }^0,i\ell }\right)\right|>u\right)\le 2p^{-\nu }.$$

Finally, W_3,2 is found to have the same bound as W_1,3 because ${\mathbf{E}}_{{\eta }^0}$ is an independent copy of E. Consequently, with probability at least 1 − O(p^−ν), we obtain

$${\Vert \mathbf{U}\left({\mathit{\theta }}^0\right)-\mathbb{E}\left[\mathbf{U}\left({\mathit{\theta }}^0\right)\right]\Vert }_{\max }\lesssim {\left(n^{-1}\log p\right)}^{1/2}.$$

This completes the proof of (a) since p^−ν/π_np = O(1).

Next we show (b) by verifying the following

$$(b-i)\underset{\left|\mathcal{A}\right|\le k,\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\mathbf{v}}_{\mathcal{A}}(\mathit{\theta })-{\mathbf{v}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)\Vert }_{\max }\lesssim s^{3/2}{\tilde{c}}_{np},$$

$$(b-ii){\Vert \mathbf{v}\left({\mathit{\theta }}^0\right)-\mathbb{E}\left[\mathbf{v}\left({\mathit{\theta }}^0\right)\right]\Vert }_{\max }\lesssim s{\left(n^{-1}\log p\right)}^{1/2},$$

Similar to the proof of (a), we need to modify the proof of Lemma 8(b) in Section D.3 for obtaining the uniform bound with respect to $\mathcal{A}$, but the obtained result is already uniform over the choice of $\mathcal{A}$. Thus the same upper bound holds and (b–i) follows. Next we show (b–ii). It holds that

$$\begin{gathered} {\Vert \mathbf{v}\left({\mathit{\theta }}^0\right)-\mathbb{E}\mathbf{v}\left({\mathit{\theta }}^0\right)\Vert }_{\max } \\ \le {\Vert n^{-1}{\mathbf{X}}^{\prime }\mathbf{y}-\mathbb{E}{n}^{-1}{\mathbf{X}}^{\prime }\mathbf{y}\Vert }_{\max }+{\Vert n^{-1}\tilde{\mathbf{X}}{\left({\mathit{\theta }}^0\right)}^{\prime }\mathbf{y}-\mathbb{E}{n}^{-1}\tilde{\mathbf{X}}{\left({\mathit{\theta }}^0\right)}^{\prime }\mathbf{y}\Vert }_{\max } \\ \le {\Vert \left(n^{-1}{\mathbf{X}}^{\prime }\mathbf{X}-\mathbb{E}{n}^{-1}{\mathbf{X}}^{\prime }\mathbf{X}\right)\beta \Vert }_{\max }+{\Vert n^{-1}{\mathbf{X}}^{\prime }\epsilon -\mathbb{E}{n}^{-1}{\mathbf{X}}^{\prime }\epsilon \Vert }_{\max } \\ +{\Vert \left(n^{-1}\tilde{\mathbf{X}}{\left({\mathit{\theta }}^0\right)}^{\prime }\mathbf{X}-\mathbb{E}\left[n^{-1}\tilde{\mathbf{X}}{\left({\mathit{\theta }}^0\right)}^{\prime }\mathbf{X}\right]\right)\beta \Vert }_{\max }+{\Vert n^{-1}\tilde{\mathbf{X}}{\left({\mathit{\theta }}^0\right)}^{\prime }\epsilon -\mathbb{E}\left[n^{-1}\tilde{\mathbf{X}}{\left({\mathit{\theta }}^0\right)}^{\prime }\epsilon \right]\Vert }_{\max } \\ =:Z_1+Z_2+Z_3+Z_4. \end{gathered}$$

These terms can be bounded by the results obtained in the proof of (a–ii). We see that

$$Z_1\le s^{1/2}{\Vert n^{-1}{\mathbf{X}}^{\prime }{\mathbf{X}}_{{\mathcal{S}}^0}-\mathbb{E}{n}^{-1}{\mathbf{X}}^{\prime }{\mathbf{X}}_{{\mathcal{S}}^0}\Vert }_{\max }{\Vert {\beta }_{{\mathcal{S}}^0}\Vert }_2\lesssim s{W}_1\lesssim s{\left(n^{-1}\log p\right)}^{1/2}$$

with probability at least 1 − O(p^−ν). Next we deduce

$$Z_2\le {\Vert n^{-1}{\mathbf{\Lambda }}^0{\mathbf{F}}^{0^{\prime }}\epsilon \Vert }_{\max }+{\Vert n^{-1}{\mathbf{E}}^{\prime }\epsilon \Vert }_{\max }.$$

The first and second terms can be bounded by the same ways as W_1,3 and W_3,1 in the proof of (a) above with E and ${\mathbf{E}}_{{\eta }^0}$ replaced by ε, respectively. Then the first term dominates the second and hence Z₂ ≲ (n⁻¹ log p)^1/2 with probability at least 1−O(p^−ν). Similarly, we can obtain

$$Z_3\le s^{1/2}{\Vert n^{-1}\tilde{\mathbf{X}}{\left({\mathit{\theta }}^0\right)}^{\prime }{\mathbf{X}}_{{\mathcal{S}}^0}-\mathbb{E}{n}^{-1}\tilde{\mathbf{X}}{\left({\mathit{\theta }}^0\right)}^{\prime }{\mathbf{X}}_{{\mathcal{S}}^0}\Vert }_{\max }{\Vert {\beta }_{{\mathcal{S}}^0}\Vert }_2\lesssim s{W}_3\lesssim s{\left(n^{-1}\log p\right)}^{1/2}$$

with probability at least 1−O(p^−ν). Note that Z₄ has the same bound as Z₂. Consequently, collecting terms leads to the result, Z₁ + ⋯ + Z₄ ≲ s(n⁻¹ log p)^1/2 with probability at least 1 − O(p^−ν). This proves (b–ii) and concludes the proof of Lemma 4.

C.6. Lemma 5 and its proof

Lemma 5

Assume that all the conditions of Theorem 2 hold. Then with probability at least 1 − O(π_np), the Lasso solution in (19) satisfies

$$\begin{gathered} \underset{\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\widehat{\beta }}^{aug}(\mathit{\theta })-{\beta }^{aug}\Vert }_2=O\left(s^{1/2}\lambda \right), \\ \underset{\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\widehat{\beta }}^{aug}(\mathit{\theta })-{\beta }^{aug}\Vert }_1=O(s\lambda ), \end{gathered}$$

where λ = c₁n^1/2 log p with c₁ some positive constant.

Proof. Let $\delta (:=\delta (\mathit{\theta })):={\widehat{\beta }}^{\mathrm{aug}}(\mathit{\theta })-{\beta }^{\text{aug}}$. We start with introducing two inequalities

$$\underset{\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert n^{-1}{[\mathbf{X},\tilde{\mathbf{X}}(\mathit{\theta })]}^{\prime }\epsilon \Vert }_{\max }\le 2^{-1}\lambda ,$$ (A.15)

$$\underset{\mathit{\theta }\in {\mathbf{\Theta }}_{np},\delta \in \mathbb{V}}{\inf }{\delta }^{\prime }\mathbf{U}(\mathit{\theta })\delta /\Vert \delta {\Vert }_2^2\ge {\sigma }_e^2(1+o(1)),$$ (A.16)

where λ = c₁n^−1/2 log p for some positive constant c₁ and

$$\mathbb{V}=\left\{\delta \in {ℝ}^{2p}:{\Vert {\delta }_{{\mathcal{S}}^1}\Vert }_1\le 3{\Vert {\delta }_{{\mathcal{S}}^0}\Vert }_1,\Vert \delta {\Vert }_0\le k\right\}.$$ (A.17)

It is well known that the rate of convergence of the Lasso estimator can be obtained provided that (A.15) and (A.16) hold. Thus we show that these two inequalities actually hold with high probability in Step 1, and then derive the convergence rate using (A.15) and (A.16) in Step 2.

Step 1

We check whether (A.15) and (A.16) actually hold with high probability. We first verify (A.15). By the proofs of Lemmas 8 and 4, we have

$$\begin{gathered} \underset{\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert n^{-1}{[\mathbf{X},\tilde{\mathbf{X}}(\mathit{\theta })]}^{\prime }\epsilon \Vert }_{\max } \\ \le {\Vert n^{-1}{\mathbf{X}}^{\prime }\epsilon \Vert }_{\max }+\underset{\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert n^{-1}\tilde{\mathbf{X}}{(\mathit{\theta })}^{\prime }\epsilon -n^{-1}\tilde{\mathbf{X}}{\left({\mathit{\theta }}^0\right)}^{\prime }\epsilon \Vert }_{\max }+{\Vert n^{-1}\tilde{\mathbf{X}}{\left({\mathit{\theta }}^0\right)}^{\prime }\epsilon \Vert }_{\max }. \end{gathered}$$

The first and third terms can both be upper bounded by O(n^−1/2 log p) with probability at least 1−O(p^−ν), following the same lines for deriving bound for Z₂ in the proof of Lemma 4. To evaluate the second term, we can use the argument about V₂ and its upper bound (A.24) in the proof of Lemma 8. That bound still holds with the same rate O(n^−1/2 log p) even if we take $\mathcal{A}=\{1,\cdots ,p\}$. Thus we conclude that (A.15) is true for the given λ by choosing an appropriate positive large constant c₁, with probability at least 1 − O(π_np).

Next to verify (A.16), we derive the population lower bound first and then show that the difference is negligible. From the construction, we have

$$\begin{gathered} \mathbb{E}\left[n^{-1}\tilde{\mathbf{X}}{\left({\mathit{\theta }}^0\right)}^{\prime }\tilde{\mathbf{X}}\left({\mathit{\theta }}^0\right)\right]=\mathbb{E}\left[n^{-1}{\mathbf{X}}^{\prime }\mathbf{X}\right]={\mathbf{\Lambda }}^0{\mathbf{\Sigma }}_f{\mathbf{\Lambda }}^{0^{\prime }}+{\sigma }_e^2{\mathbf{I}}_p, \\ \mathbb{E}\left[n^{-1}\tilde{\mathbf{X}}{\left({\mathit{\theta }}^0\right)}^{\prime }\mathbf{X}\right]=\mathbb{E}\left[n^{-1}{\mathbf{X}}^{\prime }\tilde{\mathbf{X}}\left({\mathit{\theta }}^0\right)\right]={\mathbf{\Lambda }}^0{\mathbf{\Sigma }}_f{\mathbf{\Lambda }}^{0^{\prime }}. \end{gathered}$$

Using these equations, we obtain the lower bound

$$\begin{gathered} \underset{\delta \in \mathbb{V}}{\inf }{\delta }^{\prime }\mathbb{E}\left[\mathbf{U}\left({\mathit{\theta }}^0\right)\right]\delta /\Vert \delta {\Vert }_2^2=\underset{\delta \in \mathbb{V}}{\inf }{\delta }^{\prime }\left(\begin{array}{cc}{\mathbf{\Lambda }}^0{\mathbf{\Sigma }}_f{\mathbf{\Lambda }}^{0^{\prime }}+{\sigma }_e^2{\mathbf{I}}_p& {\mathbf{\Lambda }}^0{\mathbf{\Sigma }}_f{\mathbf{\Lambda }}^{0^{\prime }}\\ {\mathbf{\Lambda }}^0{\mathbf{\Sigma }}_f{\mathbf{\Lambda }}^{0^{\prime }}& {\mathbf{\Lambda }}^0{\mathbf{\Sigma }}_f{\mathbf{\Lambda }}^{0^{\prime }}+{\sigma }_e^2{\mathbf{I}}_p\end{array}\right)\delta /\Vert \delta {\Vert }_2^2 \\ =\underset{\delta \in \mathbb{V}}{\inf }{\delta }^{\prime }\left\{\left(\begin{array}{cc}1& 1\\ 1& 1\end{array}\right)\otimes {\mathbf{\Lambda }}^0{\mathbf{\Sigma }}_f{\mathbf{\Lambda }}^{0^{\prime }}+{\sigma }_e^2{\mathbf{I}}_{2p}\right\}\delta /\Vert \delta {\Vert }_2^2 \\ \ge {\sigma }_e^2. \end{gathered}$$ (A.18)

Because $\delta \in \mathbb{V}$ is sparse and satisfies $|\mathcal{B}|\le k$ for $\mathcal{B}:=\mathrm{supp}(\delta )$, it holds that ${\delta }^{\prime }\mathbf{U}\left({\mathit{\theta }}^0\right)\delta ={\delta }_{\mathcal{B}}^{\prime }{\mathbf{U}}_{\mathcal{B}}\left({\mathit{\theta }}^0\right){\delta }_{\mathcal{B}}$ and ${\delta }^{\prime }\mathbb{E}\left[\mathbf{U}\left({\mathit{\theta }}^0\right)\right]\delta ={\delta }_{\mathcal{B}}^{\prime }\mathbb{E}\left[{\mathbf{U}}_{\mathcal{B}}\left({\mathit{\theta }}^0\right)\right]{\delta }_{\mathcal{B}}$. Hence from Lemma 4 together with the condition on dimensionality, we obtain

$$\begin{gathered} \underset{\left|\mathcal{B}\right|\le k,\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\mathbf{U}}_{\mathcal{B}}(\mathit{\theta })-\mathbb{E}\left[{\mathbf{U}}_{\mathcal{B}}\left({\mathit{\theta }}^0\right)\right]\Vert }_{\max }=O\left(k^{1/2}{\tilde{c}}_{np}\right) \\ =o\left(s^{-1}\right) \end{gathered}$$ (A.19)

with probability at least 1 − O(π_np). Thus using (A.19), we have for any $\delta \in \mathbb{V}$,

$$\begin{gathered} {\delta }^{\prime }\mathbb{E}\left[\mathbf{U}\left({\mathit{\theta }}^0\right)\right]\delta -{\delta }^{\prime }\mathbf{U}(\mathit{\theta })\delta ={\delta }_{\mathcal{B}}^{\prime }\left\{\mathbb{E}\left[{\mathbf{U}}_{\mathcal{B}}\left({\mathit{\theta }}^0\right)\right]-{\mathbf{U}}_{\mathcal{B}}(\mathit{\theta })\right\}{\delta }_{\mathcal{B}} \\ \le \Vert \delta {\Vert }_1^2\underset{\left|\mathcal{B}\right|\le k,\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\mathbf{U}}_{\mathcal{B}}(\mathit{\theta })-\mathbb{E}\left[{\mathbf{U}}_{\mathcal{B}}\left({\mathit{\theta }}^0\right)\right]\Vert }_{\max }={\left({\Vert {\delta }_{{\mathcal{S}}^0}\Vert }_1+{\Vert {\delta }_{{\mathcal{S}}^1}\Vert }_1\right)}^{2}o\left(s^{-1}\right) \\ \lesssim {\Vert {\delta }_{{\mathcal{S}}^0}\Vert }_1^{2}o\left(s^{-1}\right)\le {\Vert {\delta }_{{\mathcal{S}}^0}\Vert }_2^{2}O(1)\le \Vert \delta {\Vert }_2^{2}o(1). \end{gathered}$$

Rearranging the terms with (A.18) yields

$$\underset{\mathit{\theta }\in {\mathbf{\Theta }}_{np},\delta \in \mathbb{V}}{\inf }{\delta }^{\prime }\mathbf{U}(\mathit{\theta })\delta /\Vert \delta {\Vert }_2^2\ge \underset{\mathit{\delta }\in \mathbb{V}}{\inf }{\delta }^{\prime }\mathbb{E}\left[\mathbf{U}\left({\mathit{\theta }}^0\right)\right]\delta /\Vert \delta {\Vert }_2^2-|o(1)|\ge {\sigma }_e^2-|o(1)|,$$

resulting in (A.16). In consequence, two inequalities (A.15) and (A.16) hold with probability at least 1 − O(π_np).

Step 2

This part is well known in the literature (e.g., [33]) so we briefly give the proof omitting the details. Because the objective function is given by

$${\widehat{\beta }}^{\text{aug}}(\mathit{\theta })=\arg \underset{\mathbf{b}\in {ℝ}^{2p}}{\min }{n}^{-1}\Vert \mathbf{y}-[\mathbf{X},\tilde{\mathbf{X}}(\mathit{\theta })]\mathbf{b}{\Vert }_2^2+\lambda \Vert \mathbf{b}{\Vert }_1,$$

the global optimality of the Lasso estimator implies

$$\begin{gathered} {(2n)}^{-1}{\Vert \mathbf{y}-[\mathbf{X},\tilde{\mathbf{X}}(\mathit{\theta })]{\widehat{\beta }}^{\text{aug}}(\mathit{\theta })\Vert }_2^2+\lambda {\Vert {\widehat{\beta }}^{\text{aug}}(\mathit{\theta })\Vert }_1 \\ \le {(2n)}^{-1}{\Vert \mathbf{y}-[\mathbf{X},\tilde{\mathbf{X}}(\mathit{\theta })]{\beta }^{\text{aug}}\Vert }_2^2+\lambda {\Vert {\beta }^{\text{aug}}\Vert }_1, \end{gathered}$$

where the true parameter vector β^aug was defined in the proof of Theorem 2. Note that ${\sup }_{\mathit{\theta }\in {\mathbf{\Theta }}_{np}}\Vert \delta (\mathit{\theta }){\Vert }_0\le k$ by the assumption. Expanding the inequality and collecting terms with (A.15) yield

$$2^{-1}{\delta }^{\prime }\mathbf{U}(\mathit{\theta })\delta \le {\Vert n^{-1}{\epsilon }^{\prime }[\mathbf{X},\tilde{\mathbf{X}}(\mathit{\theta })]\Vert }_{\max }\Vert \delta {\Vert }_1+\lambda \Vert \delta {\Vert }_1\le (3/2)\lambda \Vert \delta {\Vert }_1.$$ (A.20)

On the other hand, applying Lemma 1 of [33] to our model reveals that $\delta \in \mathbb{V}$. Thus we can use (A.16), (A.20), and (A.17) to get

$$\Vert \delta {\Vert }_2^2\left({\sigma }_e^2+o(1)\right)\le 3\lambda \Vert \delta {\Vert }_1=3\lambda \left({\Vert {\delta }_{{\mathcal{S}}^1}\Vert }_1+{\Vert {\delta }_{{\mathcal{S}}^0}\Vert }_1\right)\le 12\lambda {\Vert {\delta }_{{\mathcal{S}}^0}\Vert }_1.$$

Since $\left|{\mathcal{S}}^0\right|=s$ and ${\Vert {\delta }_{{\mathcal{S}}^0}\Vert }_1\le s^{1/2}{\Vert {\delta }_{{\mathcal{S}}^0}\Vert }_2$, it holds that $\Vert \delta {\Vert }_2\le 12s^{1/2}\lambda /\left({\sigma }_e^2+o(1)\right)$. Since ${\Vert {\delta }_{{\mathcal{S}}^0}\Vert }_2\le \Vert \delta {\Vert }_2$, we obtain the desired bound $\Vert \delta {\Vert }_1\le 48s\lambda /\left({\sigma }_e^2+o(1)\right)$. This bound holds uniformly over θ ∈ Θ_np, which completes the proof of Lemma 5.

D. Additional technical lemmas and their proofs

D.1. Lemma 6 and its proof

Lemma 6

Denote by V ∈ ℝ^r×r a diagonal matrix with its entries the r largest eigenvalues of (np)⁻¹XX′ and define $\mathbf{H}=\left({\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{\Lambda }}^0/p\right)\left({\mathbf{F}}^{0^{\prime }}\widehat{\mathbf{F}}/n\right){\mathbf{V}}^{-1}$. Assume that Conditions 2–5 hold. Then ‖H‖₂ + ‖H⁻¹‖₂ + ‖V‖₂ + ‖V⁻¹‖₂ is bounded from above by some constant with probability at least 1 − O(p^−ν).

Proof. Let λ^k[A] denote the kth largest eigenvalue of square matrix A throughout the proof. Because ‖Λ⁰⁰Λ⁰/p‖₂ ≤ M and

$$\begin{gathered} {\Vert {\mathbf{F}}^{0^{\prime }}\widehat{\mathbf{F}}/n\Vert }_2\le {\Vert n^{-1/2}{\mathbf{F}}^0\Vert }_2{\Vert n^{-1/2}\widehat{\mathbf{F}}\Vert }_2 \\ \le {(rn)}^{1/2}{\Vert n^{-1/2}{\mathbf{F}}^0\Vert }_{\max }{\left({\lambda }^1\left[n^{-1}{\widehat{\mathbf{F}}}^{\prime }\widehat{\mathbf{F}}\right]\right)}^{1/2}\le r^{1/2}M \end{gathered}$$

by Conditions 2–3, and ${\widehat{\mathbf{F}}}^{\prime }\widehat{\mathbf{F}}/n={\mathbf{I}}_r$, we have

$$\Vert \mathbf{H}{\Vert }_2\le {\Vert {\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{\Lambda }}^0/p\Vert }_2{\Vert {\mathbf{F}}^{0^{\prime }}\widehat{\mathbf{F}}/n\Vert }_2{\Vert {\mathbf{V}}^{-1}\Vert }_2\lesssim {\Vert {\mathbf{V}}^{-1}\Vert }_2,$$

where ‖V⁻¹‖₂ is equal to the reciprocal of the rth largest eigenvalue of (np)⁻¹XX′. Similarly, under Conditions 2–3 we also have

$${\Vert {\mathbf{H}}^{-1}\Vert }_2\le \Vert \mathbf{V}{\Vert }_2{\Vert {\left({\mathbf{F}}^{0\prime }\widehat{\mathbf{F}}/n\right)}^{-1}\Vert }_2{\Vert {\left({\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{\Lambda }}^0/p\right)}^{-1}\Vert }_2\lesssim \Vert \mathbf{V}{\Vert }_2{\Vert {\left({\mathbf{F}}^{0^{\prime }}\widehat{\mathbf{F}}/n\right)}^{-1}\Vert }_2,$$

where ‖V‖₂ is equal to the largest eigenvalue of (np)⁻¹XX′ and the inverse matrix in the upper bound is well defined by [2]. To see if ${\Vert {\left({\mathbf{F}}^{0\prime }\widehat{\mathbf{F}}/n\right)}^{-1}\Vert }_2$ is bounded from above, it suffices to bound the minimum eigenvalue of ${\mathbf{F}}^{0^{\prime }}\widehat{\mathbf{F}}{\widehat{\mathbf{F}}}^{\prime }{\mathbf{F}}^0/n^2$ away from zero uniformly in n. Regarding r eigenvalues of the matrix, Sylvester’s law of inertia (e.g., [31], Theorem 4.5.8) entails that all the r eigenvalues are positive for all n. Moreover, by Proposition 1 of [2] we know that the limiting matrix of ${\widehat{\mathbf{F}}}^{\prime }{\mathbf{F}}^0/n$ is nonsingular under Conditions 2 and 5. Therefore, we can conclude that $\lim {\inf }_{n\to \infty }{\lambda }^r\left[{\mathbf{F}}^{0^{\prime }}\widehat{\mathbf{F}}{\widehat{\mathbf{F}}}^{\prime }{\mathbf{F}}^0/n^2\right]>0$ a.s., and hence ‖H⁻¹‖₂ ≲ ‖V‖₂ follows.

To complete the proof, it is sufficient to show that the maximum and rth largest eigenvalues of (np)⁻¹XX′ are bounded from above and away from zero, respectively, for all large n and p. By the definition of the spectral norm and triangle inequality, we have

$$\begin{gathered} {\{{\lambda }^1\left[{(np)}^{-1}\mathbf{X}{\mathbf{X}}^{\prime }\right)]\}}^{1/2}={\Vert {(np)}^{-1/2}\mathbf{X}\Vert }_2\le {\Vert {(np)}^{-1/2}{\mathbf{F}}^0{\mathbf{\Lambda }}^{0^{\prime }}\Vert }_2+{\Vert {(np)}^{-1/2}\mathbf{E}\Vert }_2 \\ \le {\Vert n^{-1/2}{\mathbf{F}}^0\Vert }_2{\Vert p^{-1/2}{\mathbf{\Lambda }}^0\Vert }_2+{\Vert {(np)}^{-1/2}\mathbf{E}\Vert }_2. \end{gathered}$$

By Conditions 2 and 3, the first term is a.s. bounded by a constant as discussed above. The second term is O((n ∧ p)^−1/2) = o(1) with probability at least 1 − 2exp(−|O(n ∨ p)|) by Lemma 9(a) under Condition 4. Therefore, the largest eigenvalue of (np)⁻¹XX′ is bounded from above by some constant with probability at least 1 − 2exp(−|O(n ∨ p)|).

Next we bound the rth largest eigenvalue of (np)⁻¹XX′ away from zero. Since the matrix is symmetric, Weyl’s inequality (e.g., [31], Theorem 4.3.1) yields

$$\begin{gathered} {\lambda }^r\left[{(np)}^{-1}\mathbf{X}{\mathbf{X}}^{\prime }\right]={\lambda }^r\left[{(np)}^{-1}\left\{{\mathbf{F}}^0{\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{\Lambda }}^0{\mathbf{F}}^{0^{\prime }}+\left(\mathbf{E}{\mathbf{\Lambda }}^0{\mathbf{F}}^{0^{\prime }}+{\mathbf{F}}^0{\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{E}}^{\prime }\right)+\mathbf{E}{\mathbf{E}}^{\prime }\right\}\right] \\ \ge {\lambda }^r\left[{(np)}^{-1}{\mathbf{F}}^0{\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{\Lambda }}^0{\mathbf{F}}^{0^{\prime }}\right]+{\lambda }^n\left[{(np)}^{-1}\left(\mathbf{E}{\mathbf{\Lambda }}^0{\mathbf{F}}^{0^{\prime }}+{\mathbf{F}}^0{\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{E}}^{\prime }\right)\right]+{\lambda }^n\left[{(np)}^{-1}\mathbf{E}{\mathbf{E}}^{\prime }\right]. \end{gathered}$$ (A.21)

The third term of lower bound (A.21) is obviously nonnegative. For the first term of lower bound (A.21), let $\mathcal{V}$ denote a subspace of ℝⁿ. Because ${\mathbf{F}}^0{\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{\Lambda }}^0{\mathbf{F}}^{0^{\prime }}$ is symmetric, the Courant–Fischer min-max Theorem (e.g., [31], Theorem 4.2.6) yields

$$\begin{gathered} {\lambda }^r\left[{(np)}^{-1}{\mathbf{F}}^0{\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{\Lambda }}^0{\mathbf{F}}^{0^{\prime }}\right]=\underset{\mathcal{V}:\dim (\mathcal{V})=r}{\max }\underset{\mathbf{v}\in \mathcal{V}\backslash \left\{0\right\}}{\min }\left\{{(np)}^{-1}\frac{{\mathbf{v}}^{\prime }{\mathbf{F}}^0{\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{\Lambda }}^0{\mathbf{F}}^{0^{\prime }}\mathbf{v}}{{\mathbf{v}}^{\prime }\mathbf{v}}\right\} \\ \ge \underset{\mathcal{V}:\dim (\mathcal{V})=r}{\max }\underset{\mathbf{v}\in \mathcal{V}\backslash \left\{0\right\}}{\min }\left(n^{-1}\frac{{\mathbf{v}}^{\prime }{\mathbf{F}}^0{\mathbf{F}}^{0^{\prime }}\mathbf{v}}{{\mathbf{v}}^{\prime }\mathbf{v}}\right)\underset{{\mathbf{F}}^{0\prime }\mathbf{v}\in {ℝ}^r\backslash \left\{0\right\}}{\min }\left(p^{-1}\frac{{\mathbf{v}}^{\prime }{\mathbf{F}}^0{\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{\Lambda }}^0{\mathbf{F}}^{0^{\prime }}\mathbf{v}}{{\mathbf{v}}^{\prime }{\mathbf{F}}^0{\mathbf{F}}^{0\prime }\mathbf{v}}\right) \\ ={\lambda }^r\left[n^{-1}{\mathbf{F}}^0{\mathbf{F}}^{0^{\prime }}\right]{\lambda }^r\left[p^{-1}{\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{\Lambda }}^0\right]={\lambda }^r\left[n^{-1}{\mathbf{F}}^{0^{\prime }}{\mathbf{F}}^0\right]{\lambda }^r\left[p^{-1}{\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{\Lambda }}^0\right] \\ \ge {\lambda }^r\left[{\mathbf{\Sigma }}_f\right]{\lambda }^r\left[p^{-1}{\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{\Lambda }}^0\right]-{\Vert n^{-1}{\mathbf{F}}^{0^{\prime }}{\mathbf{F}}^0-{\mathbf{\Sigma }}_f\Vert }_2 \\ \ge {\lambda }^r\left[{\mathbf{\Sigma }}_f\right]{\lambda }^r\left[p^{-1}{\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{\Lambda }}^0\right]-r{\Vert n^{-1}{\mathbf{F}}^{0^{\prime }}{\mathbf{F}}^0-{\mathbf{\Sigma }}_f\Vert }_{\max }. \end{gathered}$$

In this lower bound, the first term is bounded away from zero by Conditions 2–3. Meanwhile, to evaluate the second term we use Lemma 7(d) in Section D.2, which together with the union bound establishes

$$\begin{gathered} \mathbb{P}\left({\Vert n^{-1}{\mathbf{F}}^{0^{\prime }}{\mathbf{F}}^0-{\mathbf{\Sigma }}_f\Vert }_{\max }>u\right)\le r^2\underset{k,\ell \in \{1,\cdots ,r\}}{\max }\mathbb{P}\left(\left|n^{-1}\sum _{i=1}^n\left(f_{ik}^0{f}_{i\ell }^0-\mathbb{E}{f}_{ik}^0{f}_{i\ell }^0\right)\right|>u\right) \\ \le 2r^2\exp \left(-n{u}^2/C\right) \end{gathered}$$

for any 0 ≤ u ≤ c. Thus the second one turns out to be O((n⁻¹ log p)^1/2) = o(1) with probability at least 1 − O(p^−ν) once we set u = (Cνn⁻¹ log p)^1/2 and assume n⁻¹ log p ≤ c²/(Cν) without loss of generality. Therefore, the first term of lower bound (A.21) is bounded away from zero eventually. For the second term of (A.21), since the spectral norm gives the upper bound of the spectral radius we have

$$\begin{gathered} \left|{\lambda }^n\left[{(np)}^{-1}\left(\mathbf{E}{\mathbf{\Lambda }}^0{\mathbf{F}}^{0^{\prime }}+{\mathbf{F}}^0{\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{E}}^{\prime }\right)\right]\right|\le {\Vert {(np)}^{-1}\left(\mathbf{E}{\mathbf{\Lambda }}^0{\mathbf{F}}^{0^{\prime }}+{\mathbf{F}}^0{\mathbf{\Lambda }}^{0^{\prime }}{\mathbf{E}}^{\prime }\right)\Vert }_2 \\ \le 2{\Vert {(np)}^{-1/2}\mathbf{E}\Vert }_2{\Vert p^{-1/2}{\mathbf{\Lambda }}^0\Vert }_2{\Vert n^{-1/2}{\mathbf{F}}^0\Vert }_2 \\ =O\left({(n\wedge p)}^{-1/2}\right)O(1)O(1)=o(1), \end{gathered}$$

which holds with probability at least 1 − 2exp(−|O(n ∨ p)|) by Lemma 9(a) in Section D.4. As a consequence, the desired result holds with probability at least 1 − O(p^−ν) and this concludes the proof of Lemma 6.

D.2. Lemma 7 and its proof

Lemma 7

Assume that Conditions 2–4 hold. Then there exist some positive constants c and C such that the following inequalities hold

1. For all ℓ,i ∈ {1,⋯, n} and 0 ≤ u ≤ c, we have

   $$\mathbb{P}\left(\left|p^{-1}\sum _{j=1}^p\left(e_{\ell j}{e}_{ij}-\mathbb{E}\left[e_{\ell j}{e}_{ij}\right]\right)\right|>u\right)\le 2\exp \left(-p{u}^2/C\right).$$
2. For all k ∈ {1,⋯, r}, j ∈ {1,⋯, p}, and 0 ≤ u ≤ c, we have

   $$\mathbb{P}\left(\left|n^{-1}\sum _{i=1}^n{f}_{ik}^0{e}_{ij}\right|>u\right)\le 2\exp \left(-n{u}^2/C\right).$$
3. For all k ∈ {1,⋯, r}, i ∈ {1,⋯, n}, and u ≥ 0, we have

   $$\mathbb{P}\left(\left|p^{-1}\sum _{j=1}^p{\lambda }_{jk}^0{e}_{ij}\right|>u\right)\le 2\exp \left(-p{u}^2/C\right).$$
4. For all k, ℓ ∈ {1,⋯, r} and 0 ≤ u ≤ c, we have

   $$\mathbb{P}\left(\left|n^{-1}\sum _{i=1}^n\left(f_{ik}^0{f}_{i\ell }^0-\mathbb{E}\left[f_{ik}^0{f}_{i\ell }^0\right]\right)\right|>u\right)\le 2\exp \left(-n{u}^2/C\right).$$

Proof. (a) To obtain the first result, we rely on the Hanson–Wright inequality. Let ξ = (ξ₁,⋯, ξ_m)⁰ ∈ ℝ^m denote a random vector whose components are independent copies of $e\sim \mathrm{subG}\left(C_e^2\right)$. Then the inequality states that for any (nonrandom) matrix A ∈ ℝ^m×m,

$$\mathbb{P}\left(\left|{\xi }^{\prime }\mathbf{A}\xi -\mathbb{E}{\xi }^{\prime }\mathbf{A}\xi \right|>u\right)\le 2\exp \left\{-{\tilde{C}}_H\min \left(\frac{u^2}{K^4\Vert \mathbf{A}{\Vert }_F^2},\frac{u}{K^2\Vert \mathbf{A}{\Vert }_2}\right)\right\},$$ (A.22)

where K is a positive constant such that ${\sup }_{k\ge 1}{k}^{-1/2}{\left(\mathbb{E}|e{|}^k\right)}^{1/k}\le K$ and ${\tilde{C}}_H$ is a positive constant. In our setting, we can take $K=3C_e^2$ (e.g., Lemma 1.4 of [34]). Using this inequality, we first prove the case when ℓ = i. If we set m = p and A = diag(p⁻¹, …, p⁻¹), then we have

$$\left|{\xi }^{\prime }\mathbf{A}\xi -\mathbb{E}{\xi }^{\prime }\mathbf{A}\xi \right|=\left|p^{-1}\sum _{j=1}^p\left({\xi }_j^2-\mathbb{E}{\xi }_j^2\right)\right|\stackrel{d}{=}\left|p^{-1}\sum _{j=1}^p\left(e_{ij}^2-\mathbb{E}\left[e_{ij}^2\right]\right)\right|$$

for all i. Moreover, we obtain $\Vert \mathbf{A}{\Vert }_F^2=p^{-1}$ and ‖A‖₂ = p⁻¹ in this case. The assumed condition $0<u\le 9C_e^2=K^2$ entails that u²/K⁴ ≤ u/K² so the result follows from (A.22) with ${\tilde{C}}_H$ replaced by $C_H=81C_e^4/{\tilde{C}}_H$.

Similarly, we prove the case when ℓ ≠ i. We set m = p + 1 and A = (a₁,⋯, a_p+1), where a₁ = (0,p⁻¹,⋯, p⁻¹)′ and a_j = 0 for j = 2,⋯, p + 1. That is, the entries of A are all zero except that the second to (p + 1)th components in the first column vector are p⁻¹. Under this setting, we observe that

$$\left|{\xi }^{\prime }\mathbf{A}\xi -\mathbb{E}{\xi }^{\prime }\mathbf{A}\xi \right|=\left|p^{-1}\sum _{j=2}^{p+1}{\xi }_1{\xi }_j\right|\stackrel{d}{=}\left|p^{-1}\sum _{j=1}^p{e}_{\ell j}{e}_{ij}\right|$$

for all ℓ ≠ i. Moreover, we obtain $\Vert \mathbf{A}{\Vert }_F^2=\Vert \mathbf{A}{\Vert }_2=p^{-1}$ in this case. Therefore, the same bound holds as in the case of ℓ = i from (A.22) again. Consequently, for any $0\le u\le 9C_e^2$ we have

$$\mathbb{P}\left(\left|p^{-1}\sum _{j=1}^p\left(e_{\ell j}{e}_{ij}-\mathbb{E}\left[e_{\ell j}{e}_{ij}\right]\right)\right|>u\right)\le 2\exp \left(-p{u}^2/C_H\right)$$

(b) We prove the second assertion by Bernstein’s inequality for the sum of a martingale difference sequence (e.g., Theorem 3.14 in [11]). Fix k = 1 and j = 1. Define ${\mathcal{F}}_{i-1}$ as the σ-field generated from $\left\{f_{\ell 1}^0:\ell =i,i-1,\cdots \right\}$. Then $\left(f_{i1}^0{e}_{i1},{\mathcal{F}}_i\right)$ forms a martingale difference sequence because $\mathbb{E}\left|f_{i1}^0{e}_{i1}\right|<\infty$ and $\mathbb{E}\left[f_{i1}^0{e}_{i1}|{\mathcal{F}}_{i-1}\right]=0$ under Conditions 2 and 4. Since the sub-Gaussianity of e_i1 implies $\mathbb{E}{e}_{i1}^2\le 4C_e^2$ (e.g., Lemma 1.4 of [34]), we have $V_i:=\mathbb{E}\left[f_{ik}^{02}{e}_{ij}^2|{\mathcal{F}}_{i-1}\right]\le 4C_e^2{M}^2$, and hence ${\sum }_{i=1}^n{V}_i\le 4n{C}_e^2{M}^2$ a.s. due to boundedness $\left|f_{i1}^0\right|\le M$ a.s. On the other hand, by the sub-Gaussianity of e_ij and boundedness of $\left|f_{i1}^0\right|$ again we observe that for all p ≥ 3 and i ∈ {1,⋯, n},

$$\mathbb{E}\left[{\left(0\vee f_{i1}^0{e}_{i1}\right)}^p|{\mathcal{F}}_{i-1}\right]\le M^p{\left(2C_e^2\right)}^{p/2}p\mathbf{\Gamma }(p/2)\le p!{\left(2C_{e}M\right)}^{p-2}{V}_i/2,$$

where Γ denotes the Gamma function and we have used the estimates pΓ(p/2) ≤ p! and 2^p/2−2 ≤ 2^p−2/2 for p ≥ 3 in the last inequality. Then an application of Theorem 3.14 in [11] by putting x = u, $y=4M^2{C}_e^2$, and c = 2MC_e in their notation gives the one-sided result. Making twice the bound yields

$$\mathbb{P}\left(\left|n^{-1}\sum _{i=1}^n{f}_{ik}^0{e}_{ij}\right|>u\right)\le 2\exp \left(-\frac{n{u}^2}{8M^2{C}_e^2+4M{C}_{e}u}\right).$$

For all $0\le u\le M{C}_e^2$, the upper bound is further bounded by $2\exp \left(n{u}^2/\left(12M^2{C}_e^2\right)\right)$. We set $C_I=12M^2{C}_e^2$. Consequently, for any $0\le u\le M{C}_e^2$ we have

$$\mathbb{P}\left(\left|n^{-1}\sum _{i=1}^n{f}_{ik}^0{e}_{ij}\right|>u\right)\le 2\exp \left(-n{u}^2/C_I\right).$$

(c) We prove the third inequality. Note that

$$\mathbb{P}\left(\left|{\lambda }_{jk}^0{e}_{ij}\right|>u\right)\le 2\exp \left\{-\frac{u^2}{2{\lambda }_{jk}^{02}{C}_e^2}\right\}\le 2\exp \left\{-\frac{u^2}{2M^2{C}_e^2}\right\}.$$

This implies that ${\lambda }_{jk}^0{e}_{ij}$ is a sequence of i.i.d. $\mathrm{subG}\left(M^2{C}_e^2\right)$. Thus the result is obtained directly by Bernstein’s inequality for the sum of independent sub-Gaussian random variables. Consequently, for any u ≥ 0 putting $C_J=M^2{C}_e^2$ leads to

$$\mathbb{P}\left(\left|p^{-1}\sum _{j=1}^p{\lambda }_{jk}^0{e}_{ij}\right|>u\right)\le 2\exp \left(-p{u}^2/C_J\right).$$

(d) We show the last inequality. Note that for each k, (f_ik)_i ∼ i.i.d. subG(M²) since $\left|f_{ik}^0\right|\le M$ a.s. by Lemma 1.8 of [34] under Condition 2. Thus the remaining is the same as (a). Set $C_K=81M^4/{\tilde{C}}_H$ here. Then for any 0 ≤ u ≤ 9M², we have

$$\mathbb{P}\left(\left|n^{-1}\sum _{i=1}^n\left(f_{ik}^0{f}_{i\ell }^0-\mathbb{E}\left[f_{ik}^0{f}_{i\ell }^0\right]\right)\right|>u\right)\le 2\exp \left(-n{u}^2/C_K\right).$$

Finally the obtained inequalities hold even if the constant in the upper bound is replaced with arbitrary fixed constant C such that C ≥ max{C_H,C_I,C_J,C_K}. Similarly, we can also restrict the range of u for each inequality to be 0 ≤ u ≤ c for arbitrary fixed constant c that satisfies $0<c\le \min \left(9C_e^2,M{C}_e^2,9M^2\right)$. This completes the proof of Lemma 7.

D.3. Lemma 8 and its proof

Lemma 8

Assume that Conditions 1–4 hold. Then for any set $\mathcal{A}$ satisfying $|\mathcal{A}|\le k$, the following statements hold with probability at least 1 − O(π_np)

$$\underset{\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\mathbf{U}}_{\mathcal{A}}(\mathit{\theta })-{\mathbf{U}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)\Vert }_{\max }=O\left(k^{1/2}{\tilde{c}}_{np}\right),$$

$$\underset{\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\mathbf{v}}_{\mathcal{A}}(\mathit{\theta })-{\mathbf{v}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)\Vert }_{\max }=O\left(s^{3/2}{\tilde{c}}_{np}\right),$$

where Θ_np was defined in Lemma 3 and ${\tilde{c}}_{np}=n^{-1/2}\log p+p^{-1/2}\log n$. Consequently, we have

$$\underset{\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\mathbf{T}}_{\mathcal{A}}(\mathit{\theta })-{\mathbf{T}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)\Vert }_{\max }=O\left(\left(k^{1/2}+s^{3/2}\right){\tilde{c}}_{np}\right).$$

Proof. We first state some results that are useful in the proof. Since ‖n^−1/2F⁰‖₂ = O(1) a.s. by Condition 2 and ${\Vert k^{-1/2}{\mathbf{\Lambda }}_{\mathcal{A}}^0\Vert }_2=O(1)$ for any $\mathcal{A}$ such that $|\mathcal{A}|\le k$ under Condition 3, we first have

$${\Vert n^{-1/2}{\mathbf{C}}_{\mathcal{A}}^0\Vert }_2\le {\Vert n^{-1/2}{\mathbf{F}}^0\Vert }_2{k}^{1/2}{\Vert k^{-1/2}{\mathbf{\Lambda }}_{\mathcal{A}}^0\Vert }_2\lesssim k^{1/2}.$$

Next Lemma 9(b) in Section D.4 gives directly

$${\Vert n^{-1/2}{\mathbf{E}}_{{\eta }^0\mathcal{A}}\Vert }_2\lesssim 1$$ (A.23)

with probability at least 1 − O(p^−ν). By Condition 4, we also deduce

$$\begin{gathered} \mathbb{P}\left(\underset{\eta \in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\mathbf{E}}_{\eta }-{\mathbf{E}}_{{\eta }^0}\Vert }_{\max }>u\right)\le np\underset{i,j}{\max }\mathbb{P}\left(\underset{\eta \in {\mathbf{\Theta }}_{np}}{\sup }\left|e_{\eta ij}-e_{{\eta }^{0}ij}\right|>u\right) \\ \le np\underset{i,j}{\max }\mathbb{P}\left(\left|Z\right|>u/\left(M^{1/2}{c}_{np}^{1/2}\right)\right) \\ \le 2np\exp \left(-u^2/\left(c_e^{2}M{c}_{np}\right)\right) \end{gathered}$$

for any u ≥ 0. Thus setting $u=2c_e{M}^{1/2}{c}_{np}^{1/2}{\log }^{1/2}(np)$ with some large enough positive constant M, we obtain that with probability at least 1 − O((np)^−ν),

$$\underset{\eta \in \mathbf{\Theta }}{\sup }{\Vert {\mathbf{E}}_{\eta }-{\mathbf{E}}_{{\eta }^0}\Vert }_{\max }\lesssim c_{np}{\log }^{1/2}(np)=O\left({\tilde{c}}_{np}\right).$$

We will use these results and Lemma 10 in Section D.5 in the proofs below.

To prove (a), we have

$$\begin{gathered} {\Vert {\mathbf{U}}_{\mathcal{A}}(\mathit{\theta })-{\mathbf{U}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)\Vert }_{\max }\le {\Vert n^{-1}{\tilde{\mathbf{X}}}_{\mathcal{A}}{(\mathit{\theta })}^{\prime }{\tilde{\mathbf{X}}}_{\mathcal{A}}(\mathit{\theta })-n^{-1}{\tilde{\mathbf{X}}}_{\mathcal{A}}{\left({\mathit{\theta }}^0\right)}^{\prime }{\tilde{\mathbf{X}}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)\Vert }_{\max } \\ +2{\Vert n^{-1}{\mathbf{X}}_{\mathcal{A}}^{\prime }{\tilde{\mathbf{X}}}_{\mathcal{A}}(\mathit{\theta })-n^{-1}{\mathbf{X}}_{\mathcal{A}}^{\prime }{\tilde{\mathbf{X}}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)\Vert }_{\max }=:U_1+U_2. \end{gathered}$$

Observe that U₁ is further bounded as

$$\begin{gathered} U_1\le {\Vert n^{-1}{\mathbf{C}}_{\mathcal{A}}^{\prime }{\mathbf{C}}_{\mathcal{A}}-n^{-1}{\mathbf{C}}_{\mathcal{A}}^{0^{\prime }}{\mathbf{C}}_{\mathcal{A}}^0\Vert }_{\max }+{\Vert n^{-1}{\mathbf{E}}_{\eta \mathcal{A}}^{\prime }{\mathbf{E}}_{\eta \mathcal{A}}-n^{-1}{\mathbf{E}}_{{\eta }^0\mathcal{A}}^{\prime }{\mathbf{E}}_{{\eta }^0\mathcal{A}}\Vert }_{\max } \\ +2{\Vert n^{-1}{\mathbf{E}}_{\eta \mathcal{A}}^{\prime }{\mathbf{C}}_{\mathcal{A}}-n^{-1}{\mathbf{E}}_{{\eta }^0\mathcal{A}}^{\prime }{\mathbf{C}}_{\mathcal{A}}^0\Vert }_{\max }=:U_{11}+U_{12}+U_{13}. \end{gathered}$$

By Lemma 10, it is easy to see that

$$\begin{gathered} U_{11}\le {\Vert n^{-1}{\left({\mathbf{C}}_{\mathcal{A}}-{\mathbf{C}}_{\mathcal{A}}^0\right)}^{\prime }\left({\mathbf{C}}_{\mathcal{A}}-{\mathbf{C}}_{\mathcal{A}}^0\right)\Vert }_{\max }+2{\Vert n^{-1}{\mathbf{C}}_{\mathcal{A}}^{0^{\prime }}\left({\mathbf{C}}_{\mathcal{A}}-{\mathbf{C}}_{\mathcal{A}}^0\right)\Vert }_{\max } \\ \le n^{-1/2}{\Vert {\mathbf{C}}_{\mathcal{A}}-{\mathbf{C}}_{\mathcal{A}}^0\Vert }_{\max }{\Vert {\mathbf{C}}_{\mathcal{A}}-{\mathbf{C}}_{\mathcal{A}}^0\Vert }_2+2{\Vert n^{-1/2}{\mathbf{C}}_{\mathcal{A}}^0\Vert }_2{\Vert {\mathbf{C}}_{\mathcal{A}}-{\mathbf{C}}_{\mathcal{A}}^0\Vert }_{\max } \\ \lesssim k^{1/2}{\Vert \mathbf{C}-{\mathbf{C}}^0\Vert }_{{\max }^2}+k^{1/2}{\Vert \mathbf{C}-{\mathbf{C}}^0\Vert }_{\max } \\ =O\left(k^{1/2}{c}_{np}^2+k^{1/2}{c}_{np}\right)=O\left(k^{1/2}{c}_{np}\right), \end{gathered}$$

where the last estimate follows from Lemma 3. Similarly, we deduce

$$\begin{gathered} U_{12}\le {\Vert n^{-1}{\left({\mathbf{E}}_{\mathit{\eta }\mathcal{A}}-{\mathbf{E}}_{{\mathit{\eta }}^0\mathcal{A}}\right)}^{\prime }\left({\mathbf{E}}_{\mathit{\eta }\mathcal{A}}-{\mathbf{E}}_{{\mathit{\eta }}^0\mathcal{A}}\right)\Vert }_{\max }+2{\Vert n^{-1}{\mathbf{E}}_{{\mathit{\eta }}^0\mathcal{A}}^{\prime }\left({\mathbf{E}}_{\mathit{\eta }\mathcal{A}}-{\mathbf{E}}_{{\mathit{\eta }}^0\mathcal{A}}\right)\Vert }_{\max } \\ \le n^{-1/2}{\Vert {\mathbf{E}}_{\mathit{\eta }\mathcal{A}}-{\mathbf{E}}_{{\mathit{\eta }}^0\mathcal{A}}\Vert }_{\max }{\Vert {\mathbf{E}}_{\mathit{\eta }\mathcal{A}}-{\mathbf{E}}_{{\mathit{\eta }}^0\mathcal{A}}\Vert }_2+2{\Vert n^{-1/2}{\mathbf{E}}_{{\mathit{\eta }}^0\mathcal{A}}\Vert }_2{\Vert {\mathbf{E}}_{\mathit{\eta }\mathcal{A}}-{\mathbf{E}}_{{\mathit{\eta }}^0\mathcal{A}}\Vert }_{\max } \\ \lesssim k^{1/2}{\Vert {\mathbf{E}}_{\mathit{\eta }}-{\mathbf{E}}_{{\mathit{\eta }}^0}\Vert }_{{\max }^2}+{\Vert {\mathbf{E}}_{\mathit{\eta }}-{\mathbf{E}}_{{\mathit{\eta }}^0}\Vert }_{\max } \\ =O\left(k^{1/2}{\tilde{c}}_{np}^2+{\tilde{c}}_{np}\right) \end{gathered}$$

and

$$\begin{gathered} U_{13}\le {\Vert n^{-1}{\left({\mathbf{E}}_{\mathit{\eta }\mathcal{A}}-{\mathbf{E}}_{{\mathit{\eta }}^0\mathcal{A}}\right)}^{\prime }\left({\mathbf{C}}_{\mathcal{A}}-{\mathbf{C}}_{\mathcal{A}}^0\right)\Vert }_{\max } \\ +{\Vert n^{-1}{\mathbf{E}}_{{\mathit{\eta }}^0\mathcal{A}}^{\prime }\left({\mathbf{C}}_{\mathcal{A}}-{\mathbf{C}}_{\mathcal{A}}^0\right)\Vert }_{\max }+{\Vert n^{-1}{\mathbf{C}}_{\mathcal{A}}^{0^{\prime }}\left({\mathbf{E}}_{\mathit{\eta }\mathcal{A}}-{\mathbf{E}}_{{\mathit{\eta }}^0\mathcal{A}}^0\right)\Vert }_{\max } \\ \le k^{1/2}{\Vert {\mathbf{E}}_{\mathit{\eta }}-{\mathbf{E}}_{{\mathit{\eta }}^0}\Vert }_{\max }{\Vert \mathbf{C}-{\mathbf{C}}^0\Vert }_{\max } \\ +{\Vert n^{-1/2}{\mathbf{E}}_{{\mathit{\eta }}^0\mathcal{A}}\Vert }_2{\Vert \mathbf{C}-{\mathbf{C}}^0\Vert }_{\max }+{\Vert n^{-1/2}{\mathbf{C}}_{\mathcal{A}}^0\Vert }_2{\Vert {\mathbf{E}}_{\mathit{\eta }}-{\mathbf{E}}_{{\mathit{\eta }}^0}^0\Vert }_{\max } \\ =O\left(k^{1/2}{\tilde{c}}_{np}{c}_{np}+c_{np}+k^{1/2}{\tilde{c}}_{np}\right)=O\left(k^{1/2}{\tilde{c}}_{np}\right). \end{gathered}$$

Combining these bounds of U₁₁–U₁₃, we have

$$U_1\le U_{11}+U_{12}+U_{13}\lesssim k^{1/2}{\tilde{c}}_{np}.$$

This holds uniformly in θ ∈ Θ_np with probability at least 1 − O(π_np) by Lemma 3 and the discussion above. Next we obtain

$$\begin{gathered} U_2\le {\Vert n^{-1}{\mathbf{C}}_{\mathcal{A}}^{0^{\prime }}\left({\mathbf{C}}_{\mathcal{A}}-{\mathbf{C}}_{\mathcal{A}}^0\right)\Vert }_{\max }+{\Vert n^{-1}{\mathbf{C}}_{\mathcal{A}}^{0^{\prime }}\left({\mathbf{E}}_{\mathit{\eta }\mathcal{A}}-{\mathbf{E}}_{{\mathit{\eta }}^0\mathcal{A}}\right)\Vert }_{\max } \\ +{\Vert n^{-1}{\mathbf{E}}_{\mathcal{A}}^{\prime }\left({\mathbf{C}}_{\mathcal{A}}-{\mathbf{C}}_{\mathcal{A}}^0\right)\Vert }_{\max }+{\Vert n^{-1}{\mathbf{E}}_{\mathcal{A}}^{\prime }\left({\mathbf{E}}_{\mathit{\eta }\mathcal{A}}-{\mathbf{E}}_{{\mathit{\eta }}^0\mathcal{A}}\right)\Vert }_{\max } \\ \le {\Vert n^{-1/2}{\mathbf{C}}_{\mathcal{A}}^0\Vert }_2{\Vert {\mathbf{C}}_{\mathcal{A}}-{\mathbf{C}}_{\mathcal{A}}^0\Vert }_{\max }+{\Vert n^{-1/2}{\mathbf{C}}_{\mathcal{A}}^0\Vert }_2{\Vert {\mathbf{E}}_{\mathit{\eta }\mathcal{A}}-{\mathbf{E}}_{{\mathit{\eta }}^0\mathcal{A}}\Vert }_{\max } \\ +{\Vert n^{-1/2}{\mathbf{E}}_{{\mathit{\eta }}^0\mathcal{A}}\Vert }_2{\Vert {\mathbf{C}}_{\mathcal{A}}-{\mathbf{C}}_{\mathcal{A}}^0\Vert }_{\max }+{\Vert n^{-1/2}{\mathbf{E}}_{{\mathit{\eta }}^0\mathcal{A}}\Vert }_2{\Vert {\mathbf{E}}_{\mathit{\eta }}-{\mathbf{E}}_{{\mathit{\eta }}^0}\Vert }_{\max } \\ =O\left(k^{1/2}{c}_{np}+k^{1/2}{\tilde{c}}_{np}+c_{np}+{\tilde{c}}_{np}\right) \\ =O\left(k^{1/2}{\tilde{c}}_{np}\right). \end{gathered}$$

This also holds uniformly in θ ∈ Θ_np with probability at least 1 − O(π_np) by Lemma 3 and the discussion above. Consequently, it holds that

$$\underset{\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\mathbf{U}}_{\mathcal{A}}(\mathit{\theta })-{\mathbf{U}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)\Vert }_{\max }\lesssim k^{1/2}{\tilde{c}}_{np}$$

with probability at least 1 − O(π_np).

To prove (b), we have

$$\begin{gathered} {\Vert {\mathbf{v}}_{\mathcal{A}}(\mathit{\theta })-{\mathbf{v}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)\Vert }_{\max }\le {\Vert n^{-1}{\tilde{\mathbf{X}}}_{\mathcal{A}}{(\mathit{\theta })}^{\prime }\mathbf{y}-n^{-1}{\tilde{\mathbf{X}}}_{\mathcal{A}}{\left({\mathit{\theta }}^0\right)}^{\prime }\mathbf{y}\Vert }_{\max } \\ \le {\Vert n^{-1}{\tilde{\mathbf{X}}}_{\mathcal{A}}{(\mathit{\theta })}^{\prime }\mathbf{X}\beta -n^{-1}{\tilde{\mathbf{X}}}_{\mathcal{A}}{\left({\mathit{\theta }}^0\right)}^{\prime }\mathbf{X}\beta \Vert }_{\max }+{\Vert n^{-1}{\tilde{\mathbf{X}}}_{\mathcal{A}}{(\mathit{\theta })}^{\prime }\epsilon -n^{-1}{\tilde{\mathbf{X}}}_{\mathcal{A}}{\left({\mathit{\theta }}^0\right)}^{\prime }\epsilon \Vert }_{\max } \\ =:V_1+V_2. \end{gathered}$$

First, because $\mathbf{X}\beta ={\mathbf{X}}_{{\mathcal{S}}^0}{\beta }_{{\mathcal{S}}^0}$ we see that

$$\begin{gathered} V_1\le s^{1/2}{\Vert n^{-1}{\tilde{\mathbf{X}}}_{\mathcal{A}}{(\mathit{\theta })}^{\prime }{\mathbf{X}}_{{\mathcal{S}}^0}-n^{-1}{\tilde{\mathbf{X}}}_{\mathcal{A}}{\left({\mathit{\theta }}^0\right)}^{\prime }{\mathbf{X}}_{{\mathcal{S}}^0}\Vert }_{\max }{\Vert {\beta }_{{\mathcal{S}}^0}\Vert }_2 \\ \lesssim s{\Vert n^{-1}{\tilde{\mathbf{X}}}_{\mathcal{A}}{(\mathit{\theta })}^{\prime }{\mathbf{X}}_{{\mathcal{S}}^0}-n^{-1}{\tilde{\mathbf{X}}}_{\mathcal{A}}{\left({\mathit{\theta }}^0\right)}^{\prime }{\mathbf{X}}_{{\mathcal{S}}^0}\Vert }_{\max }. \end{gathered}$$

Recall that $\left|{\mathcal{S}}^0\right|=s$ and s ≤ n ∧ p. By a similar bound of U₂, the norm just above can be bounded further as

$$\begin{gathered} {\Vert n^{-1/2}{\mathbf{C}}_{{\mathcal{S}}^0}^0\Vert }_2{\Vert {\mathbf{C}}_{\mathcal{A}}-{\mathbf{C}}_{\mathcal{A}}^0\Vert }_{\max }+{\Vert n^{-1/2}{\mathbf{C}}_{{\mathcal{S}}^0}^0\Vert }_2{\Vert {\mathbf{E}}_{\mathit{\eta }\mathcal{A}}-{\mathbf{E}}_{{\mathit{\eta }}^0\mathcal{A}}\Vert }_{\max } \\ +{\Vert n^{-1/2}{\mathbf{E}}_{{\mathit{\eta }}^0{\mathcal{S}}^0}\Vert }_2{\Vert {\mathbf{C}}_{\mathcal{A}}-{\mathbf{C}}_{\mathcal{A}}^0\Vert }_{\max }+{\Vert n^{-1/2}{\mathbf{E}}_{{\mathit{\eta }}^0{\mathcal{S}}^0}\Vert }_2{\Vert {\mathbf{E}}_{\mathit{\eta }\mathcal{A}}-{\mathbf{E}}_{{\mathit{\eta }}^0\mathcal{A}}\Vert }_{\max } \\ \lesssim s^{1/2}{\Vert \mathbf{C}-{\mathbf{C}}^0\Vert }_{\max }+s^{1/2}{\Vert {\mathbf{E}}_{\mathit{\eta }}-{\mathbf{E}}_{{\mathit{\eta }}^0}\Vert }_{\max }+{\Vert \mathbf{C}-{\mathbf{C}}^0\Vert }_{\max }+{\Vert {\mathbf{E}}_{\mathit{\eta }}-{\mathbf{E}}_{{\mathit{\eta }}^0}\Vert }_{\max } \\ =O\left(s^{1/2}{c}_{np}+s^{1/2}{\tilde{c}}_{np}+c_{np}+{\tilde{c}}_{np}\right)=O\left(s^{1/2}{\tilde{c}}_{np}\right). \end{gathered}$$

Thus we have

$$V_1\lesssim s{s}^{1/2}{\tilde{c}}_{np}=s^{3/2}{\tilde{c}}_{np}$$

with probability at least 1 − O(π_np). Next the same procedure yields

$$\begin{gathered} V_2\le {\Vert {\tilde{\mathbf{X}}}_{\mathcal{A}}(\mathit{\theta })-{\tilde{\mathbf{X}}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)\Vert }_{\max }{\Vert n^{1/2}\epsilon \Vert }_2 \\ \lesssim {\Vert \mathbf{C}-{\mathbf{C}}^0\Vert }_{\max }+{\Vert {\mathbf{E}}_{\mathit{\eta }}-{\mathbf{E}}_{{\mathit{\eta }}^0}\Vert }_{\max }\lesssim {\tilde{c}}_{np}, \end{gathered}$$ (A.24)

where ${\Vert n^{1/2}\epsilon \Vert }_2={\left(\mathbb{E}{\epsilon }^2\right)}^{1/2}+o(1)$ a.s. by the law of large numbers for independent random variables. Since the results hold uniformly in θ ∈ Θ_np, combining them leads to

$$\underset{\mathit{\theta }\in {\mathbf{\Theta }}_{np}}{\sup }{\Vert {\mathbf{v}}_{\mathcal{A}}(\mathit{\theta })-{\mathbf{v}}_{\mathcal{A}}\left({\mathit{\theta }}^0\right)\Vert }_{\max }\lesssim s^{3/2}{\tilde{c}}_{np}$$

with probability at least 1 − O(π_np). This concludes the proof of Lemma 8.

D.4. Lemma 9 and its proof

Lemma 9

Assume that Condition 4 holds. Then the following statements hold

1. We have

   $$\mathbb{P}\left({\Vert {(n\vee p)}^{-1/2}\mathbf{E}\Vert }_2\lesssim 1\right)\ge 1-2\exp (-|O(n\vee p)|);$$
2. For any fixed set $\mathcal{A}$ with $|\mathcal{A}|\le k\le n$, we have

   $$\mathbb{P}\left({\Vert n^{-1/2}{\mathbf{E}}_{\mathcal{A}}\Vert }_2\lesssim 1\right)\ge 1-2p^{-\nu };$$
3. For all k ≤ n, we have

   $$\mathbb{P}\left(\underset{\left|\mathcal{A}\right|\le k}{\max }{\Vert n^{-1/2}{\mathbf{E}}_{\mathcal{A}}\Vert }_2\lesssim 1\vee {\left(n^{-1}k\log p\right)}^{1/2}\right)\ge 1-2p^{-\nu },$$

where ν > 0 is a predetermined constant.

Proof. Result (a) is obtained by Theorem 5.39 of [40]. Moreover, by the same theorem there exist some positive constants c and C such that for any $\mathcal{A}$ with $|\mathcal{A}|\le k\le n$ and every t ≥ 0,

$$\mathbb{P}\left({\sigma }_e^{-1}{\Vert n^{-1/2}{\mathbf{E}}_{\mathcal{A}}\Vert }_2>1+C+n^{-1/2}t\right)\le 2\exp \left(-c{t}^2\right),$$

where ${\sigma }_e^2=\mathbb{E}{e}^2$. Therefore, result (b) is immediately obtained by putting t² = c⁻¹ν log p since n^−1/2t = o(1) and exp (−ct²) = p^−v in this case.

For (c), taking the union bound leads to

$$\begin{gathered} \mathbb{P}\left({\sigma }_e^{-1}{\max }_{\left|\mathcal{A}\right|\le k}{\Vert n^{-1/2}{\mathbf{E}}_{\mathcal{A}}\Vert }_2>1+C+n^{-1/2}t\right) \\ \le \left(\begin{array}{l}p\\ k\end{array}\right)\underset{\left|\mathcal{A}\right|\le k}{\max }\mathbb{P}\left({\sigma }_e^{-1}{\Vert n^{-1/2}{\mathbf{E}}_{\mathcal{A}}\Vert }_2>1+C+n^{-1/2}t\right)\le 2p^k\exp \left(-c{t}^2\right). \end{gathered}$$

Set t² = c⁻¹(ν + k)log p in this inequality. Then we have n^1/2t = O((n⁻¹k log p)^1/2) and

$$2p^k\exp \left(-c{t}^2\right)\le 2p^k\exp (-(\nu +k)\log p)=2p^{-\nu },$$

which gives result (c) and completes the proof of Lemma 9.

D.5. Lemma 10 and its proof

Lemma 10

For matrices $\mathbf{A}\in {ℝ}^{k_1\times n}$ and $\mathbf{B}\in {ℝ}^{n\times k_2}$, we have ‖AB‖_max ≤ n^1/2‖A‖₂‖B‖_max and ‖AB‖_max ≤ n^1/2‖A‖_max‖B‖₂.

Proof. For any matrix M = (m_ij) ∈ ℝ^k×n, let ‖M‖_∞,∞ denote the induced ℓ_∞-norm. First, we have

$$\Vert \mathbf{M}{\Vert }_{\infty ,\infty }:=\underset{\mathbf{v}\in {ℝ}^n\backslash \left\{0\right\}}{\sup }\frac{\Vert \mathbf{M}v{\Vert }_{\max }}{\Vert \mathbf{v}{\Vert }_{\max }}\le \underset{\mathbf{v}\in {ℝ}^n\backslash \left\{0\right\}}{\sup }\frac{\Vert \mathbf{M}v{\Vert }_2}{\Vert \mathbf{v}{\Vert }_2}\frac{\Vert \mathbf{v}{\Vert }_2}{\Vert \mathbf{v}{\Vert }_{\max }}\le n^{1/2}\Vert \mathbf{M}{\Vert }_2.$$

Therefore, by a simple calculation we see that

$$\begin{gathered} \Vert \mathbf{A}B{\Vert }_{\max }=\Vert \mathrm{vec}(\mathbf{A}B){\Vert }_{\max }={\Vert \left({\mathbf{I}}_{k_2}\otimes \mathbf{A}\right)\mathrm{vec}(\mathbf{B})\Vert }_{\max } \\ =\frac{{\Vert \left({\mathbf{I}}_{k_2}\otimes \mathbf{A}\right)\mathrm{vec}(\mathbf{B})\Vert }_{\max }}{\Vert \mathrm{vec}(\mathbf{B}){\Vert }_{\max }}\Vert \mathrm{vec}(\mathbf{B}){\Vert }_{\max } \\ \le {\Vert {\mathbf{I}}_{k_2}\otimes \mathbf{A}\Vert }_{\infty ,\infty }\Vert \mathrm{vec}(\mathbf{B}){\Vert }_{\max }=\Vert \mathbf{A}{\Vert }_{\infty ,\infty }\Vert \mathbf{B}{\Vert }_{\max }\le n^{1/2}\Vert \mathbf{A}{\Vert }_2\Vert \mathbf{B}{\Vert }_{\max }. \end{gathered}$$

The second assertion follows from applying this inequality to B′A′. This concludes the proof of Lemma 10.

E. Additional numerical details and results

E.1. Estimation procedure

In implementing the IPAD algorithm suggested in Section 2, we use the PC_p1 criterion proposed in [3] to estimate the number of factors r. With an estimated number of factors $\widehat{r}$, we use the principle component method discussed in Section 3.2 to obtain an estimate $\widehat{\mathbf{C}}$ of matrix C⁰. Denote by $\widehat{\mathbf{E}}=\left({\widehat{e}}_{ij}\right)=\mathbf{X}-\widehat{\mathbf{C}}$. Recall that in the construction of knockoff variables, the distribution of E needs to be estimated. Throughout our simulation studies, we misspecify the model and treat the entries of E as i.i.d. Gaussian random variables. Under this working model assumption, the only unknown parameter is the variance which can be estimated by the following maximum likelihood estimator

$${\widehat{\sigma }}^2={(np)}^{-1}\sum _{i=1}^n\sum _{j=1}^p{\widehat{e}}_{ij}^2.$$

Then the knockoffs matrix $\widehat{\mathbf{X}}$ is constructed using (8) with the entries of ${\mathbf{E}}_{\widehat{\mathit{\eta }}}$ drawn independently from $\mathcal{N}\left(0,{\widehat{\sigma }}^2\right)$. For the two comparison methods BCKnockoff and HD-BCKnockoff, we follow the implementation in [4] and [5], respectively. Thus it is seen that neither BCKnockoff nor HD-BCKnockoff uses the factor structure in X when constructing the knockoff variables.

In Designs 1–3, with the constructed empirical knockoffs matrix $\widehat{\mathbf{X}}$ we apply the Lasso method to fit the model with y the response vector and $[\mathbf{X},\widehat{\mathbf{X}}]$ the augmented design matrix. The value of the regularization parameter λ is chosen by the tenfold cross-validation. Then the LCD discussed in Section 2.2 is used in the construction of knockoff statistics. In Design 4, we assume the nonlinear relationship between the response and the covariates. In this case, random forest is used for estimation of the model. To construct the knockoff statistics, we use the variable importance measure of mean decrease accuracy (MDA) introduced in [14]. This measure is based on the idea that if a variable is unimportant, then rearranging its values should not degrade the prediction accuracy. The MDA for the jth variable, denoted as ${\widehat{\text{MDA}}}_j$, measures the amount of increase in prediction error when the values of the jth variable in the out-of-sample prediction are permuted randomly. Then intuitively, ${\widehat{\text{MDA}}}_j$ will be small and around zero if the jth variable is unimportant in predicting the response. For each original variable x_j, we compute W_j statistic as $\left|{\widehat{\text{MDA}}}_j\right|-\left|{\widehat{\text{MDA}}}_{j+p}\right|,j=1,\cdots ,p$.

E.2. Simulation study

To evaluate the performance of IPAD approach in terms of empirical FDR and power with real economic data, we set up one additional Monte Carlo simulation study. In this design, we use the transformed macroeconomic variables described above as the design matrix X, but simulate response y from the model in Design 1 in Section 4.1. We set the number of true signals, the amplitude of signals, and the target FDR level to s = 10, A = 4, and q = 0.2, respectively.

Table 7 shows the results for IPAD and HD-BCKnockoff approaches. As expected, HD-BCKnockoff can control FDR but suffers from lack of power. On the other hand, IPAD has empirical FDR slightly higher than the target level (q = 0.2) while its power is reasonably high. These results are consistent with our theory in Section 3 because IPAD only controls FDR asymptotically. Additional reason for having slightly higher FDR than the target level can be deviation of the design matrix from our factor model assumption. Overall this simulation study indicates that IPAD can control FDR at around the target level with reasonably high power when we use the macroeconomic data set. In the next section, using the same data set we will compare the forecasting performance of IPAD with that of some commonly used forecasting methods in the literature.

Table 7:

Real data simulation results with (n,p) = (195,109)

	FDR	Power	FDR+	Power+	R2
	c = 0.2
IPAD	0.278	0.812	0.223	0.796	0.747
HD-BCKnockoff	0.096	0.009	0.010	0.002	0.758
	c = 0.3
IPAD	0.280	0.757	0.221	0.723	0.665
HD-BCKnockoff	0.149	0.121	0.027	0.036	0.678
	c = 0.5
IPAD	0.286	0.661	0.215	0.571	0.560
HD-BCKnockoff	0.119	0.009	0.008	0.001	0.554

E.3. Methods of comparison in empirical analysis

We compare the following different methods in the empirical analysis presented in Section 5, where each method is implemented in a same way as IPAD for one-step ahead prediction.

1. Autoregression of order one (AR(1)). Assume that

   $$y_t={\alpha }_0+\rho y_{t-1}+{\epsilon }_t,$$

   where y_t is regressed on y_t−1, and α₀ and ρ are the AR(1) coefficients that need to be estimated. With the ordinary least squares estimates ${\widehat{\alpha }}_0$ and $\widehat{\rho }$, the one-step ahead prediction based on this model is ${\widehat{y}}_{T+1}={\widehat{\alpha }}_0+\widehat{\rho }{y}_T$.
2. Factor augmented AR(1) (FAR). We first extract m factors f₁,⋯, f_m form the 109 transformed macroeconomic variables by principal component analysis (PCA). Denote by ${\tilde{\mathbf{f}}}_t\in {ℝ}^m$ the factor vector at time t extracted from the rows of matrix [f₁,⋯, f_m] ∈ ℝ^n×m. Then we regress y_t on y_t−1 and ${\tilde{\mathbf{f}}}_{t-1}$ and fit the following model

   $$y_t={\alpha }_0+\rho y_{t-1}+{\gamma }^{\prime }{\tilde{\mathbf{f}}}_{t-1}+{\epsilon }_t$$

   with γ ∈ ℝ^m. The number of factors m is determined using the PC_p1 criterion in [3]. Similar to AR(1) model, one-step ahead forecast of y_t at time T is

   $${\widehat{y}}_{T+1}={\widehat{\alpha }}_0+\widehat{\rho }{y}_T+{\widehat{\mathit{\gamma }}}^{\prime }{\tilde{\mathbf{f}}}_T.$$
3. Lasso method. The y_t is regressed on y_t−1, ${\tilde{\mathbf{f}}}_{t-1}$, and the 108 transformed macroeconomic variables z_t−1 ∈ ℝ¹⁰⁸ at time t − 1

   $$y_t={\alpha }_0+\rho y_{t-1}+{\gamma }^{\prime }{\tilde{\mathbf{f}}}_{t-1}+{\delta }^{\prime }{\mathbf{z}}_{t-1}+{\epsilon }_t,$$

   where ${\tilde{\mathbf{f}}}_t$ is the same as in the FAR(1) model, and α₀, ρ, and δ ∈ ℝ¹⁰⁸ are regression coefficients that need to be estimated. The coefficients are estimated by Lasso method with regularization parameter chosen by the cross-validation. With the estimated Lasso coefficient vector ${\widehat{\mathit{\beta }}}_{\text{Lasso}}$, one-step ahead forecast of y_t at time T is

   $${\widehat{y}}_{T+1}={\widehat{\beta }}_{\text{Lasso}}^{\prime }{\mathbf{x}}_T,$$

   where x_T is the augmented predictor vector at time T.
4. IPAD method. We regress y_t on the augmented vector ${\left(y_{t-1},{\mathbf{z}}_{t-1}^{\prime }\right)}^{\prime }$. The lagged variable y_t−1 is assumed to be always in the model. To account for this, we implement IPAD in three steps. First, we regress y_t on y_t−1 and obtain the residuals e_y,t. Second, we regress each of the 108 variables in z_t−1 on y_t−1 and obtain the residual vector e_z,t−1. At last, we fit model (1)–(2) using the IPAD approach by treating e_y,t as the response and e_z,t−1 as predictors, which returns us a set of selected variables (a subset of the 108 macroeconomic variables). With the set of variables $\widehat{\mathcal{S}}$ selected by IPAD, we fit the following model by the least-squares regression

   $$y_t={\alpha }_0+\rho y_{t-1}+{\delta }^{\prime }{\mathbf{z}}_{t-1,\widehat{\mathcal{S}}}+{\epsilon }_t,$$ (A.25)

   where ${\mathbf{z}}_{t,\widehat{\mathcal{S}}}$ stands for the subvector of z_t corresponding to the set of variables $\widehat{\mathcal{S}}$ selected by IPAD at time t. Since $\widehat{\mathcal{S}}$ from IPAD is random due to the randomness in generating knockoff variables, we apply the IPAD procedure 100 times and compute the average of these 100 one-step ahead predictions based on (A.25) and use the mean value as the final predicted value of y_T+1.