MODEL-FREE FORWARD SCREENING VIA CUMULATIVE DIVERGENCE

Abstract

Feature screening plays an important role in the analysis of ultrahigh dimensional data. Due to complicated model structure and high noise level, existing screening methods often suffer from model misspecification and the presence of outliers. To address these issues, we introduce a new metric named cumulative divergence (CD), and develop a CD-based forward screening procedure. This forward screening method is model-free and resistant to the presence of outliers in the response. It also incorporates the joint effects among covariates into the screening process. With a data-driven threshold, the new method can automatically determine the number of features that should be retained after screening. These merits make the CD-based screening very appealing in practice. Under certain regularity conditions, we show that the proposed method possesses sure screening property. The performance of our proposal is illustrated through simulations and a real data example.

1. INTRODUCTION

Regression analysis with ultrahigh dimensional covariates arises in many scientific fields such as agriculture, biomedicine, economics, finance, and genetics. It is desirable to identify the important covariates that are truly influential to the response. Traditional best subset selection methods are computationally infeasible in the presence of ultra-high dimensional covariates. In the past two decades, many regularization methods, such as LASSO (Tibshirani, 1996), SCAD (Fan and Li, 2001), adaptive LASSO (Zou, 2006), and Dantzig selector (Candes and Tao, 2007), have been proposed for variable selection. However, when the covariates are ultrahigh dimensional, Fan et al. (2009) stated that these regularization methods suffer from the simultaneous challenges of computational expediency, statistical accuracy, and algorithmic stability.

To deal with ultrahigh dimensionality, Fan and Lv (2008) suggested screening out most unimportant covariates before implementing an elaborative variable selection. They proposed a sure independent screening procedure (SIS) for linear models using marginal Pearson correlation between each covariate and the response. Since the seminal work of Fan and Lv (2008), feature screening has received extensive attention in the past decade. In particular, Wang (2009) proposed a forward regression and Chang et al. (2013) suggested a marginal likelihood ratio test to screen out unimportant covariates in linear models. Li et al., (2012) suggested replacing Pearson correlation with Kendall’s rank correlation in the presence of outliers. Ma, Li and Tsai (2017) proposed quantile partial correlation for feature screening in linear quantile regression. Fan and Song (2010) and Xu and Chen (2014) suggested maximum likelihood estimate and Mai and Zou (2013) proposed Kolmogorov-Smirnov statistic to screen out unimportant features in generalized linear models. Fan et al. (2011) and He, Wang and Hong (2013) suggested nonparametric screening procedures for additive models. Song et al. (2014) proposed an independent screening procedure for varying coefficient models. These model-based screening procedures are effective if the working model is close to the underlying true model, and may be very ineffective otherwise.

To minimize the impact of model misspecification, several model-free screening methods have been developed. For instance, Zhu et al., (2011) proposed a sure independent ranking and screening procedure for a general class of index models. Li, Zhong and Zhu (2012) suggested distance correlation for feature screening, which can simultaneously deal with grouped covariates and multivariate response. Shao and Zhang (2014) introduced martingale difference correlation to perform screening as long as the mean function of the response is concerned. These model-free methods are favored when we are lack of prior information on the regression structure. However, most of them are based on marginal correlations and are vulnerable in the presence of outliers.

In the present work, we develop a model-free forward screening procedure for ultra-high dimensional data. Forward screening is related to but much more challenging than conditional screening. For conditional screening, the conditioning set is fixed. However, for our proposed forward screening procedure, the conditioning set is iteratively updated in a data-driven fashion. Moreover, existing conditional screening procedures are model-based and there is little literature on model-free conditional screening (Wang, 2009; Xu and Chen, 2014; Barut, Fan and Verhasselt, 2016). To the best of our knowledge, how to design a model-free forward screening has not been studied yet. We aim to fill in this gap in this paper. To this end, we first introduce the concept of cumulative divergence (CD), a new correlation metric to characterize functional dependence. We show that the CD is robust to the presence of outliers in the conditioning variable. We further propose a CD-based forward screening procedure. At each step of the forward screening, a new covariate will be added to an active index set based on its conditional CD with the response. This procedure stops when the conditional CD of all remaining covariates is less than a certain threshold. Compared with marginal screening methods, the forward screening incorporates the joint correlation among the covariates. With a data-driven threshold, our proposal can adaptively determine the number of features that should be retained after screening. Therefore, it is convenient for implementation without ad hoc tuning steps. Due to its robust property, our proposal performs well even when the underlying true model is misspecified. It is also robust in the presence of outliers. This appealing property makes the CD-based forward screening attractive for handling ultrahigh dimensional noisy data. Under some regularity conditions, we show that our forward screening method possesses the sure screening property in the terminology of Fan and Lv (2008). We further demonstrate the finite sample performance of the proposed procedures through simulations and a real data example.

We summarize the major contributions of this paper as follows. (1) The proposed forward screening approach is distinguished from marginal screening approaches in that the joint correlations among the covariates are taken into account by the proposed forward screening procedure and yet are ignored by the marginal screening methods (Zhu et al.,, 2011; Li, Zhong and Zhu, 2012). (2) The proposed forward screening procedure is model-free, and hence robust to model misspecification. Thus, the proposed procedure is different from existing model-based forward regression and conditional screening methods (Wang, 2009; Xu and Chen, 2014; Barut, Fan and Verhasselt, 2016). This model-free property is very appealing in ultrahigh dimensional data analysis, especially when we are often lack of information on the underlying regression structure. (3) We propose the CD to quantify deviation from mean independence. The CD is robust to the presence of outliers in the conditioning variable, is thus different from the martingale difference correlation (Shao and Zhang, 2014). Our proposed CD-based forward screening approach inherits this robustness property and is robust to the presence of outliers in the response.

This paper is organized as follows. In Section 2, we introduce the notion of cumulative divergence and study its properties. In Section 3, we propose a model-free forward screening procedure and establish its sure screening property. In Section 4, we assess the finite sample performance of our proposed forward screening procedure through comprehensive numerical studies. Some concluding remarks are given in Section 5. All technical details are relegated to the Appendix and a supplementary document.

2. THE CUMULATIVE DIVERGENCE

In each step of the forward screening procedure to be developed, we have to determine whether a covariate should be selected through testing whether the conditional mean function of the response variable is independent of this covariate. This motivates us to start with a simplified problem by testing mean independence that

$$H_0:E(Y|X)=E(Y)\hspace{0.17em}\text{almost}\hspace{0.17em}\text{surely}\hspace{0.17em}\hspace{0.17em}\text{versus}\hspace{0.17em}\hspace{0.17em}{H}_1:\text{otherwise}.$$ (2.1)

Let ($\tilde{X}$, $\tilde{Y}$) be an independent copy of (X, Y). We assume var(X) > 0 and 0 < var(Y) < ∞ throughout. We do not require var(X) < ∞. Let “⇔” stand for “the statements on both the left and the right hand sides are equivalent”, and supp(X) stand for the support of the conditioning variable X. We first note that

$$\begin{gathered} \hspace{1em}\hspace{1em}E(Y|X)=E(Y)\hspace{0.17em}\hspace{0.17em}\text{almost}\hspace{0.17em}\text{surely} \\ \iff \hspace{1em}E(Y|X<x_0)=E(Y),\hspace{1em}\text{for}\hspace{0.17em}\text{all}\hspace{0.17em}{x}_0\in \text{supp}(X) \\ \iff \hspace{1em}\mathrm{cov}\{Y,\mathbf{1}(X<x_0)\}=0,\hspace{1em}\text{for}\hspace{0.17em}\text{all}\hspace{0.17em}{x}_0\in \text{supp}(X) \\ \iff \hspace{1em}E[{\mathrm{cov}}^2\{Y,\mathbf{1}(X<\tilde{X})|\tilde{X}\}]=0. \end{gathered}$$ (2.2)

This motivates us to define the cumulative covariance (CCov) and the CD as follows.

Definition 2.1. Assume var(X) > 0 and 0 < var(Y) < ∞. The cumulative covariance, denoted CCov(Y | X), and the cumulative divergence, denoted CD(Y | X), between random variables X and Y are defined, respectively, by

$$\text{CCov}(Y|X)\stackrel{\text{def}}{=}E[{\mathrm{cov}}^2\{Y,\mathbf{1}(X<\tilde{X})|\tilde{X}\}]\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\mathit{\text{and}}$$ (2.3)

$$\text{CD}(Y|X)\stackrel{\text{def}}{=}\text{CCov}(Y|X)/\mathrm{var}(Y).$$ (2.4)

The definition of CD allows for var(X) = ∞, indicating that the distribution of X can be heavy-tailed. Since the rank of X is used in the definition of CCov(Y | X), this also indicates that CD(Y | X) is robust to outliers in the conditioning variable X. The following theorem states that the CD possesses several other appealing properties.

Theorem 1. The CD has the following properties.

1. Assume var(X) > 0 and 0 < var(Y) < ∞, then 0 ≤ CD(Y | X) ≤ ¼ and CD(Y | X) = 0 if and only if E(Y | X) = E(Y) almost surely. In addition, CD(X | Y) = CD(Y | X) = 0 if F (y | X) = F (y) for all $y\in ℝ$, where $F(y|X)\stackrel{def}{=}pr(Y<y|X)$ and F(y) = pr(Y < y), for $y\in ℝ$.

2. For a, $b\in ℝ$ with a ≠ 0, and an arbitrary strictly monotone transformation M(X), CD(Y | X) = CD{aY + b | M (X)}.

3. If X and Y are jointly normal with Pearson correlation ρ, then $\text{CD}(Y|X)=\text{CD}(X|Y)={\rho }^2/(2\sqrt{3}\pi )$. In particular, $\text{CD}(X|X)=1/(2\sqrt{3}\pi )$.

4. Let $\tilde{X}$ be an independent copy of X. If Y is normal and all involved moments exist, $\text{CD}(Y|X)/\mathrm{var}(Y)=E[E^2\{\partial F(\tilde{X}|Y)/\partial Y|\tilde{X}\}]$.

The first assertion of Theorem 1 indicates that CD(Y | X) is a useful measure to detect whether the conditional mean function of Y depends on X functionally. In particular, CD(Y | X) = 0 if and only if E(Y | X) = E(Y). This ensures that the CD is a useful tool to test (2.1). In general, CD(Y | X) ≠ CD(X | Y ) even if var(X) = var(Y). If X and Y are independent, then CD(Y | X) = CD(X | Y ) = 0; and if X and Y are jointly normal, CD(Y | X) = CD(X | Y ).

The second assertion of Theorem 1 indicates that the CD is invariant with respect to strictly monotone transformation of X. This invariant property matches the fact that E(Y | X) = E{Y | M (X)} and is however not shared by other popular correlation measures, such as Pearson correlation, martingale difference (Shao and Zhang, 2014), or distance correlation (Székely, Rizzo and Bakirov, 2007; Székely and Rizzo, 2009). This property implies that the CD is robust against model misspecification and the presence of outliers, because it merely uses the rank rather than the observed values of X. The virtue of robustness makes the associated forward screening procedure to be developed in Section 3 potentially attractive for ultrahigh dimensional noisy data.

The third assertion of Theorem 1 implies that, when X and Y are jointly normal with Pearson correlation ρ and unit variance, our proposed CD is closely related to other popular correlation measures through ρ. In particular, Kendall’s rank correlation (Huber and Ronchetti, 2009) equals to 2 arcsin(ρ)/π, the squared martingale difference correlation equals to ${\rho }^2{\{4(1-\sqrt{3}+\pi /3)\}}^{-1/2}$, and the squared distance correlation is $\{\rho \arcsin (\rho )+{(1-{\rho }^2)}^{1/2}-\rho \arcsin (\rho /2)-{(4-{\rho }^2)}^{1/2}+1\}/(1+\pi /3-\sqrt{3})$.

A sample version of the CD can be conveniently constructed. Specifically, let {(X_i, Y_i), i = 1, …, n} be a random sample from the joint distribution of (X, Y). We estimate CCov(Y | X) and CD(Y | X) respectively by

$$\begin{gathered} \widehat{\text{CCov}}(Y|X)\stackrel{\text{def}}{=}{n}^{-3}\sum _{j=1}^n{[\sum _{i=1}^n(Y_i-\overline{Y})\{\mathbf{1}(X_i<X_j)-F_n(X_j)\}]}^2\hspace{0.17em}\hspace{0.17em}\text{and} \\ \hspace{0.17em}\hspace{0.17em}\widehat{\text{CD}}(Y|X)\stackrel{\text{def}}{=}\widehat{\text{CCov}}(Y|X)/\widehat{\mathrm{var}}(Y), \end{gathered}$$ (2.5)

where

$$\overline{Y}\stackrel{\text{def}}{=}{n}^{-1}\sum _{i=1}^n{Y}_i,\hspace{0.17em}\hspace{0.17em}{F}_n(X_j)\stackrel{\text{def}}{=}{n}^{-1}\sum _{i=1}^n\mathbf{1}(X_i<X_j)\hspace{1em}\text{and}\hspace{1em}\widehat{\mathrm{var}}(Y)\stackrel{\text{def}}{=}{n}^{-1}\sum _{i=1}^n{(Y_i-\overline{Y})}^2.$$

To decide critical values in the test for the hypothesis (2.1), we propose a wild bootstrap procedure as follows. Define ${\epsilon }_i=Y_i-\overline{Y}$ and $Y_i^{\ast }=\overline{Y}+a_i{\epsilon }_i$, where a_i satisfies pr(a_i = 1) = pr(a_i = −1) = ½. The wild bootstrap sample is $\{(X_i,Y_i^{\ast }),\hspace{0.17em}i=1\dots ,n\}$. We repeat the wild bootstrap procedure m times to obtain ${\widehat{\text{CD}}}^{(1)}(Y^{\ast }|X),\hspace{0.17em}\dots \hspace{0.17em},{\widehat{\text{CD}}}^{(m)}(Y^{\ast }|X)$. Denote τ the (1 – α)-th quantile of $\left\{{\widehat{\text{CD}}}^{\left(1\right)}(Y^{*}|X),\dots ,{\widehat{\text{CD}}}^{\left(m\right)}(Y^{*}|X)\right\}$. We reject H₀ at the significance level α if CD(Y | X) calculated from the original sample {(X_i, Y_i), i = 1, …, n} is greater than τ and accept H₀ otherwise.

We conduct a simulation study to compare the finite-sample performance of the CD with that of four commonly-used correlation: Pearson correlation, rank correlation, distance correlation and martingale difference correlation. We consider two scenarios for generating the conditioning variable X. In the first scenario X is standard normal and in the second scenario X follows Cauchy distribution. Let Y = c exp(−X²) + ε, where ε ~ N(0, 1). We set c = 0.0, 0.5, 1.0, 1.5 and 2.0. The null hypothesis H₀ in (2.1) holds true when c = 0. We set the sample size n = 100 and summarize the simulation results in Figure 1 when the significance level α = 0.05.

Figure 1.

The power curves of the Pearson correlation test (dashed line marked with circles), the Kendall’s rank correlation test (dotted line marked with plus signs), the martingale difference correlation test (dotdash line marked with cross signs), the distance correlation test (longdash line marked with diamond signs) and the cumulative divergence test (solid line marked with star signs), respectively. In Figure 1 (A), both X and ε are standard normal. In Figure 1 (B), X follows Cauchy distribution and ε is standard normal.

It can be clearly seen from Figure 1 that the sizes of all tests are close to the significance level α = 0.05. Both the Pearson correlation and the rank correlation test fail to detect the non-monotone mean dependence. The CD test is much more powerful than both the martingale difference correlation test and the distance correlation test when X follows Cauchy distribution. This simulated example empirically confirms that the robustness property of the CD test.

3. A FORWARD SCREENING PROCEDURE

In this section we propose a model-free forward screening procedure based on the CD. This new forward screening procedure inherits the appealing properties of the CD.

To ease subsequent presentation, we introduce the following notations. Let Y be the response and x = (X_1, …, X_p)^T be the p-dimensional covariate vector. Let $\mathcal{F}$ be a working index set and ${\mathcal{F}}^c$ be its complement. Both $\mathcal{F}$ and ${\mathcal{F}}^c$ are subset of {1,2, …, p}. We define ${\mathbf{x}}_{\mathcal{F}}\stackrel{\text{def}}{=}\{X_k,k\in \mathcal{F}\}$ the covariate vector indexed by $\mathcal{F}$ and ${\Sigma }_{\mathcal{F}}\stackrel{\text{def}}{=}\mathrm{var}({\mathbf{x}}_{\mathcal{F}})$. Let $|\mathcal{F}|$ stand for the cardinality of $\mathcal{F}$. We assume thoughtout that E(x) = 0 for simplicity.

The goal of feature selection is to identify the smallest index set $\mathcal{A}$ such that

$$Y⫫\hspace{0.17em}\mathbf{x}\hspace{0.17em}|\hspace{0.17em}\hspace{0.17em}{\mathbf{x}}_{\mathcal{A}},$$ (3.1)

where ⫫ stands for statistical independence. Model (3.1) implies immediately that $F(y|\mathbf{x})=F(y|{\mathbf{x}}_{\mathcal{A}})$, for $y\in ℝ$. Therefore, identifying ${\mathbf{x}}_{\mathcal{A}}$ which satisfies model (3.1) is equivalent to seeking for the smallest index set

$$\mathcal{A}\stackrel{\text{def}}{=}\{k:F(y|\mathbf{x})\}\hspace{0.17em}\hspace{0.17em}\text{depends}\hspace{0.17em}\text{functionally}\hspace{0.17em}\text{on}\hspace{0.17em}{X}_k\hspace{0.17em}\text{for}\hspace{0.17em}y\in ℝ,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}k=1,\hspace{0.17em}\dots \hspace{0.17em},p\}.$$

Model (3.1) covers a wide variety of existing models. Interested readers can refer to Section 2.1 of Zhu et al., (2011) for details.

We first note that model (3.1) ensures that $Y⫫\hspace{0.17em}{X}_k|\hspace{0.17em}\hspace{0.17em}{\mathbf{x}}_{\mathcal{F}}$ for all $k\in {\mathcal{F}}^c$ and all $\mathcal{A}\subseteq \mathcal{F}$. Therefore, given a working index set $\mathcal{F}$, assessing whether X_k, $k\in {\mathcal{F}}^c$, is truly important for the response variable Y amounts to testing the hypothesis that

$$H_0:Y⫫\hspace{0.17em}{X}_k|\hspace{0.17em}\hspace{0.17em}{\mathbf{x}}_{\mathcal{F}}\hspace{0.17em}\hspace{0.17em}\text{versus}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{H}_1:\text{otherwise}.$$ (3.2)

The law of iterated expectations implies immediately that $E\{X_k-E(X_k|{\mathbf{x}}_{\mathcal{F}})|Y\}=E\{E(X_k|{\mathbf{x}}_{\mathcal{F}},Y)-E(X_k|{\mathbf{x}}_{\mathcal{F}})|Y\}$. Under H₀ in (3.2), $E(X_k|{\mathbf{x}}_{\mathcal{F}},Y)=E(X_k|{\mathbf{x}}_{\mathcal{F}})$, and hence $E\{X_k-E(X_k|{\mathbf{x}}_{\mathcal{F}})|Y\}=0$. Under H₁ in (3.2), X_k is dependent upon Y even when ${\mathbf{x}}_{\mathcal{F}}$ is given. Thus it is reasonable to expect that $E(X_k|{\mathbf{x}}_{\mathcal{F}},Y)\ne E(X_k|{\mathbf{x}}_{\mathcal{F}})$ and accordingly $E\{X_k-E(X_k|{\mathbf{x}}_{\mathcal{F}})|Y\}\ne 0$. These, together with Theorem 1, motivate us to use ${\omega }_{k|\mathcal{F}}\stackrel{\text{def}}{=}\text{CD}\{X_k-E(X_k|{\mathbf{x}}_{\mathcal{F}})|Y\}$ to test (3.2).

To ensure that ${\omega }_{k|\mathcal{F}}$ has nontrivial power in test for (3.2), we further assume that

$$\text{A}1.\hspace{0.17em}\hspace{0.17em}E\{\partial F(y|\mathbf{x})/\partial X_k\}\ne 0\hspace{0.17em}\text{for}\hspace{0.17em}\text{some}\hspace{0.17em}y\in ℝ,\hspace{0.17em}\text{for}\hspace{0.17em}\text{all}\hspace{0.17em}k\in \mathcal{A}.$$

It is remarkable that (3.1) ensures that E{∂F(y | x)/∂X_k} = 0 for all $y\in ℝ$, and all $k\in {\mathcal{A}}^c$. This fact, together with Assumption A1, ensures that the important and the unimportant covariates are separable, which is stated in Theorem 2.

Theorem 2. Under H₀ in (3.2), we have ${\omega }_{k|\mathcal{F}}=0$. If we further assume that x is normal and Assumption A1 holds, then $\underset{\mathcal{F}:{\mathcal{F}}^c\cap \mathcal{A}\ne \varnothing }{\min }\underset{k\in {\mathcal{F}}^c\cap \mathcal{A}}{\max }{\omega }_{k|\mathcal{F}}>0$.

Theorem 2 guarantees that, if all the truly important covariates have been selected into $\mathcal{F}$ already, then for any $k\in {\mathcal{F}}^c$, we have ${\omega }_{k|\mathcal{F}}=0$. However, if there are a few important covariates that have not been found yet, that is, ${\mathcal{F}}^c\cap \mathcal{A}\ne \varnothing$, then there must exist $k\in {\mathcal{F}}^c\cap \mathcal{A}$ such that $\underset{k\in {\mathcal{F}}^c\cap \mathcal{A}}{\max }{\omega }_{k|\mathcal{F}}>0$. This motivates us to reject H₀ in (3.2) when the sample version of $\underset{k\in {\mathcal{F}}^c\cap \mathcal{A}}{\max }{\omega }_{k|\mathcal{F}}$ is sufficiently large.

How to estimate ${\omega }_{k|\mathcal{F}}$ is a nontrivial task because it involves estimating $E(X_k|{\mathbf{x}}_{\mathcal{F}})$. A fully nonparametric estimate of $E(X_k|{\mathbf{x}}_{\mathcal{F}})$ is apparently undesirable, especially when ${\mathbf{x}}_{\mathcal{F}}$ is high dimensional. In the present context, we assume that

$$\text{A2}.\hspace{0.17em}\hspace{0.17em}E(X_k|{\mathbf{x}}_{\mathcal{F}})=g_{k|\mathcal{F}}({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{k|\mathcal{F}}),\hspace{0.17em}\text{where}\hspace{0.17em}{g}_{k|\mathcal{F}}\hspace{0.17em}\text{is}\hspace{0.17em}\text{known}\hspace{0.17em}\text{and}\hspace{0.17em}{\mathrm{\beta }}_{k|\mathcal{F}}\hspace{0.17em}\text{is}\hspace{0.17em}\text{unknown}.$$

We allow $E(X_k|{\mathbf{x}}_{\mathcal{F}})$ to be a general parametric function. When x follows elliptically contoured distribution, $E(X_k|{\mathbf{x}}_{\mathcal{F}})$ is indeed a linear function of ${\mathbf{x}}_{\mathcal{F}}$, for all k and $\mathcal{F}\subseteq \{1,\hspace{0.17em}\dots \hspace{0.17em},p\}$. Examples of elliptically contoured distribution include multivariate normal distribution, multivariate t-distribution, symmetric multivariate Laplace distribution and multivariate logistic distribution, etc.

Let {(x_i, Y_i), i = 1, …, n} be a random sample from (x, Y), where each covariate, for notational clarity, is assumed to be marginally standardized to have zero mean and unit variance in advance. To carry out the CD test for (3.2), we estimate ${\omega }_{k|\mathcal{F}}$ by

$${\widehat{\omega }}_{k|\mathcal{F}}=n^{-2}\sum _{j=1}^n{[\sum _{i=1}^n\mathbf{1}(Y_i<Y_j)\{X_{ik}-g_{k|\mathcal{F}}({\mathbf{x}}_{i\mathcal{F}},{\widehat{\beta }}_{k|\mathcal{F}})\}]}^2/\sum _{i=1}^n{\{X_{ik}-g_{k|\mathcal{F}}({\mathbf{x}}_{i\mathcal{F}},{\widehat{\beta }}_{k|\mathcal{F}})\}}^2,$$

where ${\widehat{\beta }}_{k|\mathcal{F}}$ is obtained through the nonlinear least squares. That is,

$${\widehat{\beta }}_{k|\mathcal{F}}\stackrel{\text{def}}{=}\underset{{\mathrm{\beta }}_{k|\mathcal{F}}}{\arg \min }\sum _{i=1}^n{\{X_{ik}-g_{k|\mathcal{F}}({\mathbf{x}}_{i\mathcal{F}},{\mathrm{\beta }}_{k|\mathcal{F}})\}}^2$$ (3.3)

We reject H₀ in (3.2) when ${\widehat{\omega }}_{k|\mathcal{F}}$ is sufficiently large. Deciding the critical value for the CD test amounts to studying the asymptotic distribution of ${\widehat{\omega }}_{k|\mathcal{F}}$. Let $g_{k|\mathcal{F}}^{\prime }({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{k|\mathcal{F}})$, $g_{k|\mathcal{F}}^{\prime \prime }({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{k|\mathcal{F}})$ and $g_{k|\mathcal{F}}^{\prime \prime \prime }({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{k|\mathcal{F}})$ be the first, the second and the third derivatives of $g_{k|\mathcal{F}}({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{k|\mathcal{F}})$ with respect to ${\mathrm{\beta }}_{k|\mathcal{F}}$, respectively. We denote $g_{l_1,k|\mathcal{F}}^{\prime }({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{k|\mathcal{F}})$ the l₁-th component of $g_{k|\mathcal{F}}^{\prime }({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{k|\mathcal{F}})$, $g_{l_1{l}_2,k|\mathcal{F}}^{\prime \prime }({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{k|\mathcal{F}})$ the (l₁, l₂)-th component of $g_{k|\mathcal{F}}^{\prime \prime }({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{k|\mathcal{F}})$ and $g_{l_1{l}_2{l}_3,k|\mathcal{F}}^{\prime \prime \prime }({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{k|\mathcal{F}})$ the (l₁, l₂, l₃)-th component $g_{k|\mathcal{F}}^{\prime \prime \prime }({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{k|\mathcal{F}})$. Let ${\delta }_{k|\mathcal{F}}\stackrel{\text{def}}{=}{X}_k-E(X_k|{\mathbf{x}}_{\mathcal{F}})$ and C be a generic constant. We assume the following conditions.

• (B1)There exists ϑ > 0 such that p = o{exp(an^ϑ)} for any a > 0.
• (B2)For any working index set $\mathcal{F}\subseteq \{1,2,\hspace{0.17em}\dots \hspace{0.17em},p\}$ and $k\in {\mathcal{F}}^c$, $E(X_k^4)\le C$, $E({\delta }_{k|\mathcal{F}}^8)\le C$, $E\{{|g_{l_1,k|\mathcal{F}}^{\prime }({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{k|\mathcal{F}})|}^2\}\le C;|g_{k|\mathcal{F}}({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{k|\mathcal{F}})|\le G_{k|\mathcal{F}}({\mathbf{x}}_{\mathcal{F}})$ with $E[{\{G_{k|\mathcal{F}}|({\mathbf{x}}_{\mathcal{F}})\}}^4]\le C;\hspace{0.17em}|g_{l_1,k|\mathcal{F}}^{\prime }({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{k|\mathcal{F}})|\le G_{l_1,k|\mathcal{F}}({\mathbf{x}}_{\mathcal{F}})$ with $E[{\{G_{l_1,k|\mathcal{F}}|({\mathbf{x}}_{\mathcal{F}})\}}^4]\le C;\hspace{0.17em}|g_{l_1{l}_2,k|\mathcal{F}}^{\prime \prime }({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{k|\mathcal{F}})|\le G_{l_1{l}_2,k|\mathcal{F}}({\mathbf{x}}_{\mathcal{F}})$ with $E[{\{G_{l_1{l}_2,k|\mathcal{F}}|({\mathbf{x}}_{\mathcal{F}})\}}^4]\le C;\hspace{0.17em}|g_{l_1{l}_2{l}_3,k|\mathcal{F}}^{\prime \prime \prime }({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{k|\mathcal{F}})|\le G_{l_1{l}_2{l}_3,k|\mathcal{F}}({\mathbf{x}}_{\mathcal{F}})$ with $E[{\{G_{l_1{l}_2{l}_3,k|\mathcal{F}}|({\mathbf{x}}_{\mathcal{F}})\}}^4]\le C$, for all l₁, l₂, l₃ and ${\mathrm{\beta }}_{k|\mathcal{F}}$.
• (B3)There exists c₀ such that ${\Vert {\sum }_{\mathcal{F}}^{-1}\Vert }_{\infty }<c_0$ for all p, where ${\Vert A\Vert }_{\infty }\stackrel{\text{def}}{=}\underset{l}{\max }\sum _m|a_{lm}|$ stands for the infinity norm of the matrix A = (a_lm).

Condition (B1) allows p to diverge exponentially faster than n. Condition (B2) is widely used to study the asymptotic behavior of nonlinear least squares estimation. See, e.g., Jennrich (1969) and White (1981). This condition can be simplified dramatically when $g_{k|\mathcal{F}}({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{k|\mathcal{F}})$ is linear. Theorem 3 requires condition (B3) holds true for $|\mathcal{F}|=o(n^{1/5})$. Many precision matrices satisfy Condition (B3). In particular, if we denote ${\sum }_{\mathcal{F}}^{-1}={({\sigma }_{-1,lm})}_{|\mathcal{F}|\times |\mathcal{F}|}$ and let σ_−1,lm equal 1 if l = m and r_n otherwise, then this condition is satisfied as long as $|r_n|\le (c_0-1)/(|\mathcal{F}|-1)$. If ${\sum }_{\mathcal{F}}^{-1}$ is a banded or block-diagonal matrix and each row has d nonzero entries, for example, σ_−1,lm equals 1 if l = m, r if 1 ≤ |l−m| < d and 0 if |l−m| ≥ d, condition (B3) simply requires |r| ≤ (c₀ −1)/(d−1). Condition (B3) can also be satisfied by many other sparse precision matrices. If ${\sum }_{\mathcal{F}}^{-1}$ is a power-decay matrix, say, ${\sigma }_{-1,lm}={\rho }_n^{|l-m|}$, for |ρ_n| < 1, condition (B3) is satisfied as long as $(1-{\rho }_n^{|\mathcal{F}|})\le c_0(1-{\rho }_n)$. Condition (B3) is also implied by ${\Vert {\Sigma }^{-1}\Vert }_{\infty }<c_0$. Similar conditions are also assumed in the literature. See, for example, Mai et al. (2012, page 34–35) and Bickel and Levina (2008, page 2580).

Theorem 3. In addition to Conditions (B1)-(B3), we further assume $|\mathcal{F}|=o(n^{1/5})$.

1. Under H₀ in (3.2), we have, ${\omega }_{k|\mathcal{F}}=0$ and $pr(n{\widehat{\omega }}_{k|\mathcal{F}}<q\hspace{0.17em}{K}_{k|\mathcal{F}})-pr(Q_{k|\mathcal{F}\hspace{0.17em}}<q)\to 0$, for any $q\in {ℝ}^{+}$, where $Q_{k|\mathcal{F}}\stackrel{\text{def}}{=}\sum _{j=1}^{\infty }{\lambda }_{j,k|\mathcal{F}}{\chi }_j^2(1)$, $K_{k|\mathcal{F}}$ is defined in (B.1), ${\chi }_j^2(1)s$ are independent χ²(1) random variables, ${\lambda }_{j,k|\mathcal{F}}s$ are nonnegative constants that depend on the joint distribution of $(X_k,{\mathbf{x}}_{\mathcal{F}},Y)$ and $E(Q_{k|\mathcal{F}})=1$.

2. Under H₁ in (3.2) and if ${\omega }_{k|\mathcal{F}}>0$ for $k\in {\mathcal{F}}^c$, we have $pr\{n^{1/2}({\widehat{\omega }}_{k|\mathcal{F}}-{\omega }_{k|\mathcal{F}})<t\}-pr(T_{k|\mathcal{F}}<t)\to 0$, for any $t\in ℝ$, where $T_{k|\mathcal{F}}$ is a normal random variable with mean zero and variance ${\Delta }_{k|\mathcal{F}}$ and ${\Delta }_{k|\mathcal{F}}$ is defined in (B.3).

The condition $|\mathcal{F}|=o(n^{1/5})$ seems somewhat stringent. By refining Assumption A2, such as $E(X_k|{\mathbf{x}}_{\mathcal{F}})=g_{k|\mathcal{F}}({\mathbf{x}}_{\mathcal{F}}^T{\mathrm{\beta }}_{k|\mathcal{F}})$, this condition can be weakened to $|\mathcal{F}|=o(n^{1/3})$. This condition is in line with that of Huber (1973), Fan and Peng (2004) and Tan and Zhu (2018). We impose this condition because ${\mathrm{\beta }}_{k|\mathcal{F}}$ is unknown and has to be estimated from data. We do not impose sparsity assumption on ${\mathrm{\beta }}_{k|\mathcal{F}}$ but we do require the convergence rate of ${\widehat{\beta }}_{k|\mathcal{F}}$ be fast enough to ensure the weak convergence of ${\widehat{\omega }}_{k|\mathcal{F}}$. The requirement on ${\widehat{\beta }}_{k|\mathcal{F}}$ can be met under the condition that $|\mathcal{F}|=o(n^{1/5})$.

Theorem 3 shows that ${\widehat{\omega }}_{k|\mathcal{F}}$ is root-n consistent under H₀ and n-consistent under H₁, indicating that the CD test has nontrivial power in test for (3.2). We adopt the wild bootstrap procedure introduced in Section 2 to determine critical values.

Next we adapt our proposed CD test for (3.2) with a working index set $\mathcal{F}$ to a forward screening procedure for ultrahigh dimensional feature selection in model (3.1). The rationale of our proposed forward screening procedure is as follows. If H₀ in (3.2) is rejected, we update $\mathcal{F}$ with $\mathcal{F}\cup \{k\}$, because X_k is possibly influential for Y. With the updated $\mathcal{F}$, we further consider testing (3.2) until H₀ is accepted for all $k\in {\mathcal{F}}^c$. It is reasonable to expect $\mathcal{A}\subseteq \mathcal{F}$ when the forward screening procedure stops. To provide theoretical justification for our proposal, we assume the following condition.

A3. There exist a positive constant C and ϖ ∈ [0, 1/2) such that

$$\underset{\mathcal{F}:{\mathcal{F}}^c\cap \mathcal{A}\ne \varnothing }{\min }\underset{k\in {\mathcal{F}}^c\cap \mathcal{A}}{\max }{\omega }_{k|\mathcal{F}}>C{n}^{-\varpi }.$$ (3.4)

Assumption A3 requires that the signal strength of the truly important covariates, conditional on the covariates ${\mathbf{x}}_{\mathcal{F}}$ that have already been selected, is strong enough to be detectable. It is also justified in Theorem 2. This assumption is different from the marginal signal assumptions used in the screening literature in that the marginal signal strength is quantified through setting the working index set $\mathcal{F}$ to be a null set. See, for example, condition 3 in Fan and Lv (2008), condition E in Fan and Song (2010), condition C in Fan et al. (2011), condition (C1) in Zhu et al., (2011), and condition (C2) in Li, Zhong and Zhu (2012). It is generally required that the marginal signal strength of all truly important covariates must be greater than a certain threshold in the existing screening literature. By contrast, Assumption A3 quantifies the signal strength of the truly important covariates conditional on the selected covariates ${\mathbf{x}}_{\mathcal{F}}$, which ensures that ${\omega }_{k|\mathcal{F}}$ plays a similar role as the regression coefficients in linear models. Similar assumptions are also made in the literature. See, for example, condition (C3) in Wang (2009, page 1513) and condition 1 in Barut, Fan and Verhasselt (2016, page 1270). These assumptions are generally regarded as mild and reasonable.

To establish the sure screening property for the proposed screening procedure, we further assume the following conditions.

• (B4)The cardinality of $\mathcal{A}$ satisfies $|\mathcal{A}|=O(n^{1/5-\mathrm{\gamma }})$ for γ ∈ (0, 1/5].
• (B5)Let M and υ be two generic positive constants. Assume that E|X_k|m ≤ m!M^m−2υ/2 for all m ≥ 2, k = 1, 2, …, p. Assume in addition that $E{|g_{k|\mathcal{F}}({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{0,k|\mathcal{F}})|}^m\le m!M^{m-2}\upsilon /2$ and $E{|g_{l_1,k|\mathcal{F}}^{\prime }({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{0,k|\mathcal{F}})|}^m\le m!M^{m-2}\upsilon /2$ for all m ≥ 2, $\mathcal{F}\subseteq \{1,2,\hspace{0.17em}\dots \hspace{0.17em},p\}$ and $k\in {\mathcal{F}}^c$.

Assumption (B4) is also a technical condition and is closely related to the assumption that $|\mathcal{F}|=o(n^{1/5})$ used in Theorem 3. Condition (B5) is milder than the sub-Gaussian assumption (Buldygin and Kozachenko, 1980, Lemma 1).

Theorem 4. Suppose that Conditions (B1)-(B5) and Assumption A3 are satisfied. If we further assume $|\mathcal{F}|=o(n^{1/5})$, 3/5 − 2ϖ − ϑ > 0 and set $\nu <C{n}^{-\varpi }/2$ in the forward screening procedure, then $pr({\min }_{\mathcal{F}:{\mathcal{F}}^c\cap \mathcal{A}\ne \varnothing }{\max }_{k\in {\mathcal{F}}^c\cap \mathcal{A}}{\widehat{\omega }}_{k|\mathcal{F}}>\nu )\to 1$ as n → ∞.

Theorem 4 ensures that the proposed procedure can retain all important covariates with an overwhelming probability if ν is chosen properly. Such a desirable property is referred to as the sure screening property. The CD is a robust correlation metric, and our forward screening procedure is also robust to model misspecification. Such merits are particularly appealing for analyzing ultrahigh dimensional data in the absence of prior knowledge of model structure and data quality. Unlike existing model-free marginal screening methods, the proposed method is a stepwise procedure, which incorporates joint correlation among ultrahigh dimensional features in the forward screening process. It thus provides more reliable results in practice. With a data-driven choice of ν, the procedure adaptively determines the number of features to be retained after selection. This makes the implementation of our proposed forward screening method practically convenient, since our proposal does not require additional ad hoc tuning steps.

We describe the algorithm for our proposed forward screening procedure as follows.

Step 1 Start with an initial index set $\mathcal{F}=\varnothing$.

Step 2 For all $k\in {\mathcal{F}}^c$, calculate ${\widehat{\omega }}_{k|\mathcal{F}}$. Denote $k^{\ast }=\underset{k\in {\mathcal{F}}^c}{\arg \min }{\widehat{\omega }}_{k|\mathcal{F}}$. If ${\widehat{\omega }}_{k^{\ast }|\mathcal{F}}>\nu$, update $\mathcal{F}$ with $\mathcal{F}\cup \{k^{\ast }\}$. The data-driven ν will be determined as follows.

1. Generate ${\tilde{X}}_{i{k}^{\ast }}=g_{k^{\ast }|\mathcal{F}}({\mathbf{x}}_{i\mathcal{F}},{\widehat{\beta }}_{k^{\ast }|\mathcal{F}})+a_i{\widehat{\delta }}_{i,k^{\ast }|\mathcal{F}}$, i = 1, 2, …, n, where ${\widehat{\delta }}_{i,k^{\ast }|\mathcal{F}}=X_{i{k}^{\ast }}-g_{k^{\ast }|\mathcal{F}}({\mathbf{x}}_{i\mathcal{F}},{\widehat{\beta }}_{k^{\ast }|\mathcal{F}})$, and a_i are independent and identically distributed random weights satisfying pr(a_i = 1) = pr(a_i = −1) = 1/2. We calculate ${\widehat{\tilde{\omega }}}_{k^{\ast }|\mathcal{F}}\stackrel{\text{def}}{=}\widehat{\text{CD}}\{{\tilde{X}}_{k^{\ast }}-E({\tilde{X}}_{k^{\ast }}|{\mathbf{x}}_{\mathcal{F}})|Y\}$ using $\{({\tilde{X}}_{i{k}^{\ast }},{\mathbf{x}}_{i\mathcal{F}},Y_i),\hspace{0.17em}i=1,2,\hspace{0.17em}\dots \hspace{0.17em},n\}$.

2. Repeat the above wild boostrap procedure for B times to obtain ${\widehat{\tilde{\omega }}}_{k^{\ast }|\mathcal{F}}^{(1)}$, ${\widehat{\tilde{\omega }}}_{k^{\ast }|\mathcal{F}}^{(2)}$, …, ${\widehat{\tilde{\omega }}}_{k^{\ast }|\mathcal{F}}^{(B)}$. Set ν to be the (1–α)-th upper quantile of $\{{\widehat{\tilde{\omega }}}_{k^{\ast }|\mathcal{F}}^{(1)},{\widehat{\tilde{\omega }}}_{k^{\ast }|\mathcal{F}}^{(2)},\hspace{0.17em}\dots \hspace{0.17em},{\widehat{\tilde{\omega }}}_{k^{\ast }|\mathcal{F}}^{(B)}\}$. We update the working index set $\mathcal{F}$ with $\mathcal{F}\cup \{k^{\ast }\}$ if ${\widehat{\omega }}_{k^{\ast }|\mathcal{F}}>\nu$.

Step 3 Repeat Step 2 until no covariate can be added into the working index set $\mathcal{F}$.

Assumption A2 requires that the minimal signal strength be greater than Cn^−ϖ, and Theorem 4 requires the cutoff ν to be smaller than one half of the minimal signal strength. These requirements ensure that our proposal possesses the desirable sure screening property. In practice, however, the magnitude of minimal signal strength is generally unknown. Consequently, how to choose an optimal cutoff ν is not straightforward. To put our proposed procedure into practice, at each step and for each covariate, we choose α = 0.01 and set the cutoff to be the 99-th percentile of asymptotic null distribution of ${\widehat{\omega }}_{k^{\ast }|\mathcal{F}}$ in our algorithm. This works satisfactorily in our numerical studies.

4. NUMERICAL STUDIES

4.1. Simulations

In this section, we conduct Monte Carlo simulations to assess the finite sample performance of the CD-based forward screening procedure. For convenience of presentation, we refer to our proposed forward screening method as C-FS. We compare C-FS with the following five competitors: the forward regression designed for linear model by Wang (2009, FR), the least absolute shrinkage and selection operator proposed by Tibshirani (1996, LASSO), the sure independent ranking and screening procedure proposed by Zhu et al., (2011, SIRS), the distance correlation based sure independence screening procedure proposed by Li, Zhong and Zhu (2012, DC-SIS), and the Pearson correlation based sure independence screening procedure proposed by Fan and Lv (2008, SIS).

To determine the number of features to be retained after screening, we use a BIC-type criterion for FR, as suggested by Wang (2009). The model size (tuning parameter) of the LASSO was chosen by 10-fold cross validation. For SIRS, DC-SIS and SIS, we follow the convention by retaining [n/log(n)] top ranked covariates into the screened model. It should be noted that our C-FS algorithm automatically determines the screening size with a wild bootstrap procedure.

We adopt the following criteria to evaluate the performance of above methods.

1. ${\mathcal{P}}_{\text{ind}}$: With a given size, ${\mathcal{P}}_{\text{ind}}$ is the empirical probability that an influential covariate is retained after screening.

2. ${\mathcal{P}}_{\text{all}}$: With a given size, ${\mathcal{P}}_{\text{all}}$ is the empirical probability that all the influential covariates are retained after screening.

3. FPR: Let $\widehat{\mathcal{A}}$ be the index set of the retained covariates and $\mathcal{A}$ be the index set of truly influential covariates. The false positive rate (FPR) is defined as $|\widehat{\mathcal{A}}\backslash \mathcal{A}|/|{\mathcal{A}}^c|$, where $\widehat{\mathcal{A}}\backslash \mathcal{A}$ is the index set of irrelevant covariates that are retained after screening and $|\mathcal{M}|$ denotes the cardinality of the set $\mathcal{M}$.

4. TPR: The true positive rate (TPR) is defined as $|\widehat{\mathcal{A}}\cap \mathcal{A}|/|\mathcal{A}|$, where $\widehat{\mathcal{A}}\cap \mathcal{A}$ denotes the set of influential covariates that are correctly retained after screening

We report both the mean and the standard errors of the FPR and TPR values based on 500 repetitions. We set the sample size n = 200, the covariate dimension p = 3, 000 and the bootstrap times B = 1, 000.

Example 1. We generate data from a linear model Y = βTx + c₀ε, where β = (5, 5, 5, −15ρ^½, 0, …, 0)^T, c₀ = 1 if ε ∼ N (0, 1) and c₀ = 0.1 if ε ∼ t(1). We consider the following two scenarios to generate the covariate x = (X₁, …, X_p)^T.

1. The elliptical case: The covariate x is drawn from multivariate normal population with mean zero and covariance matrix Σ = (σ_ij)_p×p, where σ_ii = 1, i = 1, …, p, σ_i4 = σ_4i = ρ^½ for i ≠ 4, and σ_ij = ρ, for i ≠ j, i ≠ 4 and j ≠ 4.

2. The non-elliptical case: Set $\mathbf{x}={\mathrm{\Sigma }}^{1/2}{\{\widehat{\mathrm{var}}(\mathbf{z})\}}^{-1/2}\{\mathbf{z}-\widehat{E}(\mathbf{z})\}$, where Σ is defined in the first scenario, $\mathbf{z}\stackrel{\text{def}}{=}{(Z_1,\hspace{0.17em}\dots \hspace{0.17em},Z_p)}^{\text{T}}$, Z_ks are independent of each other and follow χ²(2) distribution.

In the above two scenarios, we set ρ to be 0.1, 0.5 and 0.9, respectively, to stand for small, moderate and high correlation. This example was also used by Fan and Lv (2008) and Zhu et al., (2011). The simulation results are summarized in Tables 1–2.

Table 1:

The mean and the standard errors of both the FPR and the TPR values based on 500 repetitions for Example 1.

ε	method	ρ = 0.1		ρ = 0.5		ρ = 0.9
		FPR	TPR	FPR	TPR	FPR	TPR
		mean std	mean std	mean std	mean std	mean std	mean std
When x follows elliptical distribution
$\mathcal{N}(0,1)$	C-FS	0.00 0.00	1.00 0.00	0.00 0.00	1.00 0.00	0.00 0.00	1.00 0.00
	FR	0.00 0.00	1.00 0.00	0.00 0.00	1.00 0.00	0.00 0.00	1.00 0.00
	LASSO	0.01 0.00	1.00 0.00	0.06 0.00	0.75 0.00	0.04 0.00	0.75 0.00
	SIRS	0.00 0.00	0.75 0.00	0.01 0.00	0.75 0.01	0.01 0.00	0.59 0.29
	DC-SIS	0.00 0.00	0.75 0.00	0.01 0.00	0.75 0.02	0.01 0.00	0.58 0.29
	SIS	0.00 0.00	0.75 0.00	0.00 0.00	0.75 0.00	0.01 0.00	0.60 0.28
t(1)	C-FS	0.00 0.00	1.00 0.00	0.00 0.00	1.00 0.00	0.00 0.00	1.00 0.00
	FR	0.00 0.00	0.90 0.29	0.00 0.00	0.90 0.29	0.00 0.00	0.77 0.42
	LASSO	0.00 0.00	0.84 0.35	0.03 0.02	0.63 0.27	0.02 0.02	0.52 0.34
	SIRS	0.01 0.00	0.75 0.01	0.01 0.00	0.75 0.02	0.01 0.00	0.62 0.27
	DC-SIS	0.01 0.00	0.75 0.04	0.01 0.00	0.74 0.06	0.01 0.00	0.59 0.28
	SIS	0.01 0.00	0.72 0.14	0.01 0.00	0.71 0.16	0.01 0.00	0.51 0.33
When x follows non-elliptical distribution
$\mathcal{N}(0,1)$	C-FS	0.00 0.00	1.00 0.00	0.00 0.00	1.00 0.00	0.00 0.00	1.00 0.00
	FR	0.00 0.00	1.00 0.00	0.00 0.00	1.00 0.00	0.00 0.00	1.00 0.00
	LASSO	0.01 0.01	1.00 0.00	0.06 0.00	0.75 0.00	0.04 0.00	0.75 0.00
	SIRS	0.01 0.00	0.75 0.02	0.01 0.00	0.74 0.05	0.01 0.00	0.59 0.28
	DC-SIS	0.01 0.00	0.75 0.02	0.01 0.00	0.74 0.05	0.01 0.00	0.59 0.27
	SIS	0.01 0.00	0.75 0.02	0.01 0.00	0.74 0.04	0.01 0.00	0.62 0.25
t(1)	C-FS	0.00 0.00	1.00 0.00	0.00 0.00	1.00 0.00	0.00 0.00	1.00 0.00
	FR	0.00 0.00	0.94 0.21	0.00 0.00	0.93 0.21	0.00 0.00	0.85 0.30
	LASSO	0.00 0.00	0.86 0.32	0.03 0.02	0.62 0.28	0.02 0.02	0.49 0.35
	SIRS	0.00 0.00	0.75 0.00	0.01 0.00	0.74 0.05	0.01 0.00	0.57 0.29
	DC-SIS	0.01 0.00	0.75 0.04	0.01 0.00	0.74 0.06	0.01 0.00	0.56 0.30
	SIS	0.01 0.00	0.71 0.15	0.01 0.00	0.69 0.18	0.01 0.00	0.50 0.32

Table 2:

The empirical probabilities ${\mathcal{P}}_{\text{ind}}$ and ${\mathcal{P}}_{\text{all}}$ based on 500 repetitions for Example 1.

ε	method	ρ = 0.1					ρ = 0.5					ρ = 0.9
		${\mathcal{P}}_{\text{ind}}$				${\mathcal{P}}_{\text{all}}$	${\mathcal{P}}_{\text{ind}}$				${\mathcal{P}}_{\text{all}}$	${\mathcal{P}}_{\text{ind}}$				${\mathcal{P}}_{\text{all}}$
		X1	X2	X3	X4	ALL	X1	X2	X3	X4	ALL	X1	X2	X3	X4	ALL
When x follows elliptical distribution
$\mathcal{N}(0,1)$	C-FS	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
	FR	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
	LASSO	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	0.00	0.00	1.00	1.00	1.00	0.00	0.00
	SIRS	1.00	1.00	1.00	0.00	0.00	1.00	1.00	1.00	0.00	0.00	0.79	0.79	0.79	0.00	0.00
	DC-SIS	1.00	1.00	1.00	0.00	0.00	1.00	1.00	1.00	0.00	0.00	0.76	0.77	0.78	0.00	0.00
	SIS	1.00	1.00	1.00	0.00	0.00	1.00	1.00	1.00	0.00	0.00	0.81	0.80	0.81	0.00	0.00
t(1)	C-FS	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
	FR	0.90	0.90	0.90	0.89	0.89	0.90	0.90	0.91	0.90	0.89	0.77	0.77	0.77	0.77	0.76
	LASSO	0.85	0.85	0.86	0.79	0.79	0.84	0.83	0.83	0.00	0.00	0.70	0.77	0.77	0.00	0.00
	SIRS	1.00	1.00	1.00	0.00	0.00	1.00	1.00	1.00	0.00	0.00	0.83	0.82	0.83	0.00	0.00
	DC-SIS	0.99	1.00	0.99	0.00	0.00	0.99	0.99	0.99	0.00	0.00	0.80	0.78	0.78	0.00	0.00
	SIS	0.95	0.96	0.96	0.00	0.00	0.95	0.95	0.94	0.00	0.00	0.69	0.68	0.68	0.00	0.00
When x follows non-elliptical distribution
$\mathcal{N}(0,1)$	C-FS	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
	FR	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
	LASSO	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
	SIRS	1.00	1.00	1.00	0.01	0.01	0.99	0.99	0.99	0.00	0.00	0.78	0.80	0.80	0.00	0.00
	DC-SIS	1.00	1.00	1.00	0.01	0.01	0.99	0.99	0.99	0.00	0.00	0.77	0.80	0.80	0.00	0.00
	SIS	1.00	1.00	1.00	0.01	0.99	1.00	0.99	0.00	0.00	0.00	0.83	0.83	0.84	0.00	0.00
t(1)	C-FS	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
	FR	0.94	0.94	0.95	0.93	0.91	0.93	0.93	0.93	0.95	0.90	0.83	0.82	0.83	0.91	0.78
	LASSO	0.88	0.87	0.88	0.80	0.80	0.83	0.83	0.82	0.00	0.00	0.65	0.66	0.65	0.00	0.00
	SIRS	1.00	1.00	1.00	0.00	0.00	0.99	0.99	0.99	0.00	0.00	0.76	0.76	0.76	0.00	0.00
	DC-SIS	1.00	1.00	1.00	0.00	0.00	0.99	0.99	0.99	0.00	0.00	0.75	0.74	0.74	0.00	0.00
	SIS	0.94	0.94	0.94	0.96	0.00	0.92	0.93	0.92	0.00	0.00	0.65	0.66	0.69	0.00	0.00

In this example, X₄ is marginally independent of Y. It is thus not surprising to observe from Table 2 that SIRS, DC-SIS and SIS fail to retain X₄, as they consider only the marginal effects. The performance of LASSO is decent for ρ = 0.1 and $\epsilon ~\mathcal{N}(0,\hspace{0.17em}1)$, but it deteriorates sharply as ρ increases. Both FR and C-FS perform well when ε is normal. However, when ε ∼ t(1) and x is elliptical, FR has an average TPR as low as 0.77 for ρ = 0.9. In this scenario, FR is also quite unstable in terms of the large standard deviations. By contrast, the proposed C-FS attains stable and satisfactory performance in all scenarios. The simulation results when x follows non-elliptical distribution are quite similar to those when x follows elliptical distribution.

Example 2. We consider three models where Y depends on ${\mathbf{x}}_{\mathcal{A}}$ nonlinearly.

1. Y = X₁ + 0.8X₂ + 0.6X₃ + 0.4X₄ + 0.2 exp(X₂₀ + c₀ε).

2. Y = X₁ + 0.8X₂ + 0.6(X₅ + 1)² + $0.4X_{10}^3$ + 0.2 exp(|X₂₀ + 1|+ c₀ε).

3. Y = β₁(X₁)X₂ + β₂(X₁)X₃ + β₃(X₁)X₄ + β₄(X₁)X₅ + c₀ε.

In all three models, ε and c₀ are generated in the same way as in Example 1. In Example 2(a) and 2(b), we consider two scenarios for generating x.

1. The elliptical case: The covariate x is drawn from multivariate normal population with mean zero and covariance matrix $\mathrm{\Sigma }={({0.5}^{|i-j|})}_{p\times p}$.

2. The non-elliptical case: Set $\mathbf{x}={\mathrm{\Sigma }}^{1/2}{\{\widehat{\mathrm{var}}(\mathbf{z})\}}^{-1/2}\{\mathbf{z}-\widehat{E}(\mathbf{z})\}$, where $\mathrm{\Sigma }={({0.5}^{|i-j|})}_{p\times p}$, $\mathbf{z}\stackrel{\text{def}}{=}{(Z_1,\hspace{0.17em}\dots \hspace{0.17em},Z_p)}^{\text{T}}$, Z_ks are independent and follow χ²(2) distribution.

In Example 2(c), we generate U₁ and U₂ independently from uniform distribution on [0, 1], and set X₁ = (U₁ + U₂)/2, ${\beta }_1(X_1)=\exp 4(1-X_1^2)$, β₂(X₁) = 3{1 + sin(2πX₁)}, β₃(X₁) = 2{1 +(1 − X₁)³/2}, and β₄(X₁) = exp(|X₁|). Define X_k = (Z_k + 3U₁)/4, k = 2, 3, …, p, where Z_ks are independently drawn from (1) the standard normal distribution in the elliptical case and (2) the χ²(2) distribution in the non-elliptical case.

The simulation results for Example 2 are charted in Tables 3–4. In Example 2(a), none of SIRS, DC-SIS or SIS is able to identify X₂₀ as an important covariate. This is because these methods are relatively sensitive to the transformation of variables. When ε ∼ t(1), the new C-FS is the only method that has satisfactory performance. This confirms the robustness behavior of C-FS. In Example 2(b), those model based methods, such as FR, LASSO and SIS, fail to retain all the important covariates, because the linear model assumption is violated. In comparison, the model free methods (C-FS, SIRS, DC-SIS) perform relatively better. The performance of C-FS is the best, due to its robustness property against the outliers in the response. In Example 2(c), the marginal screening methods, such as SIRS, DC-SIS and SIS, fail to detect X₁ in all scenarios. Both C-FS and LASSO outperform FR when ε is normal, our C-FS is the only method that remains satisfactory performance when ε ∼ t(1).

Table 3:

The mean and the standard errors of both the FPR and the TPR values based on 500 repetitions for Example 2.

	method	x follows elliptical distribution				x follows non-elliptical distribution
		$\epsilon ~\mathcal{N}(0,1)$		ε ∼ t(1)		$\epsilon ~\mathcal{N}(0,1)$		ε ∼ t(1)
		FPR	TPR	FPR	TPR	FPR	TPR	FPR	TPR
		mean std	mean std	mean std	mean std	mean std	mean std	mean std	mean std
(a)	C-FS	0.01 0.00	0.98 0.06	0.01 0.00	0.98 0.07	0.01 0.00	0.97 0.07	0.01 0.00	0.98 0.07
	FR	0.00 0.00	0.88 0.18	0.00 0.00	0.25 0.40	0.00 0.00	0.63 0.28	0.00 0.00	0.21 0.34
	LASSO	0.00 0.00	0.83 0.28	0.00 0.00	0.22 0.38	0.00 0.00	0.45 0.44	0.00 0.00	0.14 0.32
	SIRS	0.01 0.00	0.90 0.10	0.01 0.00	0.86 0.09	0.01 0.00	0.98 0.07	0.01 0.00	0.97 0.08
	DC-SIS	0.01 0.00	0.91 0.10	0.01 0.00	0.44 0.42	0.01 0.00	0.98 0.08	0.01 0.00	0.48 0.46
	SIS	0.01 0.00	0.94 0.10	0.01 0.00	0.32 0.41	0.01 0.00	0.84 0.27	0.01 0.00	0.31 0.42
(b)	C-FS	0.00 0.00	0.98 0.06	0.00 0.00	1.00 0.03	0.00 0.00	0.99 0.04	0.00 0.00	1.00 0.02
	FR	0.00 0.00	0.60 0.28	0.00 0.00	0.19 0.35	0.00 0.00	0.44 0.23	0.00 0.00	0.14 0.24
	LASSO	0.00 0.00	0.54 0.42	0.00 0.00	0.17 0.35	0.00 0.00	0.15 0.29	0.00 0.00	0.04 0.17
	SIRS	0.01 0.00	0.96 0.08	0.01 0.00	0.96 0.08	0.01 0.00	0.99 0.04	0.01 0.00	1.00 0.03
	DC-SIS	0.01 0.00	0.98 0.06	0.01 0.00	0.41 0.46	0.01 0.00	0.96 0.14	0.01 0.00	0.39 0.46
	SIS	0.01 0.00	0.93 0.15	0.01 0.00	0.29 0.41	0.01 0.00	0.60 0.30	0.01 0.00	0.22 0.33
(c)	C-FS	0.01 0.00	0.95 0.09	0.01 0.00	0.99 0.05	0.01 0.00	0.94 0.11	0.01 0.00	0.98 0.06
	FR	0.00 0.00	0.89 0.12	0.00 0.00	0.48 0.40	0.00 0.00	0.88 0.14	0.00 0.00	0.49 0.39
	LASSO	0.00 0.00	0.96 0.09	0.00 0.00	0.43 0.42	0.00 0.00	0.93 0.12	0.00 0.00	0.43 0.42
	SIRS	0.01 0.00	0.73 0.10	0.01 0.00	0.76 0.09	0.01 0.00	0.70 0.13	0.01 0.00	0.73 0.10
	DC-SIS	0.01 0.00	0.72 0.11	0.01 0.00	0.74 0.12	0.01 0.00	0.69 0.12	0.01 0.00	0.70 0.13
	SIS	0.01 0.00	0.73 0.10	0.01 0.00	0.54 0.29	0.01 0.00	0.72 0.11	0.01 0.00	0.53 0.28

Table 4:

The empirical probabilities ${\mathcal{P}}_{\text{ind}}$ and ${\mathcal{P}}_{\text{all}}$ based on 500 repetitions for Example 2.

	method	$\epsilon ~\mathcal{N}(0,1)$						ε ∼ t(1)
		${\mathcal{P}}_{\text{ind}}$					${\mathcal{P}}_{all}$	${\mathcal{P}}_{\text{ind}}$					${\mathcal{P}}_{all}$
When x follows elliptical distribution
		X1	X2	X3	X4	X20	ALL	X1	X2	X3	X4	X20	ALL
(a)	C-FS	1.00	1.00	0.99	0.93	0.97	0.90	1.00	1.00	1.00	0.97	0.91	0.90
	FR	0.96	0.97	0.91	0.62	0.96	0.58	0.28	0.29	0.27	0.20	0.20	0.19
	LASSO	0.92	0.94	0.90	0.70	0.71	0.62	0.25	0.25	0.24	0.19	0.15	0.15
	SIRS	1.00	1.00	1.00	1.00	0.48	0.48	1.00	1.00	1.00	1.00	0.32	0.32
	DC-SIS	1.00	1.00	1.00	1.00	0.56	0.56	0.53	0.53	0.52	0.48	0.14	0.14
	SIS	1.00	1.00	1.00	0.99	0.71	0.71	0.37	0.37	0.37	0.33	0.14	0.14
(b)	C-FS	1.00	0.96	1.00	1.00	0.96	0.91	1.00	0.99	1.00	1.00	0.99	0.98
	FR	0.59	0.38	0.67	0.64	0.73	0.10	0.21	0.15	0.22	0.22	0.16	0.11
	LASSO	0.59	0.55	0.57	0.53	0.48	0.31	0.18	0.18	0.18	0.17	0.12	0.11
	SIRS	1.00	1.00	1.00	0.98	0.84	0.83	1.00	1.00	1.00	0.99	0.82	0.82
	DC-SIS	1.00	1.00	1.00	0.97	0.95	0.97	0.44	0.43	0.42	0.39	0.34	0.32
	SIS	0.96	0.95	0.93	0.88	0.95	0.76	0.31	0.31	0.29	0.29	0.23	0.19
		X1	X2	X3	X4	X5	ALL	X1	X2	X3	X4	X5	ALL
(c)	C-FS	1.00	1.00	1.00	0.96	0.78	0.75	1.00	1.00	1.00	1.00	0.94	0.94
	FR	0.99	1.00	1.00	0.93	0.56	0.51	0.45	0.59	0.64	0.45	0.28	0.24
	LASSO	0.89	1.00	1.00	0.99	0.90	0.80	0.24	0.53	0.54	0.46	0.36	0.22
	SIRS	0.00	0.99	0.99	0.93	0.73	0.00	0.00	1.00	1.00	0.97	0.81	0.00
	DC-SIS	0.00	0.99	1.00	0.92	0.69	0.00	0.00	0.99	0.99	0.94	0.76	0.00
	SIS	0.00	1.00	1.00	0.94	0.72	0.00	0.00	0.78	0.79	0.64	0.49	0.00
When x follows non-elliptical distribution
		X1	X2	X3	X4	X20	ALL	X1	X2	X3	X4	X20	ALL
(a)	C-FS	1.00	1.00	0.98	0.89	1.00	0.86	1.00	1.00	0.98	0.93	0.99	0.89
	FR	0.67	0.69	0.54	0.27	0.97	0.21	0.23	0.26	0.17	0.09	0.32	0.08
	LASSO	0.52	0.54	0.46	0.28	0.48	0.26	0.16	0.17	0.14	0.08	0.14	0.08
	SIRS	1.00	1.00	1.00	1.00	0.88	0.88	1.00	1.00	1.00	1.00	0.83	0.83
	DC-SIS	0.99	1.00	0.99	0.98	0.93	0.91	0.52	0.53	0.51	0.46	0.40	0.37
	SIS	0.84	0.87	0.81	0.71	0.98	0.67	0.32	0.32	0.31	0.25	0.35	0.22
(b)	C-FS	0.99	0.96	1.00	1.00	1.00	0.95	1.00	0.99	1.00	1.00	1.00	0.99
	FR	0.16	0.08	0.50	0.53	0.90	0.00	0.06	0.03	0.16	0.20	0.27	0.00
	LASSO	0.09	0.08	0.21	0.18	0.22	0.04	0.03	0.02	0.06	0.05	0.06	0.01
	SIRS	1.00	1.00	1.00	0.98	0.99	0.97	1.00	1.00	1.00	0.99	1.00	0.98
	DC-SIS	0.96	0.96	0.98	0.88	0.99	0.86	0.39	0.39	0.42	0.36	0.41	0.33
	SIS	0.37	0.37	0.63	0.66	0.96	0.24	0.15	0.14	0.24	0.26	0.32	0.08
		X1	X2	X3	X4	X5	ALL	X1	X2	X3	X4	X5	ALL
(c)	C-FS	1.00	0.99	0.97	0.95	0.78	0.73	1.00	1.00	1.00	0.98	0.91	0.89
	FR	0.96	1.00	0.98	0.92	0.55	0.49	0.48	0.62	0.63	0.45	0.29	0.24
	LASSO	0.82	1.00	0.99	0.99	0.87	0.73	0.26	0.53	0.54	0.45	0.35	0.22
	SIRS	0.00	0.99	0.95	0.89	0.66	0.00	0.00	1.00	0.99	0.95	0.70	0.00
	DC-SIS	0.00	0.99	0.99	0.88	0.59	0.00	0.00	0.99	0.99	0.92	0.60	0.00
	SIS	0.00	1.00	0.99	0.92	0.69	0.00	0.00	0.78	0.77	0.66	0.66	0.00

4.2. An Application

We further illustrate the performance of the proposed C-FS method through a rat eye expression dataset, which was previously studied by Scheetz et al. (2006) and Huang et al. (2008). This dataset consists of 31,042 probe sets of 120 twelve-week-old male rats, yet only 18,976 probes were sufficiently expressed. The response variable TRIM32 is among these 18,976 probes. This probe was found to cause Bardet-Biedl syndrome (Chiang et al., 2006). We rank the remaining 18,975 probes according to their variances and retain only 3,000 probes with the largest variances. Our analysis is based on the selected 3,000 probes, in addition to the probe TRIM32. The goal is to identify the probes that affect the expression level of TRIM32 considerably.

The sample size n = 120 is small compared with the covariate dimension p = 3, 000. We apply the aforementioned six feature selection/screening methods to this dataset and denote the retained covariates as ${\mathbf{x}}_{\widehat{\mathcal{A}}}$. We order the entries of $\widehat{\mathcal{A}}$ according to the relative importance of each retained covariate. Specifically, for SIS, DC-SIS, and SIRS, $\widehat{\mathcal{A}}$ is the index set of the covariates with s largest marginal effects; for C-FS, FR, and LASSO, $\widehat{\mathcal{A}}$ is the index set of the first s covariates that enter the active set.

We assess the performance of these methods as follows. Given a model size s, we fit an additive model

$$Y=\sum _{j=1}^s{f}_{kj}(X_{kj})+{\epsilon }_k,$$ (4.1)

where k = 1, …, 6 represents C-FS, FR, LASSO, SIRS, DC-SIS, and SIS respectively. The subscript kj denotes the jth element in ${\widehat{\mathcal{A}}}_k$ and s is set from 1 to 10. We stop our comparison at the 10-th step because the C-FS algorithm with critical values decided by bootstrap stops at this step. In other words, all remaining null hypotheses are accepted at the significance level 0.01 and there is no need to add additional covariates.

We estimate the unknown functions f_kj by the R package mgcv, where the adjusted R² and the explained deviance are summarized in Table 5. Since the deviance explained is defined as the proportion of the null deviance explained by the fitted model, the method with larger deviance has a better performance. From Table 5, we observe that, as s increases, the proposed C-FS tends to outperform all other marginal effect based methods. This is partly because these procedures may fail to identify some truly important covariates. In comparison, the performances of C-FS, FR and LASSO are relatively satisfactory as they consider the joint effects. Among these methods, the proposed C-FS has the highest R² and explained deviance.

Table 5:

The adjusted R² and the explained deviance of the six methods.

model size	adjusted R2						dev.explained
	C-FS	FR	LASSO	SIRS	DC-SIS	SIS	C-FS	FR	LASSO	SIRS	DC-SIS	SIS
1	0.33	0.62	0.62	0.33	0.33	0.62	0.35	0.62	0.62	0.35	0.35	0.62
2	0.63	0.68	0.68	0.68	0.68	0.68	0.66	0.69	0.69	0.71	0.69	0.69
3	0.66	0.70	0.68	0.69	0.69	0.69	0.71	0.71	0.70	0.71	0.71	0.71
4	0.72	0.72	0.70	0.70	0.70	0.70	0.74	0.73	0.72	0.73	0.71	0.72
5	0.72	0.74	0.70	0.74	0.70	0.70	0.75	0.75	0.73	0.77	0.72	0.72
6	0.79	0.75	0.73	0.76	0.71	0.73	0.83	0.77	0.77	0.80	0.73	0.77
7	0.81	0.76	0.74	0.76	0.72	0.74	0.84	0.78	0.77	0.80	0.74	0.77
8	0.81	0.77	0.77	0.75	0.72	0.73	0.84	0.79	0.80	0.79	0.74	0.77
9	0.82	0.77	0.77	0.75	0.71	0.74	0.86	0.79	0.81	0.80	0.74	0.78
10	0.84	0.78	0.77	0.74	0.73	0.74	0.88	0.80	0.81	0.78	0.76	0.78

We use the five-fold cross-validation to further compare the prediction performance. Specifically, we randomly partition the dataset into five equal sized subsamples, denoted ${\mathcal{D}}_1,\hspace{0.17em}\dots \hspace{0.17em},{\mathcal{D}}_5$. For each subsample ${\mathcal{D}}_k$, we use the remaining four subsamples to fit model (4.1) with s = 10, then calculate the mean squared prediction error on the subsample ${\mathcal{D}}_k$. We repeat this procedure such that the prediction is performed on each subsample exactly once. The mean squared prediction error of C-FS, FR, LASSO, SIRS, DC-SIS and SIS are 0.43, 0.74, 0.49, 0.78, 0.45 and 0.60, respectively. In this example, the C-FS gives the best prediction, followed by the DC-SIS and the LASSO.

5. CONCLUDING REMARKS

In this article, we proposed a CD-based forward screening procedure, which is model-free and robust to the presence of outliers in the response. By using a stepwise searching framework, the proposed procedure incorporates joint correlations among features in the screening process and thus provides more reliable results in applications.

In general, this forward screening procedure shares similar spirit to the iterative screening approaches (Zhu et al.,, 2011; Zhong and Zhu, 2015). However, how to decide model sizes for the iterative screening approaches and how to study their theoretical properties are rarely discussed in existing literature. Equipped with our proposed model-free forward screening procedure, a data-driven method is proposed to determine which covariates should be retained. We also show that our proposed forward screening procedure possesses the desirable sure screening property.

A model-free and robust method is often computationally intensive. Our experiences shows that the proposed C-FS procedure is more computationally demanding than other procedures. This is possibly caused by stepwise searching and adaptive thresholding, which make the proposed method more reliable and fully automatic. We conjecture that a simplified procedure may lead to less computational cost with a small sacrifice of numerical stability. In our simulation setup, each run takes no more than 4 minutes on average for n = 200 and p = 3000 on PC Intel Core2 Duo T9600 2.8GHz 4GB RAM server. This numerical cost is often acceptable in practice. It would be interesting to further develop a more efficient algorithm for the CD-based method.

APPENDIX: LEMMAS AND PROOFS OF THEOREMS

Appendix A: Some Lemmas

Lemma 1. (Bernstein’s Inequality, Van and Wellner (1996, Lemma 2.2.11)) Let X₁, X₂, …, X_n be independent random variables with mean 0 and E|X_i|^m ≤ m!M^m−2υ_i/2 for every m ≥ 2 and i = 1, 2, …, n, where M and υ_i are positive constants. Then

$$pr\{|X_1+X_2+\hspace{0.17em}\dots \hspace{0.17em}+X_n|>\epsilon \}\le 2\exp \left\{-\frac{{\epsilon }^2}{2(\upsilon +M\epsilon )}\right\},\hspace{0.17em}\hspace{0.17em}for\hspace{0.17em}\upsilon \ge \sum _{i=1}^n{\upsilon }_i.$$

For notational clarity, we denote $s=|\mathcal{F}|$ in what follows.

Lemma 2. If s = o(n^1/5) and conditions (B1)–(B5) hold true, then for any $k\in {\mathcal{F}}^c$, and ε_n = Cn^−κ, C and κ are positive constants,

$$pr\left\{{({\widehat{\beta }}_{k|\mathcal{F}}-{\mathrm{\beta }}_{0,k|\mathcal{F}})}^T({\widehat{\beta }}_{k|\mathcal{F}}-{\mathrm{\beta }}_{0,k|\mathcal{F}})>{\epsilon }_n\right\}<2s\exp (-c_{1}n{s}^{-1}{\epsilon }_n),$$

where c₁ is a positive constant and ${\mathrm{\beta }}_{0,k|\mathcal{F}}$ is defined by $\underset{{\mathrm{\beta }}_{k|\mathcal{F}}}{\arg \min }E{\{X_k-g_{k|\mathcal{F}}({\mathbf{x}}_{\mathcal{F}},,{\mathrm{\beta }}_{k|\mathcal{F}})\}}^2$.

Proof of Lemma 2: The proof is given in the supplementary document. □

Appendix B: Proof of Theorems

Proof of Theorem 2: We prove the first part. Under H₀, for any $y\in ℝ$, we have

$$\begin{gathered} E\{\mathbf{1}(Y<y)E(X_k|{\mathbf{x}}_{\mathcal{F}})\}=E[E\{\mathbf{1}(Y<y)|{\mathbf{x}}_{\mathcal{F}}\}{X}_k]=E[E\{\mathbf{1}(Y<y)|{\mathbf{x}}_{\mathcal{F}},X_k\}{X}_k] \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=E\{\mathbf{1}(Y<y)E(X_k|{\mathbf{x}}_{\mathcal{F}},X_k)\}=E\{\mathbf{1}(Y<y)X_k\}. \end{gathered}$$

This completes the proof of the first part. Next we prove the second part. Define $\xi (y)={\{{\xi }_1(y),\hspace{0.17em}\dots \hspace{0.17em},{\xi }_p(y)\}}^{\text{T}}\stackrel{\text{def}}{=}E\{\partial F(y|\mathbf{x})/\partial \mathbf{x}\}$. Stein’s lemma yields that ξ(y) = Σ⁻¹E{1(Y < y)x}. Assumption A1 ensures that, for any $k\in \mathcal{A}$, there exists some $y\in ℝ$ such that ξ_k(y) ≠ 0. Next, we show that for any $\mathcal{F}$ satisfying ${\mathcal{F}}^c\cap \mathcal{A}\ne \varnothing$, $\underset{k\in {\mathcal{F}}^c\cap \mathcal{A}}{\max }{\omega }_{k|\mathcal{F}}>0$ holds. Define ${\mathrm{\Omega }}_{k|\mathcal{F}}(y)\stackrel{\text{def}}{=}E[\mathbf{1}(Y<y)\{X_k-E(X_k|{\mathbf{x}}_{\mathcal{F}})\}]$. The normality of x indicates $E(X_k|{\mathbf{x}}_{\mathcal{F}})={\mathrm{\beta }}_{k|\mathcal{F}}^{\text{T}}{\mathbf{x}}_{\mathcal{F}}$, where ${\mathrm{\beta }}_{k|\mathcal{F}}={\sum }_{\mathcal{F}}^{-1}{\sum }_{\mathcal{F},k}$. Thus ${\mathrm{\Omega }}_{k|\mathcal{F}}(y)=(1,-{\mathrm{\beta }}_{k|\mathcal{F}}^{\text{T}})[E\{\mathbf{1}(Y<y)X_k\}],E\{\mathbf{1}(Y<y){\mathbf{x}}_{\mathcal{F}}^{\text{T}}\}{]}^{\text{T}}$. Without loss of generality, we assume $({\mathbf{x}}_{\mathcal{F}},X_k)$ be the first $|\mathcal{F}|+1$ element of x. It follows that

$${\mathrm{\Omega }}_{k|\mathcal{F}}(y)=(-{\mathrm{\beta }}_{k|\mathcal{F}}^{\text{T}},1)({\mathbf{I}}_{|\mathcal{F}|+1},0_{(|\mathcal{F}|+1)\times (p-|\mathcal{F}|-1)})\mathrm{\Sigma }\xi (y)=({\mathrm{\Sigma }}_{k,\mathcal{S}}-{\mathrm{\Sigma }}_{k,\mathcal{F}}{\mathrm{\Sigma }}_{\mathcal{F}}^{-1}{\mathrm{\Sigma }}_{\mathcal{F},\mathcal{S}})\xi (y),$$

where ${\mathrm{\Sigma }}_{{\mathcal{F}}_1,{\mathcal{F}}_2}=E({\mathbf{x}}_{{\mathcal{F}}_1}^{\text{T}}{\mathbf{x}}_{{\mathcal{F}}_2})$, for ${\mathcal{F}}_1,{\mathcal{F}}_2\subseteq \mathcal{S}$. Under model (3.1), ${\mathrm{\Omega }}_{k|\mathcal{F}}(y)=({\mathrm{\Sigma }}_{k,{\mathcal{F}}^c\cap \mathcal{A}}-{\mathrm{\Sigma }}_{k,\mathcal{F}}{\mathrm{\Sigma }}_{\mathcal{F}}^{-1}{\mathrm{\Sigma }}_{\mathcal{F},{\mathcal{F}}^c\cap \mathcal{A}}){\xi }_{{\mathcal{F}}^c\cap \mathcal{A}}(y)$. This yields that

$$\begin{gathered} \sum _{k\in {\mathcal{F}}^c\cap \mathcal{A}}{\mathrm{\Omega }}_{k|\mathcal{F}}^2(y)=\sum _{k\in {\mathcal{F}}^c\cap \mathcal{A}}{\xi }_{{\mathcal{F}}^c\cap \mathcal{A}}{(y)}^{\text{T}}{({\mathbf{\Sigma }}_{k,{\mathcal{F}}^c\cap \mathcal{A}}-{\mathbf{\Sigma }}_{k,\mathcal{F}}{\mathbf{\Sigma }}_{\mathcal{F}}^{-1}{\mathbf{\Sigma }}_{k,{\mathcal{F}}^c\cap \mathcal{A}})}^2{\xi }_{{\mathcal{F}}^c\cap \mathcal{A}}(y) \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={\xi }_{{\mathcal{F}}^c\cap \mathcal{A}}{(y)}^{\text{T}}{({\mathbf{\Sigma }}_{k,{\mathcal{F}}^c\cap \mathcal{A},{\mathcal{F}}^c\cap \mathcal{A}}-{\mathbf{\Sigma }}_{{\mathcal{F}}^c\cap \mathcal{A},\mathcal{F}}{\mathbf{\Sigma }}_{\mathcal{F}}^{-1}{\mathbf{\Sigma }}_{\mathcal{F},{\mathcal{F}}^c\cap \mathcal{A}})}^2{\text{ξ}}_{{\mathcal{F}}^c\cap \mathcal{A}}(y). \end{gathered}$$

Define

$${\mathrm{\Sigma }}_{\mathcal{F}\cup ({\mathcal{F}}^c\cap \mathcal{A})}\stackrel{\text{def}}{=}\left(\begin{array}{cc}{\mathrm{\Sigma }}_{\mathcal{F}}& {\mathrm{\Sigma }}_{\mathcal{F},{\mathcal{F}}^c\cap \mathcal{A}}\\ {\mathrm{\Sigma }}_{{\mathcal{F}}^c\cap \mathcal{A},\mathcal{F}}& {\mathrm{\Sigma }}_{{\mathcal{F}}^c\cap \mathcal{A},{\mathcal{F}}^c\cap \mathcal{A}}\end{array}\right).$$

Because ${({\mathrm{\Sigma }}_{{\mathcal{F}}^c\cap \mathcal{A},{\mathcal{F}}^c\cap \mathcal{A}}-{\mathrm{\Sigma }}_{{\mathcal{F}}^c\cap \mathcal{A},\mathcal{F}}{\mathrm{\Sigma }}_{\mathcal{F}}^{-1}{\mathrm{\Sigma }}_{\mathcal{F},{\mathcal{F}}^c\cap \mathcal{A}})}^{-1}$ is sub-matrix of ${\mathrm{\Sigma }}_{\mathcal{F}\cup ({\mathcal{F}}^c\cap \mathcal{A})}^{-1}$, we have

$${\rho }_{\min }({({\mathrm{\Sigma }}_{{\mathcal{F}}^c\cap \mathcal{A},{\mathcal{F}}^c\cap \mathcal{A}}-{\mathrm{\Sigma }}_{{\mathcal{F}}^c\cap \mathcal{A},\mathcal{F}}{\mathrm{\Sigma }}_{\mathcal{F}}^{-1}{\mathrm{\Sigma }}_{\mathcal{F},{\mathcal{F}}^c\cap \mathcal{A}})}^{-1})<{\rho }_{\max }({\mathrm{\Sigma }}_{{\mathcal{F}}^c\cup ({\mathcal{F}}^c\cap \mathcal{A})}^{-1}).$$

Accordingly,

$${\rho }_{\min }({\mathrm{\Sigma }}_{{\mathcal{F}}^c\cap \mathcal{A},{\mathcal{F}}^c\cap \mathcal{A}}-{\mathrm{\Sigma }}_{{\mathcal{F}}^c\cap \mathcal{A},\mathcal{F}}{\mathrm{\Sigma }}_{\mathcal{F}}^{-1}{\mathrm{\Sigma }}_{\mathcal{F},{\mathcal{F}}^c\cap \mathcal{A}})>{\rho }_{\min }({\mathrm{\Sigma }}_{{\mathcal{F}}^c\cup ({\mathcal{F}}^c\cap \mathcal{A})})>{\rho }_{\min }(\mathrm{\Sigma }),$$

where _ρmin(M) and _ρmax(M) represent the maximum and minimum eigenvalue of matrix M, respectively. This leads to that

$$\begin{array}{l}\underset{k\in {\mathcal{F}}^c\cap \mathcal{A}}{\max }{\mathrm{\Omega }}_{k|\mathcal{F}}^2(y)\hspace{0.17em}\hspace{0.17em}\ge \hspace{0.17em}\hspace{0.17em}{|{\mathcal{F}}^c\cap \mathcal{A}|}^{-1}\sum _{k\in {\mathcal{F}}^c\cap \mathcal{A}}{\mathrm{\Omega }}_{k|\mathcal{F}}^2(y)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}\ge {|{\mathcal{F}}^c\cap \mathcal{A}|}^{-1}{\rho }_{\min }^2({\mathrm{\Sigma }}_{{\mathcal{F}}^c\cap \mathcal{A},{\mathcal{F}}^c\cap \mathcal{A}}-{\mathrm{\Sigma }}_{{\mathcal{F}}^c\cap \mathcal{A},\mathcal{F}}{\mathrm{\Sigma }}_{\mathcal{F}}^{-1}{\mathrm{\Sigma }}_{\mathcal{F},{\mathcal{F}}^c\cap \mathcal{A}}){\xi }_{{\mathcal{F}}^c\cap \mathcal{A}}^{\text{T}}(y){\xi }_{{\mathcal{F}}^c\cap \mathcal{A}}(y)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}\ge {|{\mathcal{F}}^c\cap \mathcal{A}|}^{-1}{\rho }_{\min }^2(\mathrm{\Sigma }){\xi }_{{\mathcal{F}}^c\cap \mathcal{A}}^{\text{T}}(y){\xi }_{{\mathcal{F}}^c\cap \mathcal{A}}(y)>0,\end{array}$$

for some $y\in ℝ$. This completes the proof of the second part of Theorem 2. □

For notational clarity, we denote ${\mu }_{k|\mathcal{F}}\stackrel{\text{def}}{=}E(X_k|{\mathbf{x}}_{\mathcal{F}})$ in what follows.

Proof of Theorem 3: Define

$${\zeta }_{n,k|\mathcal{F}}(y)\stackrel{\text{def}}{=}{n}^{-1/2}\sum _{i=1}^{n}1(Y_i<y)\{X_{ik}-g_{k|\mathcal{F}}({\mathbf{x}}_{i\mathcal{F}},{\widehat{\beta }}_{k|\mathcal{F}})\},\hspace{1em}\text{for}\hspace{0.17em}y\in ℝ.$$

By Taylor’s expansion, it follows that

$$\begin{gathered} {\zeta }_{n,k|\mathcal{F}}(y)=\hspace{1em}{n}^{-1/2}\sum _{i=1}^{n}1(Y_i<y)\{X_{ik}-g_{k|\mathcal{F}}({\mathbf{x}}_{i\mathcal{F}},{\widehat{\beta }}_{k|\mathcal{F}})\} \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}+\hspace{1em}E{\{\mathbf{1}(Y>y)g_{k|\mathcal{F}}^{\prime }({\mathbf{x}}_{\mathcal{F}},{\beta }_{0,k|\mathcal{F}})\}}^{\text{T}}({\widehat{\beta }}_{k|\mathcal{F}}-{\beta }_{0,k|\mathcal{F}})+o_p(1). \end{gathered}$$

(S.1) in the supplement gives that

$$\begin{gathered} {\zeta }_{n,k|\mathcal{F}}(y)=n^{-1/2}\sum _{i=1}^n{V}_{k|\mathcal{F}}(X_{ik},{\mathbf{x}}_{i\mathcal{F}},Y_i;y)+o_p(1),\hspace{0.17em}\text{where} \\ {V}_{k|\mathcal{F}}(X_{ik},{\mathbf{x}}_{i\mathcal{F}},Y_i;y)=1(Y_i<y)\{X_{ik}-g_{k|\mathcal{F}}({\mathbf{x}}_{i\mathcal{F}}{\beta }_{0,k|\mathcal{F}})\}+E\{1(Y<y)g_{k|\mathcal{F}}^{\prime }{({\mathbf{x}}_{\mathcal{F}}{\beta }_{0,k|\mathcal{F}})}^{\mathrm{T}}\} \\ {\mathrm{\Sigma }}_{k|\mathcal{F}}^{-1}\{X_{ik}-g_{k|\mathcal{F}}({\mathbf{x}}_{i\mathcal{F}},{\beta }_{0,k|\mathcal{F}})\}{g}_{k|\mathcal{F}}^{\prime }({\mathbf{x}}_{i\mathcal{F}},{\beta }_{0,k|\mathcal{F}}), \end{gathered}$$

and ${\mathrm{\Sigma }}_{k|\mathcal{F}}=E[\{g_{k|\mathcal{F}}^{\prime }({\mathbf{x}}_{\mathcal{F}},{\beta }_{0,k|\mathcal{F}})\}{\{g_{k|\mathcal{F}}^{\prime }({\mathbf{x}}_{\mathcal{F}},{\beta }_{0,k|\mathcal{F}})\}}^{\mathrm{T}}].$

Suppose H₀ in (3.2) holds true. Theorem 2 ensures that ${\omega }_{k|\mathcal{F}}=0$. Let ${\zeta }_{k|\mathcal{F}}(\cdot )$ be a mean zero Gaussian process with covariance function $\mathrm{cov}\{{\zeta }_{k|\mathcal{F}}(y_1),{\zeta }_{k|\mathcal{F}}(y_2)\}=E\{V_{k|\mathcal{F}}(X_k,{\mathbf{x}}_{\mathcal{F}},Y;y_1)V_{k|\mathcal{F}}(X_k,{\mathbf{x}}_{\mathcal{F}},Y;y_2)\}$. It is easy to verify that $E\{{\zeta }_{n,k|\mathcal{F}}(y)\}=o(1)$ and $E\{{\zeta }_{n,k|\mathcal{F}}^2(y)\}=\mathrm{cov}\{{\zeta }_{k|\mathcal{F}}(y),{\zeta }_{k|\mathcal{F}}(y)\}+o(1)$. Thus we have ${\zeta }_{n,k|\mathcal{F}}(\cdot )\stackrel{d}{\to }{\zeta }_{k|\mathcal{F}}(\cdot )$, and consequently ${\int }_{-\infty }^{+\infty }{\zeta }_{n,k|\mathcal{F}}^2(y)d{F}_n(y)\stackrel{d}{\to }{\int }_{-\infty }^{+\infty }{\zeta }_{k|\mathcal{F}}^2(y)dF(y)$. This, together with the fact than n $\widehat{\text{CCov}}\{(X_k-{\mu }_{k|\mathcal{F}})|Y\}\stackrel{d}{\to }{\int }_{-\infty }^{+\infty }{\zeta }_{n,k|\mathcal{F}}^2(y)dF(y)$, yields than n $\widehat{\text{CCov}}\{(X_k-{\mu }_{k|\mathcal{F}})|Y\}\stackrel{d}{\to }{\int }_{-\infty }^{+\infty }{\zeta }_{k|\mathcal{F}}^2(y)dF(y)$ (Kuo, 1975). Therefore,

$$n{\left[E\left\{{\int }_{-\infty }^{+\infty }{\zeta }_{k|\mathcal{F}}^2(y)dF(y)\right\}\right]}^{-1}\widehat{\text{CCov}}\{(X_k-{\mu }_{k|\mathcal{F}})|Y\}\stackrel{d}{\to }\sum _{j=1}^{\infty }{\lambda }_{j,k|\mathcal{F}}{\chi }_j^2(1).$$

By Slutsky’s theorem, it follows that $n{K}_{k|\mathcal{F}}^{-1}{\widehat{\omega }}_{k|\mathcal{F}}\stackrel{d}{\to }{\sum }_{j=1}^{\infty }{\lambda }_{j,k|\mathcal{F}}{\chi }_j^2(1)$, where

$$\begin{gathered} k_{k|\mathcal{F}}\stackrel{\text{def}}{=}E[\mathbf{1}(Y<\tilde{Y})\{X_k-g_{k|\mathcal{F}}({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{0,k|\mathcal{F}})\}+E{\{\mathbf{1}(Y<\tilde{Y})g_{k|\mathcal{F}}^{\prime }({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{0,k|\mathcal{F}})\}}^{\text{T}}{\mathrm{\Sigma }}_{k|\mathcal{F}}^{-1} \\ \hspace{1em}\hspace{1em}\hspace{1em}{\{X_k-g_{k|\mathcal{F}}({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{0,k|\mathcal{F}})\}{g}_{k|\mathcal{F}}^{\prime }({\mathbf{x}}_{\mathcal{F}},{\mathrm{\beta }}_{0,k|\mathcal{F}})]}^2/\mathrm{var}\{X_k-g_{k|\mathcal{F}}({\mathbf{x}}_{i\mathcal{F}},{\mathrm{\beta }}_{0,k|\mathcal{F}})\}. \end{gathered}$$ (B.1)

Suppose H₁ in (3.2) holds true. Recall that

$$\begin{gathered} \hspace{1em}\widehat{\text{CCov}}\{(X_k-{\mu }_{k|\mathcal{F}})|Y\}=\int \{n^{-1/2}{\zeta }_{n,k|\mathcal{F}}(y){\}}^{2}d{F}_n(y) \\ =\int {[n^{-1/2}{\zeta }_{n,k|\mathcal{F}}(y)-E\{V_{k|\mathcal{F}}(X_k,{\mathbf{x}}_{\mathcal{F}},Y;y)\}+E\{V_{k|\mathcal{F}}(X_k,{\mathbf{x}}_{\mathcal{F}},Y;y)\}]}^{2}d{F}_n(y) \\ =\int 2E\{V_{k|\mathcal{F}}(X_k,{\mathbf{x}}_{\mathcal{F}},Y;y)\}{[n^{-1/2}{\zeta }_{n,k|\mathcal{F}}(y)-E\{V_{k|\mathcal{F}}(X_k,{\mathbf{x}}_{\mathcal{F}},Y;y)\}]}^{2}d{F}_n(y) \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\int E^2\{V_{k|\mathcal{F}}(X_k,{\mathbf{x}}_{\mathcal{F}},Y;y)\}d{F}_n(y)+o_p(n^{-1/2}) \\ =\int 2E\{V_{k|\mathcal{F}}(X_k,{\mathbf{x}}_{\mathcal{F}},Y;y)\}{n}^{-1/2}{\zeta }_{n,k|\mathcal{F}}(y)dF(y)-2\text{CCov}(X_k-{\mu }_{k|\mathcal{F}})|Y\} \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\int E^2\{V_{k|\mathcal{F}}(X_k,{\mathbf{x}}_{\mathcal{F}},Y;y)\}d{F}_n(y)+o_p(n^{-1/2}). \end{gathered}$$

Thus

$$\widehat{\text{CCov}}\{(X_k-{\mu }_{k|\mathcal{F}})|Y\}-\text{CCov}\{(X_k-{\mu }_{k|\mathcal{F}})|Y\}=n^{-1}\sum _{i=1}^n{Z}_{i,k|\mathcal{F}}+o_p(n^{-1/2}),$$

with

$$\begin{gathered} Z_{i,k|\mathcal{F}}\stackrel{\text{def}}{=}2[\int E\{V_{k|\mathcal{F}}(X_k,{\mathbf{x}}_{\mathcal{F}},Y;y)\}{V}_{k|\mathcal{F}}(X_{ik},{\mathbf{x}}_{i\mathcal{F}},Y_i;y)dF(y)-\text{CCov}\{(X_k-{\mu }_{k|\mathcal{F}})|Y\}] \\ \hspace{1em}\hspace{1em}+\hspace{1em}{E}^2\{V_{k|\mathcal{F}}(X_k,{\mathbf{x}}_{\mathcal{F}},Y;Y_i)\}-\text{CCov}\{(X_k-{\mu }_{k|\mathcal{F}})|Y\}, \end{gathered}$$ (B.2)

where the expectation is taken with respect to $(X_k,{\mathbf{x}}_{\mathcal{F}},Y)$. By the central limit theorem, $n^{1/2}[\widehat{\text{CCov}}\{X_k-{\mu }_{k|\mathcal{F}}\}|Y-\text{CCov}\{X_k-{\mu }_{k|\mathcal{F}}\}|Y]$ converges in distribution to $\mathcal{N}(0,{\varsigma }_{k|\mathcal{F}}^2)$, where ${\varsigma }_{k|\mathcal{F}}^2\stackrel{\text{def}}{=}\mathrm{var}(Z_{i,k|\mathcal{F}})$. By Slutsky’s theorem, we have $n^{1/2}({\widehat{\omega }}_{k|\mathcal{F}}-{\omega }_{k|\mathcal{F}})\stackrel{d}{\to }\mathcal{N}(0,{\mathrm{\Delta }}_{k|\mathcal{F}})$, where

$${\mathrm{\Delta }}_{k|\mathcal{F}}\stackrel{\text{def}}{=}{\varsigma }_{k|\mathcal{F}}^2/{\{\mathrm{var}(X_k-{\mu }_{k|\mathcal{F}})\}}^2.$$ (B.3)

This completes the proof of Theorem 3.

The uniform consistency of ${\widehat{\omega }}_{k|\mathcal{F}}$ paves the road for proving Theorem 4.

Proposition 1. Under conditions (B1)-(B5), for any ε_n > 0, there exists positive constants c₁, c₂, c₃, c₄ and sufficiently small $s_{{\epsilon }_n}\in (0,2/{\epsilon }_n)$ such that

$$\begin{array}{l}pr\{\underset{k\in {\mathcal{F}}^c}{\max }|{\widehat{\omega }}_{k|\mathcal{F}}-{\omega }_{k|\mathcal{F}}|>{\epsilon }_n\}\le O[p\exp \{n\log (1-{\epsilon }_n{s}_{{\epsilon }_n}/2)/3\}+p\exp (-c_{1}n{\epsilon }_n^2)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+pn\exp (-c_{2}n{\epsilon }_n^2)+ps\exp (-c_{3}n{s}^{-2}{\epsilon }_n^2)+ps\exp (-c_{4}n{s}^{-2}{\epsilon }_n)].\end{array}$$

Set ε_n = Cn^−κ, for some constants C > 0 and κ > 0. If there exists ϑ > 0 such that p = o{exp(an^ϑ)} for any a > 0 and 3/5 − 2κ − ϑ > 0, then

$$\underset{\mathcal{F}:|\mathcal{F}|=o(n^{1/5})}{\max }pr\{\underset{k\in {\mathcal{F}}^c}{\max }|{\widehat{\omega }}_{k|\mathcal{F}}-{\omega }_{k|\mathcal{F}}|>C{n}^{-k}\}\to 0\hspace{0.17em}as\hspace{0.17em}n\to \infty .$$

Proof of Proposition 1: The proof is given in the supplementary document. □

Proof of Theorem 4: For notational clarity, we define the random event E₁ = {There exists an index set $\mathcal{F}$, such that ${\mathcal{F}}^c\cap \mathcal{A}\ne \varnothing$ and $\underset{k\in {\mathcal{F}}^c\cap \mathcal{A}}{\max }{\widehat{\omega }}_{k|\mathcal{F}}\le \nu$}. For such an $\mathcal{F}$, we have, by Assumption A3, $\underset{k\in {\mathcal{F}}^c\cap \mathcal{A}}{\max }{\omega }_{k|\mathcal{F}}-\underset{k\in {\mathcal{F}}^c\cap \mathcal{A}}{\max }{\widehat{\omega }}_{k|\mathcal{F}}>C{n}^{-\varpi }-\nu$. Consequently,

$$\underset{k\in {\mathcal{F}}^c\cap \mathcal{A}}{\max }({\omega }_{k|\mathcal{F}}-{\widehat{\omega }}_{k|\mathcal{F}})\ge \underset{k\in {\mathcal{F}}^c\cap \mathcal{A}}{\max }{\omega }_{k|\mathcal{F}}-\underset{k\in {\mathcal{F}}^c\cap \mathcal{A}}{\max }{\omega }_{k|\mathcal{F}}\ge C{n}^{-\varpi }/2.$$

Define the random event $E_2=\{\underset{k\in {\mathcal{F}}^c\cap \mathcal{A}}{\max }({\omega }_{k|\mathcal{F}}-{\widehat{\omega }}_{k|\mathcal{F}})>C{n}^{-\varpi }/2,\hspace{0.17em}\text{for}\hspace{0.17em}\mathcal{F}\hspace{0.17em}\text{in}\hspace{0.17em}{E}_1\}$. The above discussions imply that E₁ ⊆ E₂. It follows that

$$\text{pr}\left(\underset{\mathcal{F}:{\mathcal{F}}^c\cap \mathcal{A}\ne \varnothing }{\min }\underset{k\in {\mathcal{F}}^c\cap \mathcal{A}}{\max }{\widehat{\omega }}_{k|\mathcal{F}}>\nu \right)=1-\text{pr}(E_1)\ge 1-\text{pr}(E_2).$$

Proposition 1 implies that pr(E₂) → 0, which completes the proof of Theorem 4. □