Sparse Nonparametric Regression With Regularized Tensor Product Kernel

Summary

With growing interest to use black-box machine learning for complex data with many feature variables, it is critical to obtain a prediction model that only depends on a small set of features to maximize generalizability. Therefore, feature selection remains to be an important and challenging problem in modern applications. Most of existing methods for feature selection are based on either parametric or semiparametric models, so the resulting performance can severely suffer from model misspecification when high-order nonlinear interactions among the features are present. A very limited number of approaches for nonparametric feature selection were proposed, but they are computationally intensive and may not even converge. In this paper, we propose a novel and computationally efficient approach for nonparametric feature selection in regression field based on a tensor-product kernel function over the feature space. The importance of each feature is governed by a parameter in the kernel function which can be efficiently computed iteratively from a modified alternating direction method of multipliers (ADMM) algorithm. We prove the oracle selection property of the proposed method. Finally, we demonstrate the superior performance of our approach compared to existing methods via simulation studies and application to the prediction of Alzheimer’s disease.

1 |. INTRODUCTION

Applications with big data often contain many noisy features that will obscure true signals and deteriorate prediction. With growing interest to use complex data and black-box models to predict an outcome with noisy features, it is critical to obtain generalizable and accurate prediction models that only depend on a small set of features. Therefore, feature selection remains to be crucial for current big data applications. For example, for neurodegenerative diseases such as Parkinson’s disease and Alzheimer’s disease, identifying a few disease diagnostic and prognostic biomarkers is the focal of research for early disease detection and developing intervention. In particular, distinguishing useful biomarkers from noisy ones is essential for identifying individuals at risk long before irreversible damage has occurred, which has implications for prevention and therapeutic development. As another example, type 2 diabetic patient’s health and medical records are routinely captured electronically over time. Such electronic health records consist of patient’s vital signs (blood pressure, heart rates), disease diagnostic biomarkers (e.g., glucose and cholesterol level), co-morbidities and medication history. Thus, it is important to determine which features are predictive of diseases and their treatment outcomes in order to manage individual patient’s healthcare under the framework of precision medicine.

There is an extensive literature on variable selection methods in regression field for parametric and semiparametric models including linear, generalized linear and additive models (e.g., LASSO, Tibshirani (1996), SCAD, Fan and Li (2001), COSSO, Lin and Zhang (2006), and MCP, Zhang (2010)). In these methods, the importance of feature variables is uniquely determined by non-zero coefficients or some univariate functions in the models. However, high-order interaction is often present among the features in many biomedical applications, so parametric or semiparametric models are likely to be misspecified. Theoretical results on variable selection under these misspecified models thus no longer hold. Approaches proposed for nonparametric feature selection include filter methods and wrapper methods. The filter method performs feature selection using various dependence measures. For example, Guyon and Elisseeff (2003) assigned each feature an importance score based on its correlation or mutual information with the outcome of interest, and then the features with low score were removed; Fan and Lv(2008) proposed Sure Independence Screening (SIS) to reduce high dimensionality to relatively large scale, which can be further extended to marginal nonparametric learning (Fan, Feng, and Song (2011)). Song, Smola, Gretton, Bedo, and Borgwardt (2012) proposed a Hilbert-Schmidt Independence Criterion as the dependence measure and a greedy procedure for feature selection. However, the filter method relies on marginal relationship between each feature and the outcome so cannot correctly capture the higher-order interactions among the features. The wrapper methods (Kohavi and John (1997); Liu and Zheng (2006); Maldonado and Weber (2009); Chen and Chen (2015), Dasgupta, Goldberg, and Kosorok (2019)) adopt a greedy search algorithm to generate subsets of the features via forward or backward elimination. These methods are computationally demanding and sequential elimination procedures are likely to lead to cumulative errors over steps.

Since nonparametric prediction can be achieved using approximation from a reproducing kernel Hilbert space (RKHS), several works considered to incorporate feature selection into the construction of such space. Specifically, Weston et al. (2001) introduced a binary indicator variable for each feature in the kernel function that yielded the RKHS, and then performed variable selection using greedy search, thus is computationally intensive. In a more recent work by Allen (2013), they proposed a procedure named KerNel Iterative Feature Extraction. In this approach, the feature input was constructed in a Gaussian RKHS in order to perform any nonparametric prediction. Different bandwidths were used in the Gaussian kernel function for each feature so that a larger bandwidth implied less importance of the corresponding feature. In this way, variable selection was likely to be achieved by tuning the bandwidths data-adaptively. However, due to the nonlinearity of the Gaussian kernel and the high sensitivity to the bandwidth choices, in our numerical experience, this method is unstable even when the number of the feature variables is moderate.

In this work, we propose a novel and computationally efficient approach for nonparametric feature selection when predicting continuous outcomes. Our method considers the feature space from a RKHS that is defined based on a novel tensor-product kernel. The tensor product kernel has been commonly used to integrate features from multiple domains (c.f., Gao and Wu (2012)) in order to account for any highly nonlinear interactions among the domains. For feature selection, we treat each individual feature as a different domain so the use of the tensor product kernel can potentially capture nonlinear high-order interactions among the features, yielding an adequate approximation to any underlying prediction function. Furthermore, we introduce regularization parameters in the tensor product kernel where each parameter determines the importance of the corresponding feature variable. In this way, we can estimate the regularization parameters adaptively from data in order to achieve feature selection and nonparametric function estimation at the same time. Computationally, the estimation of the regularization parameters can be solved efficiently using an iterative procedure, where each iteration is based on a modified alternating direction method of multipliers (ADMM) algorithm. The method essentially reduces to finding optimums of quadratic functions with positive constraints. The unique construction of tensor product kernel results in much greater computational efficiency and numerical stability as compared to previous approaches that either depend on subset search or use a highly nonlinear Gaussian kernel function. We prove the theoretical properties of the proposed method including Fisher consistency and feature selection consistency.

The paper is organized as follows. In Section 2, we describe the proposed method based on a regularized tensor product kernel and then discuss the details of the computational algorithms in our method. In Section 3, we provide the theorems for the Fisher consistency and oracle variable selection property of our method. Numerical evidence based on simulations and application are given Section 4 and 5. We conclude the paper with discussion in Section 6.

2 |. METHOD

Let Y denote the outcome of interest and $\mathbf{X}=\left({\mathbf{X}}_1,\dots ,{\mathbf{X}}_{\text{p}}\right)$ denote the p-dimensional feature variables. Our goal is to learn a nonparametric prediction function, denoted by f(X), to predict Y using data from n independent subjects, denoted by (X_i, Y_i), i = 1, 2, …n. In the following sections, we focus on L₂-loss to quantify the prediction performance for our method development, although the whole framework applies any other convex loss functions.

2.1 |. Empirical risk minimization on RKHS

Let ${\mathcal{H}}_{\kappa }$ denote a RKHS with kernel function $\kappa \left(\mathbf{X},\tilde{\mathbf{X}}\right)$ (Hofmann, Schölkopf, and Smola (2008)), equipped with norm ${\Vert \cdot \Vert }_{{\mathcal{H}}_{\kappa }}$. Commonly used kernel functions for κ include the Gaussian kernel, $\kappa \left(\mathbf{x},\mathbf{y}\right)=\exp \left(-{\Vert \mathbf{x}-\mathbf{y}\Vert }^2/2{\sigma }^2\right)$, and Epanechnikov kernel, $\kappa \left(\mathbf{x},\mathbf{y}\right)=\frac{3}{4\text{h}}\left(1-\frac{{\Vert \mathbf{x}-\mathbf{y}\Vert }^2}{{\text{h}}^2}\right)|\left(\Vert \mathbf{x}-\mathbf{y}\Vert \le \text{h}\right)$, in ${\mathcal{R}}^{\text{p}}$. The empirical regularized risk minimization on RKHS for estimating f(X) solves the following problem:

$$\underset{f}{\min }{P}_n\left({\left(Y-f\left(X\right)\right)}^2\right)+{\gamma }_n{\Vert f\Vert }_{{\mathcal{H}}_{\kappa }}^2,$$

where P_n denotes the empirical measure from n observations, i.e., for any function g(Y, X), ${\mathbf{P}}_{\text{n}}\text{g}\left(\mathbf{Y},\mathbf{X}\right)={\text{n}}^{-1}{\sum }_{\text{i}=1}^{\text{n}}\text{g}\left({\mathbf{Y}}_{\text{i}},{\mathbf{X}}_{\text{i}}\right)$, and ${\gamma }_n$, is a tuning parameter to control the complexity of f. Based on the representation theory for RKHS, this optimization is equivalent to solving

$$\underset{\mathit{\alpha }}{\min }\frac{1}{n}\sum _{i=1}^n{\left\{{\mathit{Y}}_i-\sum _{j=1}^n{\alpha }_j\kappa \left(X_i,X_j\right)\right\}}^2+{\gamma }_n{\mathit{\alpha }}^{\mathbf{T}}K\mathit{\alpha },$$

where $\mathit{\alpha }={\left({\alpha }_1,{\alpha }_2,\dots ,{\alpha }_{\text{n}}\right)}^{\mathbf{T}}$. If we denote the kernel matrix as $\mathbf{K}=\left\{\kappa \left(X_{\text{i}},X_{\text{j}}\right)\right\}\in {\mathcal{R}}^{\text{n}\times \text{n}}$, then the solution for α is

$$\widehat{\mathit{\alpha }}={\left(K^{\mathbf{T}}K+n{\gamma }_{n}K\right)}^{-1}{K}^{\mathbf{T}}Y,$$

where $\mathbf{Y}={\left({\mathbf{\text{Y}}}_1,\dots ,{\mathbf{\text{Y}}}_{\text{n}}\right)}^{\mathbf{\text{T}}}$. The resulting prediction function is $\widehat{\text{f}}\left(\mathbf{X}\right)={\sum }_{\text{i=1}}^{\text{n}}{\widehat{\alpha }}_{\text{i}}\kappa \left({\mathbf{X},\mathbf{X}}_{\text{i}}\right)$. The tuning parameter ${\gamma }_{\text{n}}$ is estimated via cross-validation.

2.2 |. Feature selection using a regularized tensor product kernel

In this section, we describe our proposed method for nonparametric feature selection. First, we introduce a regularized tensor product kernel as follows: for a given nonnegative vector λ = (λ₁, λ₂, ⋯ λ_p)^T, we define λ-regularized kernel function for X = (X₁, X₂, ⋯ , X_p)^T and $\tilde{\mathbf{X}}={\left(\widetilde{{\mathbf{X}}_1},\widetilde{{\mathbf{X}}_2},\dots ,\widetilde{{\mathbf{X}}_{\text{p}}}\right)}^{\mathbf{\text{T}}}$ as

$${\kappa }_{\mathbf{\lambda },{\sigma }_n}\left(X,\tilde{\mathbf{X}}\right)=\prod _{m=1}^p\left\{1+{\lambda }_m{\kappa }_n\left(X_m,{\tilde{X}}_m\right)\right\},$$ (1)

where ${\kappa }_{\text{n}}\left(\text{x},\text{y}\right)=\exp \left\{-{\left(\text{x}-\text{y}\right)}^2/2{\sigma }_{\text{n}}^2\right\}$ so is proportional to the univariate Gaussian kernel with a pre-defined bandwidth σ_n in $\mathcal{R}$. Essentially, this kernel function is a tensor-product kernel where for each domain (individual feature in our case, sharing similar idea commonly used for multitask learning (e.g. Suzuki, Kanagawa, Kobayashi, Shimizu, and Tagami (2016)) the kernel function is given by $1+{\lambda }_{\text{m}}{\kappa }_{\text{n}}\left(\text{x},\text{y}\right)$. One significant feature of this kernel is that there is a non-negative parameter, ${\lambda }_{\text{m}}$, that regularizes the contribution of feature m to the entire feature space. In Figure 1, we plot such regularized tensor-product kernel in a 2-dimensional feature space when varying λ₁ and λ₂. Clearly, when λ₁ becomes relatively smaller than λ₂, the entire kernel function is increasingly dominated by X₂. When λ₁ decreases to λ₁ = 0 and λ₂ > 0, the kernel function is flat along the direction of X₁ and only X₂ is actively contributing to the distance measure. Similarly, when λ₁ > 0 and λ₂ = 0, only X₁ is actively contributing to the kernel function. Note that for categorical feature variables, κ_n(x, y) reduces to I(x = y) when σ_n is small enough.

FIGURE 1.

Plots of tensor product kernel in ${\mathcal{R}}^2$.

Note: The bandwidth ${\sigma }_{\text{n}}=\sqrt{5}$ and each kernel is centered at 0. Settings Of λ: from left To right: λ₁ = 0; λ₂ = 10; λ₁ = 4, λ₂ = 10; λ₁ = 2, λ₂ = 10; λ₁ = 2, λ₂ = 0.

Several interesting properties of the proposed tensor-product kernel are note-worthy. First, κ_n used in the construction is the Gaussian kernel function so it preserves the universal approximation property in RKHS of the Gaussian kernel when σ_n is chosen to be small (see Lemma A.1 and A.2 in the appendix). In this way, we expect that the estimation over the RKHS generated by this tensor-product kernel can approximate any underlying nonparametric prediction function. Second, the regularization parameters, which determine the contribution of each feature variable, can be estimated data-adaptively to reveal the true importance of the feature variables and true shape of the underlying prediction function. In particular, if λ_m=0, the kernel function no longer depends on the m-th feature variable. Therefore, we can achieve the goal of feature selection by estimating the regularization parameters from the data through this tensor-product kernel. Finally, there is significant computation advantage when searching sparse functions based on λ’s, as will be detailed below.

Denote ${\mathcal{H}}_{\lambda ,{\sigma }_{\text{n}}}$ as the RKHS corresponding to ${\kappa }_{\lambda ,{\sigma }_n}$. We aim to minimize

$$\begin{array}{l}{\mathbf{L}}_n\left(\mathbf{\lambda },f\right)=P_{\text{n}}\left({\left(\mathbf{Y}-\mathbf{f}\left(X\right)\right)}^2\right)+\gamma 1\text{n}{\Vert \mathbf{f}\Vert }_{{\mathcal{H}}_{\mathbf{\lambda },{\sigma }_{\text{n}}}}^2+\gamma 2\text{n}{\Vert \mathbf{\lambda }\Vert }_0\\ \mathrm{subject\; to}{\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_{\text{p}}\ge 0,\end{array}$$ (2)

where ${\Vert \mathbf{\lambda }\Vert }_0={\sum }_{\text{m}=1}^{\text{p}}\mathbf{\text{I}}\left({\lambda }_{\text{m}}\ne 0\right)$, and both γ_1n and γ_2n are tuning parameters. Note that in the objective function (2), in order to perform variable selection, we include a I₀-penalty in the third term to select the non-zero regularization parameters. With the L₂-loss, the optimization problem is equivalent to

$$\begin{array}{l}\underset{\mathbf{\lambda },\mathit{\alpha }}{\min }\frac{1}{\text{n}}\sum _{\text{i}=1}^{\text{n}}{\left\{{\mathbf{Y}}_{\text{i}}-\sum _{\text{j}=1}^{\text{n}}{\alpha }_{\text{j}}{\kappa }_{\mathbf{\lambda },{\sigma }_{\text{n}}}\left(X_{\text{i}},X_{\text{j}}\right)\right\}}^2+{\gamma }_{1\text{n}}{\mathit{\alpha }}^{\mathbf{T}}{K}_{\mathbf{\lambda },{\sigma }_{\text{n}}}\mathit{\alpha }+{\gamma }_{2\text{n}}{\Vert \mathbf{\lambda }\Vert }_0\\ \mathrm{subject\; to}{\gamma }_{1\text{n}}\ge 0,{\gamma }_{2\text{n}}\ge 0,\end{array}$$ (3)

where ${\mathbf{\text{K}}}_{\mathbf{\lambda },{\sigma }_{\text{n}}}$ is the matrix given by $\left({\kappa }_{\mathbf{\lambda },{\sigma }_{\text{n}}}\left(X_{\text{i}},X_{\text{j}}\right)\right)$. Note that we optimize over both α and λ, so this procedure performs estimation of nonparametric prediction function (via updating α) and search of sparse function space (via updating λ) simultaneously. This is analogous to feature selection in LASSO for (parametric) linear models where one aims to find the optimal prediction and most sparse linear functions at the same time. In fact, this optimization is NP-hard and when p is large it requires that we evaluate all possible subsets of nonzero coefficients of λ. Instead, we solve an approximate optimization problem to (3) based on a modified ADMM algorithm. First, with a surrogate parameter θ, we re-formulate the objective function (3) as,

$$\begin{array}{l}\underset{\mathbf{\lambda },\mathit{\alpha }}{\min }\frac{1}{\text{n}}\sum _{\text{i}=1}^{\text{n}}{\left({\mathbf{Y}}_{\text{i}}-\sum _{\text{j}=1}^{\text{n}}{\alpha }_{\text{j}}{\kappa }_{\mathbf{\lambda },{\sigma }_{\text{n}}}\left(X_{\text{i}},X_{\text{j}}\right)\right)}^2+{\gamma }_{1\text{n}}{\mathit{\alpha }}^{\mathbf{T}}{K}_{\mathbf{\lambda },{\sigma }_{\text{n}}}\mathit{\alpha }+{\gamma }_{2\text{n}}{\Vert \mathit{\theta }\Vert }_0\\ \mathrm{subject\; to}\sum _{\text{m}=1}^{\text{p}}\left|{\lambda }_{\text{m}}-{\theta }_{\text{m}}\right|\le 0,{\lambda }_1,\dots ,{\lambda }_{\text{p}}\ge 0.\end{array}$$

The Lagrange multiplier is denoted by ${\gamma }_{3\text{n}}\left({\gamma }_{3\text{n}}>0\right)$. The Lagrange form of the reformulated objective function becomes,

$$\frac{1}{\text{n}}\sum _{\text{i}=1}^{\text{n}}{\left({\mathbf{Y}}_{\text{i}}-\sum _{\text{j}=1}^{\text{n}}{\alpha }_{\text{j}}{\kappa }_{\mathbf{\lambda },{\sigma }_{\text{n}}}\left(X_{\text{i}},X_{\text{j}}\right)\right)}^2+{\gamma }_{1\text{n}}{\mathit{\alpha }}^{\mathbf{\text{T}}}{K}_{\mathbf{\lambda },{\sigma }_{\text{n}}}\mathit{\alpha }+{\gamma }_{2\text{n}}{\Vert \mathit{\theta }\Vert }_0+{\gamma }_{3\text{n}}\sum _{\text{m}=1}^{\text{p}}\left|{\lambda }_{\text{m}}-{\theta }_{\text{m}}\right|,$$ (4)

subject to ${\lambda }_{\text{m}}\ge 0$, m = 1, …, p. The advantage of the approximation in (4) is that the objective function is strictly convex for λ_m’s while the solution for θ_m’s is explicit given the other parameters. The details of the algorithm are given in next section.

2.3 |. Algorithms

We iteratively update all parameters to minimize (4). At the k-th iteration,

$${\mathit{\alpha }}^{\text{k}+1}={\left(K_{{\mathbf{\lambda }}^{\text{k}},{\sigma }_{\text{n}}}^{\mathbf{T}}{K}_{{\mathbf{\lambda }}^{\text{k}},{\sigma }_{\text{n}}}+\text{n}{\gamma }_{1\text{n}}{K}_{{\mathbf{\lambda }}^{\text{k}},{\sigma }_{\text{n}}}\right)}^{-1}{K}_{{\mathbf{\lambda }}^{\text{k}},{\sigma }_{\text{n}}}^{\mathbf{T}}Y$$ (5)

$${\mathbf{\lambda }}^{\text{k}+1}=\underset{\mathbf{\lambda }}{\min }\frac{1}{\text{n}}{\sum _{\text{i}=1}^{\text{n}}\left({\mathbf{Y}}_{\text{i}}-\sum _{\text{j}=1}^{\text{n}}{\alpha }_{\text{j}}^{\text{k}+1}{\kappa }_{\mathbf{\lambda },{\sigma }_{\text{n}}}\left(X_{\text{i}},X_{\text{j}}\right)\right)}^2+{\gamma }_{1\text{n}}{\left({\mathit{\alpha }}^{\text{k}+1}\right)}^{\mathbf{\text{T}}}{K}_{\mathbf{\lambda },{\sigma }_{\text{n}}}{\mathit{\alpha }}^{\text{k}+1}+{\gamma }_{3\text{n}}\sum _{\text{m}=1}^{\text{p}}\left|{\lambda }_{\text{m}}-{\theta }_{\text{m}}^{\text{k}}\right|$$ (6)

$${\mathit{\theta }}^{\text{k}+1}=\underset{\mathit{\theta }}{\min }{\gamma }_{3\text{n}}\sum _{\text{m}=1}^{\text{p}}\left|{\lambda }_{\text{m}}^{\text{k}+1}-{\theta }_{\text{m}}\right|+{\gamma }_{2\text{n}}\sum _{\text{m}=1}^{\text{p}}\mathbf{I}\left({\theta }_{\text{m}}\ne 0\right).$$ (7)

Note that (5) is an explicit expression and that the update θ in (7) is given by

$${\theta }_q^{\kappa +1}={\lambda }_q^{\kappa +1}I\left(\left|{\lambda }_q^{\kappa +1}\right|>{\rho }_n\right),$$

where ${\rho }_{\text{n}}={\gamma }_{2\text{n}}/{\gamma }_{\text{3n}}$.

The update function in (6) is essentially a regression problem with a LASSO-type penalty. Specifically, we use a coordinate descent algorithm to obtain each ${\lambda }_{\text{q}}\left(\text{q}=1,2,\dots ,\text{p}\right)$. To obtain ${\lambda }_{\text{q}}^{\text{k}+1}$, we fix ${\lambda }_1^{\text{k}+1},{\lambda }_2^{\text{k}+1},\cdots ,{\lambda }_{\text{q}+1}^{\text{k}},{\lambda }_{\text{q}+2}^{\text{k}},\dots ,{\lambda }_{\text{p}}^{\text{k}}$ and then after simple calculation, the objective function takes the following form,

$$\underset{{\lambda }_q}{\min }\frac{1}{n}\sum _{i=1}^n{\left(a_{iq}+b_{iq}{\lambda }_q\right)}^2+d_q{\lambda }_q,$$

where a_iq, b_iq, d_q’s are constants with their expressions given in Appendix S1. This is quadratic in λ_q with constraint ${\lambda }_{\text{q}}\ge 0$; Thus, its solution can be achieved easily by checking whether the minimal value is non-negative or not. The prediction function at an iteration is defined as

$${\widehat{f}}_{{\mathbf{\lambda }}^{k+1}}\left(X\right)=\sum _{i=1}^n{\alpha }_i^{k+1}{\kappa }_{{\mathbf{\lambda }}^{k+1},{\sigma }_n}\left(X,X_i\right).$$

Since our goal is to minimize the objective function which penalizes the size of the non-zero λ’s, we set the convergence criteria to be the change of both the objective function and the non-zero number of λ’s. We let $\delta =|{\mathbf{\text{L}}}_{\text{n}}({\widehat{\mathbf{\lambda }}}^{\text{k}+1},{\widehat{\mathbf{f}}}_{{\widehat{\mathbf{\lambda }}}^{\text{k}+1}})-{\mathbf{\text{L}}}_{\text{n}}({\widehat{\mathbf{\lambda }}}^{\text{k}},{\widehat{\mathbf{f}}}_{{\widehat{\mathbf{\lambda }}}^{\text{k}}})|$ and $\text{e}={\Vert {\widehat{\mathbf{\lambda }}}^{\text{k}+1}\Vert }_0$. Then our algorithm can be summarized as follows:

1. At the initial step, set ${\widehat{\mathbf{\lambda }}}_0=\mathbf{0}$, ${\widehat{\mathit{\theta }}}_0=\mathbf{0}$.

2. Fix ${\widehat{\mathbf{\lambda }}}^{\text{k}}$ then update ${\widehat{\mathit{\alpha }}}^{\text{k}+1}$.

3. For fixed ${\widehat{\mathit{\alpha }}}^{\text{k}+1}$, update ${\widehat{\mathbf{\lambda }}}^{\text{k}+1}$ and ${\widehat{\mathit{\theta }}}^{\text{k}+1}$ via coordinate descent algorithm.

4. Calculate $\delta =|{\mathbf{\text{L}}}_{\text{n}}({\widehat{\mathbf{\lambda }}}^{\text{k}+1},{\widehat{\mathbf{f}}}_{{\widehat{\mathbf{\lambda }}}^{\text{k}+1}})-{\mathbf{\text{L}}}_{\text{n}}({\widehat{\mathbf{\lambda }}}^{\text{k}},{\widehat{\mathbf{f}}}_{{\widehat{\mathbf{\lambda }}}^{\text{k}}})|$ and $\text{e}={\Vert {\widehat{\mathbf{\lambda }}}^{\text{k}+1}\Vert }_0$.

5. Stop if $\delta \le \text{c}$ (c is a given cut point) and e doses not change. Otherwise, go to step (ii) with updated ${\widehat{\mathbf{\lambda }}}^{\text{k}+1}$.

Since our computation alternates between α’s and λ, each computation given other parameters is a convex optimization problem. Thus, our algorithm guarantees that the objective function decreases over iterations and converges to a local minimum.

In our algorithm, both the tuning parameters and bandwidth need to be determined. First, following the median trick for bandwidth of Gaussian kernel (Jaakkola, Diekhans, and Haussler (1999)), we calculate the matched-pair distance of feature variables and set σ_n such that the proportion of the matched pairs with distance less than σ_n is about a half. To tune the other parameters including γ_1n, γ_3n and ρ_n (equivalently, γ_2n), we use 5-fold cross-validation by varying them on a grid of 2⁻¹⁵, 2⁻¹⁴, ⋯ , 2¹⁵.

3 |. THEORETICAL RESULTS

In this section, we provide theoretical results to justify the proposed method. In particular, we show that under some assumptions, the resulting prediction function from our method leads to Bayesian risk asymptotically. Furthermore, we show that with probability tending to one, the variable selection based on non-zero λ’s is oracle, i.e., as if we had known which variables were important. Without loss of generality, we assume that the first r feature variables are important, while the others are not; that is, the Bayes rule, $\mathbf{E}\left[\mathbf{Y}|\mathbf{X}\right]$, only depends on X₁, … , X_r in the sense that for m ≤ r,

$$E\left[{\left(E\left[Y|\mathbf{X}\right]-E\left[Y|X_{-m}\right]\right)}^2\right]>0,$$

with probability 1,

$$E\left[Y|X\right]=E\left[Y|X_1,X_2,\cdots ,X_r\right],$$

where X_−m denotes the random vector of X excluding X_m. We use ${\mathbf{\text{f}}}_0\left({\mathbf{\text{X}}}_1,\dots ,{\mathbf{\text{X}}}_{\text{r}}\right)$ to denote $\mathbf{E}\left[\mathbf{Y}|\mathbf{X}\right]$. We further denote $(\widehat{\mathbf{\lambda }},{\widehat{\mathbf{f}}}_{\widehat{\mathbf{\lambda }}})$ as the optimal solution for the objective function in (2), where $\widehat{\mathbf{\lambda }}=({\widehat{\lambda }}_1,{\widehat{\lambda }}_2,\dots ,{\widehat{\lambda }}_{\text{p}})$. Then our first main result is:

Theorem 1. Assume that ${\gamma }_{1\text{n}},{\gamma }_{2\text{n}}\to 0$ and let ${\gamma }_{1\text{n}}={\sigma }_{\text{n}}^{\text{p}/2}$ with ${\text{n}}^{1/2}{\sigma }_{\text{n}}^{\text{p}}\to \infty$. Let P denotes the true probability measure, i.e., $\mathbf{\text{P}}\text{g}\left(\mathbf{Y},\mathbf{X}\right)=\mathbf{\text{E}}\left[\text{g}\left(\mathbf{Y},\mathbf{X}\right)\right]$ for any measurable function g(Y, X) with finite first moment. Then it holds

1. with probability one, ${\lim }_{\text{n}\to \infty }\mathbf{\text{P}}\left({\left(\mathbf{Y}-{\widehat{\mathbf{f}}}_{\widehat{\mathbf{\lambda }}}\left(\mathbf{X}\right)\right)}^2\right)=\mathbf{E}{\left(\mathbf{Y}-{\text{f}}_0\left(\mathbf{X}\right)\right)}^2$;

2. $\text{Pr}\left\{{\widehat{\lambda }}_m>0\text{for all}m=1,2,\dots ,r\right\}\to 1$.

Theorem 1 implies that the loss of the estimated prediction function converges to Bayesian risk. Moreover, the ${\widehat{\lambda }}_{\text{m}}$’s associated with important feature variables should be non-zero, i.e., the estimated function does depend on X₁, .., X_r.

The following theorem states that with additional regularity conditions, our method can also identify those unimportant features with probability tending to 1.

Theorem 2. In addition to Theorem 1’s assumptions, assume that ${\text{f}}_0\left({\text{X}}_1,\dots ,{\text{X}}_{\text{r}}\right)$ is twice continuously differentiable and that γ_2n satisfies

$${\gamma }_{2n}\left(n^{1/2}{\sigma }_n^{p/2}{\gamma }_{1n}+{\sigma }_n^{-\min \left(2,p\right)}\right)\to \infty .$$

Then $\text{Pr}\left\{{\widehat{\lambda }}_m=0\text{for m}=\text{r}+1,\dots ,\text{p}\right\}\to 1$.

Theorem 2 implies that the proposed method can estimate the predicted function as if we knew which variables are important in the truth. The proofs of the theorems are given in the Appendix.

4 |. SIMULATION STUDIES

We conducted simulation studies to examine the performance of the proposed method. First, we considered a model with continuous outcomes and a total of ten variables (p = 10) and gradually increased p up to 100. We generated X₁, .., X₁₀ from a multivariate normal distribution with mean zeros and variances 1, where all were independent except that X₇, X₈, X₉, X₁₀ were correlated with $\text{corr}\left({\text{X}}_7,{\text{X}}_8\right)=0.4$, $\text{corr}\left({\text{X}}_7,{\text{X}}_9\right)=-0.3$, $\text{corr}\left({\text{X}}_8,{\text{X}}_9\right)=0.5$ and $\text{corr}\left({\text{X}}_9,{\text{X}}_{10}\right)=0.2$. We treated X₇, X₈, X₉ as important variables and simulated the continuous response Y using the following model

$$Y_i=2X_{i7}{X}_{i8}{X}_{i9}+3.3\exp \left(-X_{i9}\right)+{ϵ}_i.$$

where ${ϵ}_i\sim \text{N}\left(0,1\right)$. We centered the outcome Y to be mean zero. To examine properties of proposed method in higher dimension, we also simulated scenarios with p = 20, 40, 100, where we generated additional independent noise features from the standard normal distribution. We varied training sample size from n = 100, 200 to 400.

For each simulated data, we used the proposed method to learn the prediction function. The choices of tuning parameters followed the description in Section 2.3 and ρ_n was set to be 0.001. We reported the true positive rates, true negative rates, average number of selected variables and prediction errors in our method. In addition, we compared our method with COSSO and LASSO, where COSSO can handle variable selection in nonlinear cases based on SS-ANOVA and LASSO assumes a linear regression model where coefficients estimated to be less than 0.0001 are set to zeros. For tuning parameters of LASSO and COSSO, we used 5-fold cross validations. The comparing performance was based on the mean squared errors in an independent testing sample.

The results based on 500 replicates are summarized in Table 1. From feature selection results columns, we observe that for fixed p, as sample size n becomes larger, the number of true positive rate and true negative rate from our method becomes larger. For fixed sample size n, as p becomes larger, the true positive rate becomes smaller due to additional noise variables. As shown in Table 1, our model can successfully selected all three important variables, with both true positive rate and true negative rate around 95%. Average number of selected variables are also approximate 3. However, COSSO cannot select all important variables, which is reflected from both the true positive rate and average number of selected variables columns. This is because COSSO fits a misspecified model. LASSO does not yield any reasonable variable selection results (not shown here) this setting. Prediction error columns show the mean and median absolute deviation of the prediction errors. Clearly, LASSO gives the worst result since it misspecifies the model most. Our method has the best prediction performance and as the sample size n becomes larger, the prediction errors decrease to Bayes error, which is 1 in this case.

TABLE 1.

Summary of feature selection results and prediction errors

Feature Selection Results								Prediction Errors
		Proposed			COSSO			Proposed	COSSO	LASSO
p	n	TPR	TNR	avg.#	TPR	TNR	avg.#
10	100	88.4%	90.7%	3.3	65.4%	96.1%	2.2	2.743 (0.313)	5.650 (0.556)	5.954 (0.121)
	200	95.4%	95.4%	3.2	82.7%	97.0%	2.7	2.198 (0.205)	5.256 (0.591)	5.689 (0.057)
	400	97.5%	96.3%	3.2	91.5%	98.3%	2.9	1.889 (0.171)	4.708 (0.698)	5.606 (0.038)
20	100	85.0%	94.7%	3.5	49.5%	97.5%	1.9	3.455 (0.396)	6.664 (0.455)	7.056 (0.168)
	200	94.1%	97.5%	3.3	55.6%	96.5%	2.3	2.820 (0.253)	6.312 (0.492)	6.659 (0.077)
	400	97.3%	98.6%	3.2	74.7%	95.3%	3.0	2.464 (0.232)	5.913 (0.570)	6.500 (0.043)
40	100	81.5%	96.7%	3.7	44.3%	98.8%	1.8	2.835 (0.437)	5.693 (0.341)	6.985 (0.434)
	200	93.4%	98.6%	3.3	54.0%	98.7%	2.1	2.101 (0.220)	5.502 (0.263)	6.042 (0.174)
	400	97.1%	99.2%	3.2	58.3%	98.2%	2.4	1.783 (0.127)	5.048 (0.386)	5.672 (0.076)
100	100	74.0%	98.4%	3.8	NA	NA	NA	3.562 (0.593)	NA	30.370 (8.124)
	200	88.0%	99.1%	3.6	49.3%	99.4%	2.1	2.600 (0.312)	5.602 (0.319)	7.597 (0.504)
	400	93.3%	99.4%	3.4	61.7%	99.2%	2.6	2.172 (0.200)	5.506 (0.232)	6.427 (0.200)

Note: TPR: True positive rate; TNR: True negative rate; avg.#: Average number of selected variables. The numbers are the mean of prediction errors and the numbers within parentheses are the median absolute deviations from 500 replicates. “NA”: Results are not available due to failure of the methods.

5 |. APPLICATION TO AIZHEIMER’S DISEASE INITIATIVE STUDY

We applied the proposed method to analyze data from the Alzheimer’s disease neuroimaging initiative (ADNI) study (Toledo, Bjerke, Da, and Landau (2015)). The feature variables included demographic variables (age, gender, race, education level), APoE4 mutation status, clinical variables (functional assessment questionnaire, ADAS-cog11, MMSE) and 7 imaging biomarkers (flurodeoxyglucose, ventricles, hippocampus, whole brain, entorhinal cortex, fusiform gyrus, middle temporal gyrus). Our goal was to assess how well Alzheimer’s disease (AD) biomarkers measured from invasive cerebral spinal fluid (CSF) procedure can be predicted from clinical or biomarker data collected by non-invasive procedures. Thus, we aim to identify important feature variables to predict t-tau and Aβ-42 protein measured from CSF in the ADNI study. There were 535 subjects included in the analyses of t-tau and 542 for Aβ-42. We randomly divided the subjects so that 70% were used for training and 30% were used for testing. We applied the proposed method to learn the prediction rule using the training sample. Since gender and race variables were binary, as mentioned before, we set the individual kernels for these two features in (1) as $\kappa \left(\mathbf{\text{X}},\tilde{\mathbf{X}}\right)=\mathbf{\text{I}}\left(\mathbf{\text{X}}=\tilde{\mathbf{X}}\right)$. For the other feature variables, we used the individual Gaussian kernels with the same bandwidth as described before. We standardized the outcomes and all continuous feature variables. The tuning parameters for γ’s were obtained from 5-fold cross validation in the training sample. For comparison, we also fit LASSO and COSSO with 5-fold cross validation for tuning to the same data, where the coefficients estimated to be less than 0.0001 in LASSO are thresholded to be zeros, and then compared their prediction performance in the testing sample. To obtain a reliable comparison, we repeated the same analysis for 500 randomly divided training sample and test sample.

Figure 2 gives some smooth plots of outcome variables versus feature variables, from which we can get an intuition of the nonlinear relationship between them. Figure 3 gives the frequency of a variable selected. Prediction error by each method together with average and range number of selected variables from 500 replications of random splitting are shown in Table 2. From this table, It is clear that our method yields the smallest prediction errors. Among all 500 replications, for outcome t-tau, the most frequently chosen features in our method were Gender, APoE4, MMSE, ADAS, Ventricles, Hippocampus and Middle temporal gyrus; while for Aβ-42, APoE4, ADAS, MMSE and Hippocampus were highly selected. Also, FAQ is moderately important for outcome Aβ-42, but has no importance for outcome t-tau. In contrast, for outcome t-tau, COSSO highly selected APoE4, MMSE, ADAS, Ventricles and Middle temporal gyrus, but Gender and Hippocampus failed to be selected, which may be a reason of the large prediction error. For outcome Aβ-42, Age, APoE4, ADAS, FAQ, Hippocampus and Middle temporal gyrus were frequently chosen by COSSO. We also noticed that COSSO gave large mean prediction error with high variability but reasonable median prediction error, which indicates the existence of outliers of prediction errors among all 500 replications. LASSO nearly failed to remove any noise variables and selected approximate 15 variables in average for both outcomes. Finally, when applying our method to analyze the whole sample, for t-tau outcome, our method selected 8 feature variables as important and they were gender, APoE4, FDG, ADAS11, MMSE, Ventricles, Hippocampus and MidTemp with prediction error equals to 0.839. For Aβ-42, there were 4 important features including APoE4, ADAS11, MMSE and Hippocampus with prediction error equals to 0.811. The data that support the findings of this study are openly available in http://adni.loni.usc.edu.

FIGURE 2.

smooth plots of outcome variables versus imaging feature variables in ADNI data

FIGURE 3.

Frequency of variables selected in 500 random sample splittings

TABLE 2.

Summary of feature selection results in the application to the ADNI study

	t-tau			Aβ1-42
	Proposed	COSSO	LASSO	Proposed	COSSO	LASSO
Mean of Prediction error	0.870 (0.028)	1.745 (1.869)	0.881 (0.027)	0.830 (0.032)	6.748 (13.176)	0.838 (0.031)
Median of Prediction error	0.869	0.923	0.880	0.829	0.859	0.836
Avg.#	8.08	6.12	14.96	5.72	6.34	14.98
Range #	[4,12]	[1,14]	[14,15]	[3,13]	[1,14]	[14,15]

Note: The numbers are the mean and median of prediction errors from 500 replicates and the numbers within parentheses are their standard deviations. Avg.#: Average number of selected variables. Range #: Range number of selected variables.

6 |. CONCLUSIONS

In this work, we propose a regularized tensor product kernel for sparse nonparametric regression in the presence of nonlinear relationships. The importance of each feature is captured by a non-negative parameter in the kernel function. Our approach is computationally efficient because it can be iteratively computed by optimizing a convex quadratic function from a modified ADMM algorithm. Theoretically, we have shown that our method leads to oracle feature variable selection. The superior performance of the proposed method was demonstrated via simulation studies and a real data application. Note that both our algorithm and theory can be extended to higher dimensional feature variables, or even ultra high dimensional cases.

Here we focus on a regression problem using L₂ loss function. However, our method can be extended to other machine learning approaches with different losses, such as hinge-loss (support vector machine) and boosting. Feature selection can be simultaneously performed when training these machine learning algorithms. We expect that the same iterative algorithm applies but the step of updating regularization parameters may be different, although it remains to be a convex optimization problem with linear constraints.

Finally, our method can also be generalized to feature selection problems when feature variables collected from different domains (imaging, genomics, clinical biomarkers), which is common in integrative data analysis. By accounting for the hierarchical structure of multiple domains, one possibility is to construct a hierarchical tensor product kernel with regularization parameters for both domains and features within each domain. In this way, we can perform domain selection and feature selection at the same time. We will consider such extensions in a future work.

Abbreviations:

ANA

anti-nuclear antibodies

APC

antigen-presenting cells

IRF

interferon regulatory factor

APPENDIX

Proof of Theorems

Before proving the theorems, we need two lemmas. The proof of them can be found in supporting information file. The first lemma shows that the proposed kernel function satisfies positive-definite condition.

Lemma A.1. For any positive constants ${\lambda }_1,\dots ,{\lambda }_{\text{r}}>0$,

$${\kappa }_{\mathbf{\lambda },{\sigma }_n}\left(\mathbf{X},\tilde{\mathbf{X}}\right)=\prod _{m=1}^p\left(1+{\lambda }_m{\kappa }_n\left(X_m,{\tilde{X}}_m\right)\right),$$

is a kernel satisfying semi positive-definiteness condition.

The next lemma shows that when σ_n goes to zero, the closure of the reproducing kernel Hilbert space generated by the tensor product kernel contains the true function ${\text{f}}_0\left({\text{X}}_1,\dots ,{\text{X}}_{\text{r}}\right)$ if ${\lambda }_{\text{m}}>0$ for $\text{m}\le \text{r}$. We define $\text{d}\left({\text{f}}_0,{\mathcal{H}}_{\mathbf{\lambda },{\sigma }_{\text{n}}}\right)$ as the L₂(P) distance between f₀ and the reproducing kernel Hilbert space.

Lemma A.2. Assume that ${\sigma }_{\text{n}}\to 0$. For any $\mathbf{\lambda }=\left({\lambda }_1,{\lambda }_2\dots ,{\lambda }_{\text{p}}\right),{\lambda }_{\text{m}}\ge 0,\text{m}\le \text{p}$,

1. If ${\lambda }_{\text{m}}\ne 0$ for $\text{m}\le \text{r}$, then $\text{d}\left({\text{f}}_0\left({\mathbf{X}}_1,{\mathbf{X}}_2,\dots ,{\mathbf{X}}_{\text{r}}\right),{\mathcal{H}}_{\mathbf{\lambda },{\sigma }_{\text{n}}}\right)\to 0$. In fact, the closure of lim sup_n ${\mathcal{H}}_{\mathbf{\lambda },{\sigma }_{\text{n}}}$ contains any L₂-integrable function that only depends on (X₁, …, X_r).

2. If for some m ≤ r, ${\lambda }_{\text{m}}=0$, then $\mathrm{lim\; inf\; d}\left({\text{f}}_0\left({\mathbf{X}}_1,{\mathbf{X}}_2,\dots ,{\mathbf{X}}_{\text{r}}\right),{\mathcal{H}}_{\mathbf{\lambda },{\sigma }_n}\right)>0$.

Before proving two theorems, recall that P denotes the true probability measure; P_n denotes the empirical measure from n observations.

Proof of Theorem 1. By Lemma A.2, for any ${\mathbf{\lambda }}_0=\left({\mathbf{\lambda }}_{01},\mathbf{0}\right)$, where ${\mathbf{\lambda }}_{01}=\left({\lambda }_{011},{\lambda }_{012},\cdots ,{\lambda }_{01\text{r}}\right)$, and ${\lambda }_{01\text{m}}>0$ for any m ≤ r, there exits ${\widetilde{\mathbf{\text{f}}}}_{{\mathbf{\lambda }}_0}\in {\mathcal{H}}_{{\mathbf{\lambda }}_0,{\sigma }_{\text{n}}}\text{s.t. d}\left({\mathbf{\text{f}}}_0,{\widetilde{\mathbf{\text{f}}}}_{{\mathbf{\lambda }}_0}\right)={\Vert {\widetilde{\mathbf{\text{f}}}}_{{\mathbf{\lambda }}_0}-{\mathbf{\text{f}}}_0\Vert }_{{\text{L}}_2}\to 0$. Since $\left(\widehat{\mathbf{\text{λ}}},{\widehat{\mathbf{\text{f}}}}_{\widehat{\mathbf{\text{λ}}}}\right)$ is the optimal solution for objective function (2), we have

$${\mathbf{P}}_{n}l\left(Y,{\widehat{f}}_{\widehat{\mathbf{\lambda }}}\left(\mathbf{\text{X}}\right)\right)+{\gamma }_{1n}{\Vert \widehat{f}\Vert }_{{\mathcal{H}}_{\widehat{\mathbf{\lambda }},{\sigma }_n}}^2+{\gamma }_{2n}{\Vert \widehat{\mathbf{\lambda }}\Vert }_0\le {\mathbf{P}}_{n}l\left(Y,{\tilde{f}}_{{\mathbf{\lambda }}_0}\left(\mathbf{X}\right)\right)+{\gamma }_{1n}{\Vert \tilde{f}\Vert }_{{\mathcal{H}}_{{\mathbf{\lambda }}_0,{\sigma }_n}}^2+{\gamma }_{2n}{\Vert {\mathbf{\lambda }}_0\Vert }_0.$$

That is,

$$\begin{array}{l}\left({\mathbf{P}}_{\text{n}}-\mathbf{P}\right)l\left(\mathbf{\text{Y}},{\widehat{\mathbf{f}}}_{\widehat{\mathbf{\lambda }}}\left(\mathbf{X}\right)\right)+{\gamma }_{1n}{\Vert \widehat{\mathbf{f}}\Vert }_{{\mathcal{H}}_{\widehat{\mathbf{\lambda }},{\sigma }_{\text{n}}}}^2+\mathbf{P}l\left(\mathbf{\text{Y}},{\widehat{\mathbf{f}}}_{\widehat{\mathbf{\lambda }}}\left(\mathbf{X}\right)\right)+{\gamma }_{2\text{n}}{\Vert \widehat{\mathbf{\lambda }}\Vert }_0\\ \le \left({\mathbf{P}}_{\text{n}}-\mathbf{P}\right)l\left(\mathbf{\text{Y}},{\tilde{\mathbf{f}}}_{{\lambda }_0}\left(\mathbf{X}\right)\right)+{\gamma }_{1\text{n}}{\Vert \tilde{\mathbf{f}}\Vert }_{{\mathcal{H}}_{{\mathbf{\lambda }}_0,{\sigma }_{\text{n}}}}^2+\mathbf{P}l\left(\mathbf{\text{Y}},{\tilde{\mathbf{f}}}_{{\mathbf{\lambda }}_0}\left(\mathbf{X}\right)\right)+{\gamma }_{2\text{n}}{\Vert {\mathbf{\lambda }}_0\Vert }_0.\end{array}$$

Following some similar arguments to Theorem 3.1 in Steinwart and Scovel (2007), since ${\Vert \widehat{\mathbf{f}}\Vert }_{{\mathcal{H}}_{\widehat{\mathbf{\lambda }},{\sigma }_{\text{n}}}}\le \mathbf{\text{O}}\left({\gamma }_{1\text{n}}^{-1/2}\right)$, the uniform covering numbers of this bounded set in the reproducing kernel Hilbert space can be calculated as follows.

First, the entropy number (van der Vaart and Wellner (1996)) for the unit ball in ${\mathcal{H}}_{\widehat{\mathbf{\lambda }},{\sigma }_{\text{n}}}$, denoted by ${\mathcal{O}}_{\text{n}}$, satisfies

$$\log \mathcal{N}\left(ϵ,{\mathcal{O}}_{\text{n}},{\Vert \cdot \Vert }_{\infty }\right)\le {\text{c}}_1\left(\text{v},\text{p}\right){\sigma }_{\text{n}}^{-\left(1-\text{v}/4\right)\text{p}}{ϵ}^{-\text{v}},$$

where 0 < v < 2 and c₁ (v, p) is a constant that depends only on v and p. Then it gives

$$\log {\mathcal{N}}_{\left[\right]}\left(ϵ,{\mathcal{O}}_{\text{n}},{\Vert \cdot \Vert }_{{\text{L}}^4\left(\text{P}\right)}\right)\le {\text{c}}_1\left(\text{v},\text{p}\right){\sigma }_{\text{n}}^{-\left(1-\text{v}/4\right)\text{p}}{ϵ}^{-\text{v}}.$$

Thus, we obtain

$$\log {\mathcal{N}}_{\left[\right]}(ϵ,\{\widehat{f}:\widehat{f}\in {\mathcal{H}}_{\widehat{\mathbf{\lambda }},{\sigma }_n},{\Vert \widehat{f}\Vert }_{{\mathcal{H}}_{\widehat{\mathbf{\lambda }},{\sigma }_n}}\le O({\gamma }_{1n}^{-1/2})\},{\Vert \cdot \Vert }_{L^4\left(P\right)})\le c_1\left(v,p\right){\sigma }_n{}^{-\left(1-v/4\right)p}{ϵ}^{-v}{\gamma }_{1n}^{-v/2}.$$

Note

$$\begin{array}{l}{\Vert l\left(\mathbf{Y},{\widehat{\mathbf{f}}}_1\right)-l\left(\mathbf{Y},{\widehat{\mathbf{f}}}_2\right)\Vert }_{{\text{L}}_2\left(\text{P}\right)}={\Vert {\left(\mathbf{Y}-{\widehat{\mathbf{f}}}_1\right)}^2-{\left(\mathbf{Y}-{\widehat{\mathbf{f}}}_2\right)}^2\Vert }_{{\text{L}}_2\left(\text{P}\right)}\\ ={\left(\mathbf{E}{\left(2\mathbf{Y}-{\widehat{\mathbf{f}}}_1-{\widehat{\mathbf{f}}}_2\right)}^2{\left({\widehat{\mathbf{f}}}_2-{\widehat{\mathbf{f}}}_1\right)}^2\right)}^{\frac{1}{2}}\\ \le {\left(\mathbf{E}{\left({\widehat{\mathbf{f}}}_2-{\widehat{\mathbf{f}}}_1\right)}^4\right)}^{\frac{1}{2}}{\left(\mathbf{E}{\left(2\mathbf{Y}-{\widehat{\mathbf{f}}}_1-{\widehat{\mathbf{f}}}_2\right)}^4\right)}^{\frac{1}{2}},\end{array}$$

where ${\left(\mathbf{E}{\left(2\mathbf{Y}-{\widehat{\mathbf{f}}}_1-{\widehat{\mathbf{f}}}_2\right)}^4\right)}^{\frac{1}{2}}\le 3^{\frac{3}{2}}{\left(16\mathbf{E}\left({\mathbf{Y}}^4\right)+\mathbf{E}\left({\widehat{\mathbf{f}}}_1^4\right)+\mathbf{E}\left({\widehat{\mathbf{f}}}_2^4\right)\right)}^{\frac{1}{2}}\le {\text{c}}^{\prime }$, ${\text{c}}^{\prime }$ is a constant. Thus, we can obtain ${\Vert l\left(\mathbf{Y},{\widehat{\mathbf{f}}}_1\right)-l\left(\mathbf{Y},{\widehat{f}}_2\right)\Vert }_{{\text{L}}_2\left(\text{P}\right)}\le {\text{c}}^{\prime }{\left({\Vert {\widehat{\mathbf{f}}}_1-{\widehat{\mathbf{f}}}_2\Vert }_{{\text{L}}_2\left(\text{P}\right)}\right)}^2$, which yields

$$\log {\mathcal{N}}_{\left[\right]}(ϵ,\{l(\mathbf{Y},\widehat{f}):\widehat{f}\in {\mathcal{H}}_{\widehat{\mathbf{\lambda }},{\sigma }_n},{\Vert \widehat{f}\Vert }_{{\mathcal{H}}_{\widehat{\mathbf{\lambda }},{\sigma }_n}}\le O({\gamma }_{1n}^{-1/2})\},{\Vert \cdot \Vert }_{L^4\left(P\right)})\le c_2(v,p,c^{\prime }){\sigma }_n{}^{-\left(1-v/4\right)p}{ϵ}^{-v/2}{\gamma }_{1n}^{-v/2}.$$

Let $c^{″}$ denotes a constant. The above result implies

$$\begin{array}{l}\mathbf{E}\left[\begin{array}{cc}\underset{\widehat{\mathbf{f}}\in {\mathcal{H}}_{\widehat{\mathbf{\lambda }},{\sigma }_{\text{n}}},{\Vert \widehat{\mathbf{f}}\Vert }_{{\mathcal{H}}_{\widehat{\mathbf{\lambda }},{\sigma }_{\text{n}}}}\le \text{O}\left({\gamma }_{1\text{n}}^{-1/2}\right)}{\sup }& \left|\left({\mathbf{\text{P}}}_{\text{n}}-\mathbf{\text{P}}\right)l\left(\mathbf{Y},\mathbf{f}\right)\right|\end{array}\right]\\ \le {\text{c}}_3{\mathbf{\text{n}}}^{-1/2}\underset{0}{\overset{{\text{c}}^{″}{\gamma }_{1\text{n}}^{-1}}{\int }}\sqrt{1+\log {\mathcal{N}}_{\left[\right]}(ϵ,\{l(\mathbf{Y},\widehat{\mathbf{f}}):\widehat{\mathbf{f}}\in {\mathcal{H}}_{\widehat{\mathbf{\lambda }},{\sigma }_{\text{n}}},{\Vert \widehat{\mathbf{f}}\Vert }_{{\mathcal{H}}_{\widehat{\mathbf{\lambda }},{\sigma }_{\text{n}}}}\le \mathbf{O}({\gamma }_{1\text{n}}^{-1/2})\},{\Vert \cdot \Vert }_{{\text{L}}^4\left(\text{P}\right)})\mathbf{d}ϵ}\\ \le \text{k}\left(\text{v},\text{p}\right){\text{n}}^{-1/2}{\sigma }_{\text{n}}^{-\left(1/2-\text{v}/8\right)\text{p}}{\gamma }_{1\text{n}}^{-1}.\end{array}$$

In other words, $\left({\mathbf{\text{P}}}_{\text{n}}-\mathbf{\text{P}}\right)l(\mathbf{Y},{\widehat{\mathbf{f}}}_{\widehat{\mathbf{\lambda }}}\left(\mathbf{X}\right))=\text{O}({\text{n}}^{-1/2}{\sigma }_{\text{n}}^{-\text{p}/2}{\gamma }_{1\text{n}}^{-1})$ by the choice of γ_1n in Theorem 1. Similarly,

$$\left({\mathbf{\text{P}}}_n-\mathbf{\text{P}}\right)l(\mathbf{Y},{\tilde{\mathbf{f}}}_{{\mathbf{\lambda }}_0}\left(\mathbf{X}\right))=O(n^{-1/2}{\sigma }_n^{-\text{p}/2}{\gamma }_{1n}^{-1}).$$

Therefore, let $\text{n}\to \infty$ then since ${\gamma }_{1\text{n}}\to 0$, we have

$$\underset{\text{n}\to \infty }{\mathrm{lim\; sup}}\left(\mathbf{\text{P}}l\left(\mathbf{Y},{\widehat{\mathbf{f}}}_{\widehat{\mathbf{\lambda }}}\left(\mathbf{X}\right)\right)+{\gamma }_{2\text{n}}{\Vert \widehat{\mathbf{\lambda }}\Vert }_0\right)\le \underset{\text{n}\to \infty }{\mathrm{lim\; sup}}\left(\mathbf{\text{P}}l\left(\mathbf{Y},{\tilde{\mathbf{f}}}_{{\mathbf{\lambda }}_0}\left(\mathbf{X}\right)\right)+\text{O}\left({\text{n}}^{-1/2}{\sigma }_{\text{n}}^{-\text{p}/2}{\gamma }_{1\text{n}}^{-1}\right)+{\gamma }_{2n}{\Vert {\mathbf{\lambda }}_0\Vert }_0\right).$$

Under the assumption that ${\text{n}}^{-1/2}{\sigma }_{\text{n}}^{-\text{p}}\to 0,{\gamma }_{2\text{n}}\to 0$, we have

$$\underset{n\to \infty }{\mathrm{lim\; sup}}\mathbf{P}l(Y,{\widehat{f}}_{\widehat{\mathbf{\lambda }}}\left(\mathbf{X}\right))\le \underset{n\to \infty }{\mathrm{lim\; sup}}\mathbf{P}l(Y,{\tilde{f}}_{{\mathbf{\lambda }}_0}\left(\mathbf{X}\right))=El(Y,f_0\left(\mathbf{X}\right)).$$

Since $\mathbf{E}l\left(\mathbf{Y},f_0\left(\mathbf{X}\right)\right)\le {\mathrm{lim\; sup}}_{\text{n}\to \infty }\mathbf{P}l\left(\mathbf{Y},{\widehat{\mathbf{f}}}_{\widehat{\mathbf{\lambda }}}\left(\mathbf{X}\right)\right)$, we obtain

$$\underset{n\to \infty }{\lim }\mathbf{P}l\left(Y,{\widehat{f}}_{\widehat{\mathbf{\lambda }}}\right)=El\left(Y,f_0\left(\mathbf{X}\right)\right).$$

Together with Lemma A.2, this also gives the result that for any $\text{m}\le \text{r},{\widehat{\lambda }}_{\text{m}}>0$. Otherwise, if there is some $\text{m}\le \text{r},{\widehat{\lambda }}_{\text{m}}=0,$, by Lemma A.2 (ii), it always holds that ${\lim }_{\text{n}\to \infty }\mathbf{P}l\left(\mathbf{Y},{\widehat{\mathbf{f}}}_{\widehat{\mathbf{\lambda }}}\right)-\mathbf{\text{EI}}\left(\mathbf{\text{Y}},{\mathbf{f}}_0\left(\mathbf{X}\right)\right)\ge \mathrm{lim\; inf}\mathbf{d}\left({\mathbf{\text{f}}}_0\left({\mathbf{X}}_1,{\mathbf{X}}_2,\cdots ,{\mathbf{X}}_{\text{r}}\right),{\mathcal{H}}_{\mathbf{\lambda },{\sigma }_{\text{n}}}\right)>0$. We then have a contradiction. □

Proof of Theorem 2. For this proof, we particularly choose ${\tilde{\mathbf{f}}}_{{\mathbf{\lambda }}_0}$ to be the Gaussian-kernel convolution of f₀ in the space of (X₁, …, X_r), where the kernel is given ${\prod }_{\text{m}=1}^{\text{r}}{\text{k}}_{\text{n}}\left({\mathbf{X}}_{\text{m}},{\tilde{\mathbf{X}}}_{\text{m}}\right)$. Clearly, ${\tilde{\mathbf{f}}}_{{\mathbf{\lambda }}_0}$ belongs to ${\mathcal{H}}_{\mathbf{\lambda },{\sigma }_{\text{n}}}$ and since f₀(X) is twice continuous differentiable,

$$d({\tilde{f}}_{{\mathbf{\lambda }}_0},f_0)=O({\sigma }_n^2).$$

With the same arguments in the proof of Theorem 1, we have

$$\mathbf{\text{P}}l(\mathbf{Y},{\widehat{\mathbf{f}}}_{\widehat{\mathbf{\lambda }}}(\mathbf{X}))+{\gamma }_{2\text{n}}{\Vert \widehat{\mathbf{\lambda }}\Vert }_0\le \mathbf{P}l(\mathbf{Y},{\tilde{\mathbf{f}}}_{{\mathbf{\lambda }}_0}(\mathbf{X}))+\text{O}({\text{n}}^{-1/2}{\sigma }_{\text{n}}^{-\text{p}/2}{\gamma }_{1\text{n}}^{-1})+\text{O}({\gamma }_{1\text{n}})+{\gamma }_{2\text{n}}{\mathbf{\Vert }{\mathbf{\lambda }}_0\mathbf{\Vert }}_0.$$

Thus,

$$\mathbf{\text{P}}l(\mathbf{Y},{\widehat{\mathbf{f}}}_{\widehat{\mathbf{\lambda }}}(\mathbf{X}))+{\gamma }_{2\text{n}}{\Vert \widehat{\mathbf{\lambda }}\Vert }_0\le \mathbf{E}l(\mathbf{Y},{\mathbf{f}}_0(\mathbf{X}))+\text{O}({\sigma }_{\text{n}}^{\text{min}(2,\text{p})})+\text{O}({\text{n}}^{-1/2}{\sigma }_{\text{n}}^{-\text{p}/2}{\gamma }_{1\text{n}}^{-1})+{\gamma }_{2\text{n}}{\Vert {\mathbf{\lambda }}_0\Vert }_0.$$

Since it always holds that $\mathbf{E}l(\mathbf{Y},{\mathbf{f}}_0(\mathbf{X}))\le \mathbf{P}l(\mathbf{Y},{\widehat{\mathbf{f}}}_{\widehat{\mathbf{\lambda }}}(\mathbf{X}))$,

$${\gamma }_{2\text{n}}{\Vert \widehat{\mathbf{\lambda }}\Vert }_0\le \text{O}({\sigma }_{\text{n}}^{\text{min}(2,\text{p})})+\text{O}({\text{n}}^{-1/2}{\sigma }_{\text{n}}^{-\text{p}/2}{\gamma }_{1\text{n}}^{-1})+{\gamma }_{2\text{n}}{\Vert {\mathbf{\lambda }}_0\Vert }_0.$$

Thus, by dividing γ_2n on both sides, we have

$${\Vert \widehat{\mathbf{\lambda }}\Vert }_0\le \text{O}({\gamma }_{2\text{n}}^{-1}{\sigma }_{\text{n}}^{\text{min}(2,\text{p})})+\text{O}({\gamma }_{1\text{n}}^{-1}{\gamma }_{2\text{n}}^{-1}{\text{n}}^{-1/2}{\sigma }_{\text{n}}^{-\text{p}/2})+\text{r}.$$

Under the assumptions in Theorem 2, we conclude that with probability tending to one, the number of the non-zero components for $\widehat{\mathbf{\lambda }}$ can not be larger than r. However, Theorem 1 implies that this number should be at least r. We thus obtain Theorem 2. □