Optimal and Safe Estimation for High-Dimensional Semi-Supervised Learning

Abstract

We consider the estimation problem in high-dimensional semi-supervised learning. Our goal is to investigate when and how the unlabeled data can be exploited to improve the estimation of the regression parameters of linear model in light of the fact that such linear models may be misspecified in data analysis. We first establish the minimax lower bound for parameter estimation in the semi-supervised setting, and show that this lower bound cannot be achieved by supervised estimators using the labeled data only. We propose an optimal semi-supervised estimator that can attain this lower bound and therefore improves the supervised estimators, provided that the conditional mean function can be consistently estimated with a proper rate. We further propose a safe semi-supervised estimator. We view it safe, because this estimator is always at least as good as the supervised estimators. We also extend our idea to the aggregation of multiple semi-supervised estimators caused by different misspecifications of the conditional mean function. Extensive numerical simulations and a real data analysis are conducted to illustrate our theoretical results.

1. Introduction

Semi-supervised learning is an emerging research area in statistics and machine learning (Zhu, 2005; Chapelle et al., 2006), and can have a great potential in electronic health records (EHR) based studies for clinical research. In these types of studies, one major challenge is the lack of gold-standard health outcomes or phenotypes (Kohane, 2011). The validated phenotypes are often obtained by manual chart reviews that are prohibitively expensive (Liao et al., 2010); therefore, only can a very small subset of patients be annotated by experts in reality. For the rest of the patients, their covariate information, often high-dimensional (Weisenthal et al., 2018; Gensheimer et al., 2019; Abdullah et al., 2020), is only available.

Developing efficient statistical methods to analyze such data is a timely and important problem. Let $Y$ denote the outcome variable and $X$ the $p$-dimensional covariates. In addition to $n$ independent and identically distributed (i.i.d.) samples $(Y_1,X_1),\dots (Y_n,X_n)\sim (Y,X)$, we also observe $N$ i.i.d. data consisting of only covariates, $X_{n+1},\dots ,X_{N+n}\sim X$. Following the convention, the former is referred to as labeled data and the latter is called unlabeled data. We also denote $\mathit{Y}={(Y_1,\dots ,Y_n)}^T\in {ℝ}^n\text{and}\mathit{X}={(X_1,\dots ,X_n)}^T\in {ℝ}^{n\text{×}p}$ the outcomes and covariates from the labeled data, and $\tilde{\mathit{X}}={(X_1,\dots ,X_{N+n})}^T\in {ℝ}^{(N+n)\times p}$ the covariates from both. In this work, we focus on high-dimensional regression problems; namely, $p$ can be much larger than $n$. The size of the unlabeled data $N$ is allowed but not required to be larger than $n$.

We consider the so-called assumption lean regression framework (Buja et al., 2019; Berk et al., 2019),

$$Y=f(X)+ϵ,$$ (1.1)

where $f(X)=E(Y\mid X)$ is the unknown conditional mean function, $ϵ$ is the random error independent of $X\in {ℝ}^p$ with $𝔼(ϵ)=0,𝔼({ϵ}^2)={\sigma }^2$, and ${\sigma }^2$ is an unknown parameter. We consider the random design and assume that $X$ and $Y$ are centered with $𝔼(X)=0$ and $𝔼(f(X))=0$. On one hand, we would like to put as fewer assumptions as possible on $f(X)$ to enable model flexibility. On the other hand, for the purpose of interpretability, we often fit simple parametric models such as linear regression to explain the association between $Y$ and $X$. To meet both ends, we consider linear regression as a working model where the true data generating process follows (1.1). Since $𝔼[{(Y-X^T\mathit{\theta })}^2]=𝔼[{(f(X)-X^T\mathit{\theta })}^2]+{\sigma }^2$, the regression coefficients in a linear model correspond to the $L_2(\mathbb{P})$ projection of $f(X)$ onto the linear space spanned by $X$, i.e.,

$${\mathit{\theta }}^{*}=\text{arg}\underset{\theta \in {ℝ}^p}{\text{min}}𝔼[{(f(X)-X^T\mathit{\theta })}^2]$$

that delineates the linear dependence between $Y$ and $X$. We do not include intercept in ${\mathit{\theta }}^{*}$ simply because $𝔼(X)=0$ and $𝔼(f(X))=0$. Our goal here is to estimate the high dimensional parameter ${\mathit{\theta }}^{*}$.

In the supervised setting with $n$ labeled observations, a number of penalized estimators have been proposed to estimate ${\mathit{\theta }}^{*}$, such as lasso (Tibshirani, 1996) and Dantzig selector (Candes and Tao, 2007). While significant progress has been made towards understanding the estimation in the fully supervised setting, there is very limited research in the semi-supervised setting. It is important to observe that under (1.1), since the linear regression is the working model, the covariate $X$ is no longer the ancillary statistic for the regression parameter ${\mathit{\theta }}^{*}$. Therefore, the covariate $X$ in the unlabeled data, usually with a much bigger sample size than the labeled data, is informative and may be beneficial for estimating ${\mathit{\theta }}^{*}$.

Our first contribution is to establish the minimax lower bound for estimating ${\mathit{\theta }}^{*}$ in the semi-supervised setting. In particular, to derive this lower bound, we assume $f(\cdot )$ is unknown but belongs to some specific model class, such that methods for estimating $f(\cdot )$ are available in the existing literature. Based on the lower bound, when $N$ is sufficiently large, the error term due to the model misspecification or equivalently the nonlinearity of $f(\cdot )$ becomes negligible. This reveals one potential benefit from using the unlabeled data in that the estimation of ${\mathit{\theta }}^{*}$ can be more robust to the model misspecification. Moreover, we show that the fully supervised estimators (e.g., lasso and Dantzig selector) do not attain this lower bound. Our second contribution is to propose a new semi-supervised estimator that achieves this lower bound under some conditions. In Theorem 3.2, we show that the rate of our estimator depends on whether the unknown $f(\cdot )$ can be consistently estimated. When $f(\cdot )$ belongs to some specific model class so that it can be consistently estimated with a proper rate, the proposed estimator attains the minimax lower bound up to some logarithmic factor, and therefore improves the rate of the supervised estimators. When $f(\cdot )$ is misspecified, however, the rate of our semi-supervised estimator becomes sub-optimal and may be even slower than the supervised estimators. Our third contribution is to develop a general two-step refitting procedure that further improves the above semi-supervised estimator. The resulting estimator is guaranteed to be no worse than the supervised estimators no matter $f(\cdot )$ is misspecified or not, and remains minimax rate-optimal when $f(\cdot )$ belongs to some specific model class so that can be consistently estimated with a proper rate, hence it provides a safe use of the unlabeled data. Therefore, we call it the safe semi-supervised estimator. In empirical studies one may encounter the situation that, while all misspecified, different estimates of $f(\cdot )$ are available. We further extend the idea of creating safe semi-supervised estimator to the aggregation of multiple semi-supervised estimators under such a situation. The convergence rate of the aggregated estimator is guaranteed to be no worse than any of the un-aggregated semi-supervised estimators. Overall, our goal is to exploit both safe and efficient use of the unlabeled data by developing semi-supervised estimators of ${\mathit{\theta }}^{*}$ such that their convergence rates are faster, or at least no slower, than the standard supervised estimators (e.g., lasso and Dantzig selector).

1.1. Related work

In computer science, a large number of classification algorithms have been developed under semi-supervised setting, which mainly focus on data with discrete labels; see Zhu (2005); Chapelle et al. (2009) for some surveys. Common assumptions such as manifold assumption and cluster assumption were made in the literature in order to obtain fast rate of convergence in classification (Rigollet, 2006). In non-parametric regression problem, Wasserman and Lafferty (2008) showed that unlabeled data do not always help to improve the rate of the mean squared error, but with semi-supervised smoothness assumption the estimator with faster rate could be developed.

More recently, Van Engelen and Hoos (2020) and Yang et al. (2021) provided comprehensive surveys on many up-to-date developments in semi-supervised learning, especially with deep neural networks. For example, the methods named as Mixmatch (Berthelot et al., 2019) and Fixmatch (Sohn et al., 2020) leverage unlabeled data through pseudo-labeling and consistency regularization to assist the prediction model. The self-training method, which trains a model to fit pseudo-labels predicted by previously learned models, has also been developed for semi-supervised learning (Xie et al., 2020; Chen et al., 2020; Wei et al., 2020). The performance of many modern semi-supervised learning algorithms in some image classification tasks is compared and discussed by Oliver et al. (2018).

There are also progress considering how to make use of the unlabeled data to obtain an estimator with a smaller asymptotic variance, when the dimension $p$ is fixed and small. For example, Zhang et al. (2019); Azriel et al. (2016) and Chakrabortty and Cai (2018) investigated how to incorporate the unlabeled data to improve the estimation efficiency for the population mean $𝔼(Y)$ and regression coefficients in a working linear regression.

With high-dimensional data, Alquier and Hebiri (2012) proposed a transductive version of lasso and Dantzig selector in the semi-supervised setting. They showed that the transductive estimators have the same rate as the supervised ones. More recently, Bellec et al. (2018a) proposed a modified lasso estimator and showed that the excess risk of their estimator in prediction has the same rate of convergence as the supervised lasso estimator. These existing theoretical results neither confirm nor deny the existence of estimators with improved statistical rate when unlabeled data are available. To the best of our knowledge, it remains an open problem of when and how one can develop a semi-supervised estimator with improved statistical rate by exploiting the available unlabeled data. We bridge this gap by showing the minimax lower bound and proposing new semi-supervised estimators.

1.2. Organization of the paper

The rest of this paper is organized as follows. In Section 2, we give the minimax lower bound for semi-supervised estimation. In Section 3, we introduce the optimal semi-supervised estimator and its corresponding upper bound. In Section 4 we devote ourselves into the safe semi-supervised estimator, where we propose a two-step procedure regardless of the quality of the estimated conditional mean model. In Section 5 we extend the idea to aggregation of multiple semi-supervised estimators caused by different misspecifications of the conditional mean model. Numerical experiments and a real data application are in Sections 6 and 7, respectively. All the technical proofs are contained in the Supplement.

1.3. Notations

Let $P_{X,Y}$ and $P_X$ denote the joint distribution of $(X,Y)$ and the marginal distribution of $X$, respectively. For $v={(v_1,\dots ,v_p)}^T\in {ℝ}^p$, we define $\Vert v{\Vert }_0=|\text{supp}(v)|$ where $\text{supp}(v)=\{i:v_i\ne 0\}$ and $|A|$ is the cardinality of a set $A,\Vert v{\Vert }_q={({\sum }_{i=1}^p{|v_i|}^q)}^{1/q}$ for $1\le q<\infty$, and $\Vert v{\Vert }_{\infty }={\text{max}}_{1\le i\le p}|v_i|$. Denote $v^{\otimes 2}=v{v}^T$. For a matrix $M=[M_{ij}],M_{i\cdot }$ and $M_{\cdot j}$ denote the $i$-th row and $j$-th column respectively. Define $\Vert M{\Vert }_{\text{max}}={\text{max}}_{ij}|M_{ij}|,M{\Vert }_1={\text{max}}_j{\sum }_i|M_{ij}|,\Vert M{\Vert }_{\infty }={\text{max}}_i{\sum }_j|M_{ij}|$. If the matrix $\mathbf{M}$ is symmetric, then ${\text{Λ}}_{\text{min}}(M)$ and ${\text{Λ}}_{\text{max}}(M)$ are the minimal and maximal eigenvalues of $\mathbf{M}$. We denote ${\mathbf{I}}_p$ the $p\times p$ identity matrix. For $S\subseteq \{1,\dots ,p\}$, let $v_S=\{v_k:k\in S\}$ and $S^c$ be the complement of $S$. For matrix $\mathit{X}\in {ℝ}^{n\times p}$ and index set $D\subseteq \{1,\dots ,n\},{\mathit{X}}_D={\{X_i:i\in D\}}^T\in {ℝ}^{|D|\times p}$. For a function $f$, let $\Vert f{\Vert }_2=\sqrt{𝔼[f{(X)}^2]}$ denote the $L_2(\mathbb{P})$ norm of $f$.

For two positive sequences $a_n$ and $b_n$, we write $a_n\asymp b_n$ if $C\le a_n/b_n\le C^{\prime }$ for some $C,C^{\prime }>0$. Similarly, we use $a\lesssim b$ to denote $a\le Cb$ for some constant $C>0$. Given $a,b\in ℝ$, let $a\vee b$ and $a\wedge b$ denote the maximum and minimum of $a$ and $b$.

2. Minimax Lower Bound for Semi-Supervised Estimation

The semi-supervised learning setting refers to that we observe $n$ i.i.d. copies of $(Y,X)$ and additional $N$ i.i.d. copies of $X$, where the distributions of $X$ in both labeled and unlabeled data are the same. In Theorem 3.2 presented in Section 3, we will rigorously show that the unknown conditional mean function $f(\cdot )$ plays an important role in assessing the optimality of semi-supervised estimators. However, the correct specification and consistent estimation of $f(\cdot )$ under high dimensionality is by no means a trivial problem. In the literature, methods for consistently estimating $f(\cdot )$ with a proper rate are only available when $f(\cdot )$ belongs to some specific model class. In that regard, when analyzing the minimax lower bound here, we assume that $f(\cdot )$ belongs to one of the following two model classes.

We first define the pairwise interaction model ${ℱ}_{\text{pairwise}}=\{{\sum }_{j=1}^p{\gamma }_j{X}_j+{\sum }_{1\le j\le k\le p}{\gamma }_{jk}{X}_j{X}_k\}$, where $\gamma =({\gamma }_1,\dots ,{\gamma }_p,{\gamma }_{11},{\gamma }_{12},\dots ,{\gamma }_{pp})\in {ℝ}^{p+p(p+1)/2}$ are unknown parameters satisfying $\Vert \gamma {\Vert }_0\le \overline{s}$. In practice, the conditional mean function $f(X)$ is usually nonlinear in $X$. In ${ℱ}_{\text{pairwise}}$, we account for the nonlinearity by incorporating the quadratic terms and the pairwise interactions. To mitigate the model complexity, the parameter $\gamma$ is assumed to be $\bar{s}$-sparse (Zhao et al., 2016). We refer to Supplement S.4 for further discussions. Thus, we define the class of joint distributions of $(X,Y)$ as

$$\begin{gathered} {𝒫}_{\mathrm{\Phi },\sigma }^{\text{pairwise}}=\{P_{X,Y}|Y=f(X)+ϵ,f(\cdot )\in {ℱ}_{\text{pairwise}},{\Vert {\mathit{\theta }}^{*}\Vert }_0\le s,\text{Var}(ϵ)={\sigma }^2, \\ 𝔼{(f(X)-X^T{\mathit{\theta }}^{*})}^2\le {\mathrm{\Phi }}^2,\text{and}{P}_X\in {𝒫}_X\}, \end{gathered}$$

where ${\mathit{\theta }}^{*}$ implicitly depends on the distribution $P_{X,Y}$, the parameter $s$ controls the sparsity of ${\mathit{\theta }}^{*},{𝒫}_X=\{P_X|𝔼[X]=0,\text{Var}(X_j)=1$ and ${\text{Λ}}_{\text{min}}(\text{Cov}(X))\ge C_{\text{min}}>0\}$ with some constant $C_{\text{min}}$. For notational simplicity, we write $𝔼(\cdot )$ for ${𝔼}_{P_{X,Y}}(\cdot )$. We note that, ${𝒫}_{\mathrm{\Phi },\sigma }^{\text{pairwise}}$ is indexed by two non-negative parameters ${\mathrm{\Phi }}^2$ and ${\sigma }^2$, where the former controls the magnitude of model misspecification $f(X)-X^T{\mathit{\theta }}^{*}$ or equivalently the nonlinearity of $f(X)$ in the second moment and the latter is the variance of $ϵ$. In particular, we allow ${\mathrm{\Phi }}^2$ to grow with $n$ in our framework.

The second model class we consider is the additive model ${ℱ}_{\text{additive}}=\{{\sum }_{j=1}^p{f}_j(X_j)\}$, where $f_j$’s are unknown second-order-smooth functions (Lin and Zhang, 2006; Meier et al., 2009; Huang et al., 2010; Raskutti et al., 2012). To ease the presentation, we defer the definition of $\ell$-smooth functions to Supplement S.4. Similarly, we assume the number of nonzero functions is bounded by $\bar{s}$. Compared to ${ℱ}_{\text{pairwise}}$, the additive model does not allow interactions among covariates but offers more flexibility in associating each component $X_j$ with $Y$. Similar to ${𝒫}_{\mathrm{\Phi },\sigma }^{\text{pairwise}}$, we can define the class of distributions ${𝒫}_{\mathrm{\Phi },\sigma }^{\text{additive}}$, where we replace $f\in {ℱ}_{\text{pairwise}}$ with $f\in {ℱ}_{\text{additive}}$ in the definition.

The following theorem offers the lower bound for the convergence rate of any estimator of ${\mathit{\theta }}^{*}$ over the classes of distributions ${𝒫}_{\mathrm{\Phi },\sigma }^{\text{pairwise}}$ or ${𝒫}_{\mathrm{\Phi },\sigma }^{\text{additive}}$, in the semi-supervised setting. Throughout the paper without causing confusion, we use $C,C^{\prime },c_1$, and $c_2$, etc. to denote generic constants whose values can change from time to time.

Assumption 2.1. Assume that $s\text{log}(p/s)\le Cn$ for some constant $C,4\le s\le (n-1)/4$, and the sparsity level in ${ℱ}_{\text{pairwise}}$ and ${ℱ}_{\text{additive}}$ satisfies $\overline{s}\ge s$.

Theorem 2.2. Under Assumption 2.1, we have that for any $1\le q\le \mathrm{\infty }$,

$$\underset{\widehat{\mathit{\theta }}}{\inf }\underset{P_{X,Y}\in {𝒫}_{\mathrm{\Phi },\sigma }^{\mathrm{pairwise}}}{\sup }{\mathbb{P}}_{P_{X,Y}}[\Vert \widehat{\mathit{\theta }}-{\mathit{\theta }}^{*}{\Vert }_q\ge c_1{s}^{1/q}\left(\mathrm{\Phi }\sqrt{\frac{\log (p/s)}{n+N}}+\sigma \sqrt{\frac{\log (p/s)}{n}}\right)]>c_2,$$ (2.1)

where ${\inf }_{\widehat{\mathit{\theta }}}$ denotes the infimum over all estimators based on the labeled data $\left(Y_i,X_i\right)$ for $1\le i\le n$ and unlabeled data $X_i$ for $n+1\le i\le n+N$, and $c_1$ and $c_2$ are some positive constants. Here we denote $s^{1/\mathrm{\infty }}=1$. In addition, the same lower bound (2.1) holds when we replace $P_{X,Y}\in {𝒫}_{\mathrm{\Phi },\sigma }^{\text{pairwise}}$ with $P_{X,Y}\in {𝒫}_{\mathrm{\Phi },\sigma }^{\text{additive}}$.

Remark 2.3. The lower bound (2.1) in Theorem 2.2 is obtained by restricting $f(\cdot )$ to be in either ${ℱ}_{\text{pairwise}}$ or ${ℱ}_{\text{additive}}$. Indeed, this is a stronger result than the case without such a restriction. Theorem 2.2 implies, if one considers the class of joint distributions of $(X,Y)$ as

$$\begin{gathered} {𝒫}_{\mathrm{\Phi },\sigma }=\{P_{X,Y}|Y=f(X)+ϵ,{\Vert f\Vert }_2<\infty ,\Vert {\mathit{\theta }}^{*}{\Vert }_0\le s,\text{Var}(ϵ)={\sigma }^2, \\ 𝔼{(f(X)-X^T{\mathit{\theta }}^{*})}^2\le {\mathrm{\Phi }}^2,\text{and}{P}_X\in {𝒫}_X\} \end{gathered}$$

without specifying the model class of $f(\cdot )$, the minimax lower bound remains the same as in (2.1). This lower bound consists of two components. Up to some absolute constants, the first term $s^{1/q}\mathrm{\Phi }\sqrt{\text{log}(p/s)/(n+N)}$ corresponds to the error due to potential model misspecification and the second term $s^{1/q}\sigma \sqrt{\text{log}(p/s)/n}$ comes from the uncertainty inherited from the randomness of the error $ϵ$, which always exists even if the regression function is linear $f(X)=X^T{\mathit{\theta }}^{*}$. In this case, we have $\mathrm{\Phi }=0$ and the lower bound agrees with the existing result for sparse linear regression (Verzelen, 2012; Bellec et al., 2018b).

Remark 2.4. The sample size of the unlabeled data $N$ plays an important role in the lower bound (2.1). In Supplement S.4.1, we show that under some conditions, ${\mathrm{\Phi }}^2\asymp s$ so $\mathrm{\Phi }\to \infty$ as the sparsity grows and $\frac{\mathrm{\Phi }}{\sigma }\sqrt{\frac{n}{n+N}}\to \infty$ may happen. In this case, the dominating term in the lower bound $s^{1/q}\mathrm{\Phi }\sqrt{\text{log}(p/s)/(n+N)}$ can be reduced as $N$ increases. If $N$ is sufficiently large such that $\frac{\mathrm{\Phi }}{\sigma }\sqrt{\frac{n}{n+N}}\to c<\infty$, the lower bound attains its minimum $s^{1/q}\sigma \sqrt{\text{log}(p/s)/n}$, which can be viewed as the irreducible error in the semi-supervised setting since a further increase of $N$ would no longer decrease the lower bound. As an illustration, we plot the lower bound in Figure 1 of Supplement S.2.

Figure 1:

The $L_2$ and $L_1$ estimation errors under Model 1 with $p=500$ and $n=200$. The length of the vertical bar represents the magnitude of the sample standard deviations. $L_2$ errors for U-Dantzig are 5.75 (0.20), 7.14 (0.24), 7.47 (0.26) and $L_1$ errors for U-Dantzig are 6.15 (0.17), 6.53 (0.15), 6.73 (0.15). The numbers in the parenthesis are sample standard deviations.

Before delving into our proposed estimators, we briefly summarize some known properties of supervised estimators which shall be useful later. The supervised Dantzig selector is defined as

$${\widehat{\mathit{\theta }}}_D=\text{argmin}\Vert \mathit{\theta }{\Vert }_1,\text{s.t.}{\Vert \frac{1}{n}\sum _{i=1}^n(Y_i-X_i^T\mathit{\theta })X_i\Vert }_{\infty }\le {\mathrm{\lambda }}_D,$$ (2.2)

where ${\mathrm{\lambda }}_D$ is a tuning parameter. It is shown in Lemma S.4 of Supplement S.3 that with high probability

$${\Vert {\widehat{\mathit{\theta }}}_D-{\mathit{\theta }}^{*}\Vert }_1=O_p\left\{s(\mathrm{\Phi }+\sigma )\sqrt{\frac{\text{log}p}{n}}\right\}.$$ (2.3)

Under the condition $N\gg n$, the lower bound in (2.1) is strictly smaller in order than the upper bound (2.3) if and only if $\mathrm{\Phi }/\sigma \to \mathrm{\infty }$. In this case, the supervised estimator ${\widehat{\mathit{\theta }}}_D$ does not attain the lower bound and is thus sub-optimal in the minimax sense; see Figure 1 in Supplement S.2. Similarly, the supervised lasso estimator is defined as

$${\widehat{\mathit{\theta }}}_L=\underset{\mathit{\theta }\in {ℝ}^p}{\text{argmin}}\frac{1}{2n}\sum _{i=1}^n{(Y_i-X_i^T\mathit{\theta })}^2+{\mathrm{\lambda }}_L\Vert \mathit{\theta }{\Vert }_1,$$ (2.4)

where ${\mathrm{\lambda }}_L$ is a tuning parameter. The same upper bound as in (2.3) can be derived similarly. The Dantzig selector and the lasso estimator are theoretically equivalent (Bickel et al., 2009). To better illustrate the motivation of how to incorporate the unlabeled data, we next introduce our semi-supervised estimator with the formulation of the Dantzig selector in Section 3.

3. Optimal Semi-Supervised Estimator

3.1. Motivation and the key step

To motivate our estimator, we first briefly explain how the convergence rate of ${\widehat{\mathit{\theta }}}_D$ in (2.2) is derived. Following the standard argument in Bickel et al. (2009), the Dantizig selector satisfies ${\Vert {\widehat{\mathit{\theta }}}_D-{\mathit{\theta }}^{*}\Vert }_1=O_p(s{\mathrm{\lambda }}_D)$, where the tuning parameter ${\mathrm{\lambda }}_D\gtrsim {\Vert \frac{1}{n}{\sum }_{i=1}^n{X}_i(Y_i-X_i^T{\mathit{\theta }}^{*})\Vert }_{\infty }$. In the proof of Lemma S.4, we further show that $\Vert \frac{1}{n}{\sum }_{i=1}^n{X}_i{(Y_i-X_i^T{\mathit{\theta }}^{*})\Vert }_{\infty }\lesssim \sqrt{\frac{\text{log}p}{n}}\{𝔼(Y_i-{{X_i^T{\mathit{\theta }}^{*})}^2\}}^{1/2}$ with high probability. The desired bound (2.3) is obtained by noting that

$$𝔼{(Y_i-X_i^T{\mathit{\theta }}^{*})}^2=𝔼{(Y_i-f(X_i))}^2+𝔼{(f(X_i)-X_i^T{\mathit{\theta }}^{*})}^2\le {\sigma }^2+{\mathrm{\Phi }}^2.$$ (3.1)

In view of (2.3) and Remark 2.3, we see that the slow rate of ${\widehat{\mathit{\theta }}}_D$ is driven by the sup-norm of the score function ${\Vert \frac{1}{n}{\sum }_{i=1}^n{X}_i(Y_i-X_i^T{\mathit{\theta }}^{*})\Vert }_{\infty }$.

To find an estimator with the improved rate, our key idea is to construct a modified score function. To this end, we decompose the score function of ${\widehat{\mathit{\theta }}}_D$ as

$$\frac{1}{n}\sum _{i=1}^n{X}_i(Y_i-X_i^T{\mathit{\theta }}^{*})=\frac{1}{n}\sum _{i=1}^n{X}_i(Y_i-f(X_i))+\frac{1}{n}\sum _{i=1}^n{X}_i(f(X_i)-X_i^T{\mathit{\theta }}^{*}).$$

We propose to replace the last term as $\frac{1}{n+N}{\sum }_{i=1}^{n+N}{X}_i(f(X_i)-X_i^T{\mathit{\theta }}^{*})$, the sample average over both labeled and unlabeled data. Apparently, it is a consistent estimator of $𝔼[X_i(f(X_i)-X_i^T{\mathit{\theta }}^{*})]=-\frac{1}{2}𝔼[\frac{\partial }{\partial \mathit{\theta }}{(f(X_i)-X_i^T{\mathit{\theta }}^{*})}^2]$ with a faster rate. Thus, the unlabeled data can help estimate the expectation of the gradient of the model misspecification error. This explains why the unlabeled data may help in the case of misspecification. This leads to the following modified score function

$$\frac{1}{n}\sum _{i=1}^n{X}_i(Y_i-f(X_i))+\frac{1}{n+N}\sum _{i=1}^{n+N}{X}_i(f(X_i)-X_i^T{\mathit{\theta }}^{*})=\overline{\mathit{\xi }}-{\widehat{\Sigma }}_{n+N}{\mathit{\theta }}^{*},$$ (3.2)

where ${\widehat{\Sigma }}_{n+N}=\frac{1}{n+N}{\sum }_{i=1}^{n+N}{X}_i^{\otimes 2}$ and

$$\overline{\mathit{\xi }}=\frac{1}{n}\sum _{i=1}^n{X}_i{Y}_i-\frac{1}{n}\sum _{i=1}^n{X}_{i}f(X_i)+\frac{1}{n+N}\sum _{i=1}^{n+N}{X}_{i}f(X_i).$$ (3.3)

Further insight on the modified score function can be found in Supplement S.4.5.

3.2. Computation of $\bar{\xi }$

To compute $\overline{\mathit{\xi }}$, we need to find an estimator for $f(\cdot )$, the unknown conditional mean function. In the rest of the paper, we use $\widehat{h}(\cdot )$ to denote the estimate of the conditional mean function and will discuss some examples in Remark 3.3. To account for the possible model misspecification of the unknown conditional mean function, we assume that there exists a function $h(\cdot )$ with $\Vert h{\Vert }_2<\infty$ such that the estimate $\widehat{h}(\cdot )$ converges to $h(\cdot )$ in the $L_2(\mathbb{P})$ norm. We refer to $h$ as a conditional mean model. When the conditional mean model is correctly specified, we would expect that $f=h$ and $\widehat{h}(\cdot )$ is consistent for $f(\cdot )$.

A serious challenge may arise from deriving the theoretical property of our proposed semi-supervised estimator if we use all data to obtain $\widehat{h}(\cdot )$ due to the dependence between the estimator $\widehat{h}(\cdot )$ and the data $\left(X_i,Y_i\right)$ in the sample average from $\bar{\mathit{\xi }}$. To bypass this challenge, we adopt the cross-fitting technique that was devised for semiparametric estimation problems (Bickel, 1982; Schick, 1986) as well as for high-dimensional data (Robins et al., 2017; Chernozhukov et al., 2018). For notational simplicity, we denote by $D^{*}$ the labeled data and $D$ the full dataset. Without loss of generality, we split the labeled data $D^{*}$ into two folds $D_1^{*}$ and $D_2^{*}$ with size $n_1=n_2=n/2$. Similarly, we split the unlabeled data into two folds $U_1$ and $U_2$ with size $N_1=N_2=N/2$. Merging $U_1$ and $U_2$ with $D_1^{*}$ and $D_2^{*}$ respectively, we obtain two independent data sets $D_1=D_1^{*}\cup U_1$ and $D_2=D_2^{*}\cup U_2$. Next, for $j=\{1,2\}$, we train the estimator ${\widehat{h}}^{-j}$ using the data $D^{*}\setminus D_j^{*}$ and then construct

$${\widehat{\mathit{\xi }}}_j=\frac{1}{n_j}\sum _{i\in D_j^{*}}{X}_i{Y}_i-\frac{1}{n_j}\sum _{i\in D_j^{*}}{X}_i{\widehat{h}}^{-j}(X_i)+\frac{1}{n_j+N_j}\sum _{i\in D_j}{X}_i{\widehat{h}}^{-j}(X_i).$$ (3.4)

In view of the modified score function (3.2), replacing $\overline{\mathit{\xi }}$ with $\widehat{\mathit{\xi }}=\left({\widehat{\mathit{\xi }}}_1+{\widehat{\mathit{\xi }}}_2\right)/2$, we propose the following semi-supervised Dantzig selector

$${\widehat{\mathit{\theta }}}_{SD}=\text{arg min}\Vert \mathit{\theta }{\Vert }_1,\text{s.t.}\Vert {\widehat{\Sigma }}_{n+N}\mathit{\theta }-\widehat{\mathit{\xi }}\Vert {}_{\infty }\le {\mathrm{\lambda }}_{SD}.$$ (3.5)

Similarly, we define the semi-supervised lasso estimator as

$${\widehat{\mathit{\theta }}}_{SL}=\text{arg}\underset{\mathit{\theta }\in {ℝ}^p}{\text{min}}{\mathit{\theta }}^T{\widehat{\Sigma }}_{n+N}\mathit{\theta }-2{\widehat{\mathit{\xi }}}^T\mathit{\theta }+2{\mathrm{\lambda }}_{SL}\Vert \mathit{\theta }{\Vert }_1.$$ (3.6)

3.3. Theoretical property of the proposed estimator

We develop the theoretical property for the proposed optimal semi-supervised estimator.

Assumption 3.1. We make the following assumptions:

• (A1)${\Sigma }^{-1/2}X$ is a zero mean sub-Gaussian vector with bounded sub-Gaussian norm and $\mathrm{C}\mathrm{o}\mathrm{v}(X)=\mathbf{\Sigma }$ has smallest eigenvalue ${\text{Λ}}_{\text{min}}(\Sigma )\ge C_{\text{min}}>0$ for some positive constant $C_{\text{min}}$. Moreover, ${\text{max}}_{1\le j\le p}{\text{Σ}}_{jj}=O(1)$.
• (A2)${\text{max}}_{1\le i\le n+N}\Vert X_i{\Vert }_{\infty }\le K_1$ where we allow $K_1$ to diverge with $(n,N,p)$.
• (A3)$𝔼({ϵ}^2)={\sigma }^2$ and $𝔼[{(f(X)-X^T{\mathit{\theta }}^{*})}^2]\le {\mathrm{\Phi }}^2$.
• (A4)${\mathit{\theta }}^{*}$ is $s$-sparse with ${\Vert {\mathit{\theta }}^{*}\Vert }_0\le s$, and $\frac{s\text{log}p}{n+N}=O(1)$.

Assumption (A1) is a standard technical condition for $X$ in order to verify the restricted eigenvalue (RE) condition (Bickel et al., 2009). Assumption (A2) imposes the boundedness of the covariates, which simplifies the analysis when the linear model is misspecified (Bühlmann and Van de Geer, 2015). In particular, when $X_i$ is uniformly bounded, $K_1$ becomes a constant. If each component of $X_i$ is Gaussian or sub-Gaussian, Assumption (A2) still holds with high probability with $K_1=C\sqrt{\text{log}[p(n+N)]}$ for some constant $C$. Assumption (A3) only requires the existence of the second moment of $ϵ$ and $f(X)-X^T{\mathit{\theta }}^{*}$. We note that, unlike Bickel et al. (2009), we do not assume the residual $Y-X^T{\mathit{\theta }}^{*}$ to be sub-Gaussian. This is because the residual in the misspecified model $Y-X^T{\mathit{\theta }}^{*}=ϵ+(f(X)-X^T{\mathit{\theta }}^{*})$ contains the nonlinear term $f(X)-X^T{\mathit{\theta }}^{*}$ which can be large. While we only assume the moment condition in Assumption (A3), the boundedness in Assumption (A2) enables us to apply the Nemirovski moment inequality (Lemma S.2) to control the deviation of the sample estimates from their population. Assumption (A4) is the sparsity condition. In particular, Bühlmann and Van de Geer (2015) provided some sufficient conditions on $f(X)$ and the distribution of $X$ under which ${\mathit{\theta }}^{*}$ is sparse in the misspecified model. We further require $\frac{s\text{log}p}{n+N}=O(1)$ to verify the RE condition under the random design; see Lemma S.3.

Given Assumption 3.1, we establish the convergence rate of the semi-supervised Dantzig selector ${\widehat{\mathit{\theta }}}_{SD}$ in (3.5). By Bickel et al. (2009), one can easily show that the same error bounds hold for the semi-supervised lasso estimator ${\widehat{\mathit{\theta }}}_{SL}$. For simplicity, we only present the asymptotic results for ${\widehat{\mathit{\theta }}}_{SD}$, where $n,p\to \infty$ and $N$ can be either fixed or tends to infinity as well.

Theorem 3.2. Suppose Assumption 3.1 holds and the estimator ${\widehat{h}}^{-j}(\cdot )$ satisfies

$$\Vert {\widehat{h}}^{-j}-h{\Vert }_2=O_p(b_n),$$

for $j=\mathrm{1,\; 2}$, where $b_n$ is a deterministic sequence. Denote $G_h=\Vert h-f{\Vert }_2$. With some tuning parameter ${\mathrm{\lambda }}_{SD}\asymp K_1\left(\mathrm{\Phi }\sqrt{\frac{\log p}{n+N}}+\left(\sigma +b_n+G_h\right)\sqrt{\frac{\log p}{n}}\right)$, the estimator ${\widehat{\mathit{\theta }}}_{SD}$ in (3.5) achieves the following error bounds

$${\Vert {\widehat{\mathit{\theta }}}_{SD}-{\mathit{\theta }}^{*}\Vert }_q=O_p\left(K_1{s}^{1/q}\left\{\mathrm{\Phi }\sqrt{\frac{\text{log}p}{n+N}}+(\sigma +b_n+G_h)\sqrt{\frac{\text{log}p}{n}}\right\}\right),$$ (3.7)

for $q=\mathrm{1,\; 2}$. Moreover, if $G_h=0$, i.e. $f=h,b_n/\sigma =o(1)$ and $K_1=O(1)$, we obtain

$${\Vert {\widehat{\mathit{\theta }}}_{SD}-{\mathit{\theta }}^{*}\Vert }_q=O_p\left(s^{1/q}\left\{\mathrm{\Phi }\sqrt{\frac{\text{log}p}{n+N}}+\sigma \sqrt{\frac{\text{log}p}{n}}\right\}\right).$$ (3.8)

The error bound (3.7) depends on $b_n$, the stochastic error from the estimator ${\widehat{h}}^{-j}$, and $G_h$ that can be viewed as the asymptotic bias of the estimator. If we assume $f$ belongs to ${ℱ}_{\text{pairwise}}$ or ${ℱ}_{\text{additive}}$ introduced in Section 2, there exist estimators ${\widehat{h}}^{-j}$ in the literature such that $G_h=0$ and $b_n/\sigma =o(1)$; see further explanations in Remark 3.3 and more detailed examples in Supplement S.5. Thus, with $K_1=O(1)$, the error bound (3.7) reduces to (3.8), which matches the minimax lower bound with $q=\mathrm{1,\; 2}$ in Theorem 2.2 up to a $\log s$ factor. Therefore we call the estimator under this situation the optimal semi-supervised estimator. On the other hand, if the bias term $G_h$ is large, the error bound (3.7) implies that our estimator ${\widehat{\mathit{\theta }}}_{SD}$ may have a slow rate. We will revisit this problem in Section 4.

Practically, we can set the tuning parameter ${\mathrm{\lambda }}_{SD}=C{K}_1(\widehat{\mathrm{\Phi }}\sqrt{\frac{\text{log}p}{n+N}}+\widehat{\sigma }\sqrt{\frac{\text{log}p}{n}})$ and choose the constant $C$ by cross-validation. To account for the magnitude of ${\mathrm{\Phi }}^2$ and ${\sigma }^2$ in ${\mathrm{\lambda }}_{SD}$, we suggest to estimate ${\mathrm{\Phi }}^2$ and ${\sigma }^2$ by ${\widehat{\mathrm{\Phi }}}^2=({\widehat{\mathrm{\Phi }}}_1^2+{\widehat{\mathrm{\Phi }}}_2^2)/2$ and ${\widehat{\sigma }}^2=({\widehat{\sigma }}_1^2+{\widehat{\sigma }}_2^2)/2$, where ${\widehat{\mathrm{\Phi }}}_j^2=\frac{1}{n_j}{\sum }_{i\in D_j}{({\widehat{h}}^{-j}(X_i)-{\widehat{\mathit{\theta }}}_D^T{X}_i)}^2$ and ${\widehat{\sigma }}_j^2=\frac{1}{n_j}{\sum }_{i\in D_j^{*}}{(Y_i-{\widehat{h}}^{-j}(X_i))}^2$. The cross-validation method works well in our simulations.

Remark 3.3. (Examples of $\widehat{h}$). If the knowledge of $f(X)$ is available to some extent, we can leverage this information to construct estimators of $f(X)$. We introduced two model classes ${ℱ}_{\text{pairwise}}$ and ${ℱ}_{\text{additive}}$ in Section 2. For instance, if $f(\cdot )\in {ℱ}_{\text{additive}}$, we can directly apply the existing estimators in the literature; see Lin and Zhang (2006); Meier et al. (2009); Huang et al. (2010); Raskutti et al. (2012), among many others. In particular, Corollary 2 in Huang et al. (2010) implies that their adaptive group lasso estimator $\widehat{h}$ constructed with spline basis functions satisfies $\Vert \widehat{h}-f{\Vert }_2=O_p(n^{-d/(2d+1)})$, where $d$ is the smoothness of the function $f_k(\cdot )$.

Remark 3.4. (Comparison with Alquier and Hebiri (2012)). In the semi-supervised setting, Alquier and Hebiri (2012) proposed a transductive version of lasso and Dantzig selector based on the imputation method. Let $\tilde{\mathit{Y}}=({\tilde{Y}}_1,\dots ,{\tilde{Y}}_{n+N})$ denote the imputed outcomes (or pseudo-outcomes) from some preliminary estimator. The transductive Dantzig selector is given by

$${\widehat{\mathit{\theta }}}_T=\text{argmin}\Vert \mathit{\theta }{\Vert }_1\mathit{s.t.}\frac{1}{n+N}\tilde{\mathit{X}}{(\tilde{\mathit{Y}}-{\tilde{\mathit{X}}}^T\mathit{\theta })}_{\infty }\le {\mathrm{\lambda }}_T.$$ (3.9)

If the imputation satisfies $\frac{1}{n+N}\Vert \tilde{\mathit{X}}(\tilde{\mathit{Y}}-\tilde{\mathit{X}}{\mathit{\theta }}^{*}){\Vert }_{\infty }\lesssim (\sigma +\mathrm{\Phi })\sqrt{\frac{\text{log}p}{n}}$, it can be shown that ${\Vert {\widehat{\mathit{\theta }}}_T-{\mathit{\theta }}^{*}\Vert }_1\lesssim (\sigma +\mathrm{\Phi })s\sqrt{\frac{\text{log}p}{n}}$ with high probability. We can see that the error bound is of the same order as the supervised Dantzig selector (2.3). Thus, the transductive Dantzig selector is also sub-optimal in the minimax sense; see the last paragraph of Section 2.

To see how the transductive Dantzig selector differs from our estimator ${\widehat{\mathit{\theta }}}_{SD}$, we assume that the imputation is attained by using our estimator $\widehat{h}(\cdot )$, i.e., ${\tilde{Y}}_i=\widehat{h}(X_i)$ for $1\le i\le n+N$. For simplicity, let us omit the cross-fitting step here and defer more derivations in Supplement S.4.3. Then the modified score function ${\widehat{\Sigma }}_{n+N}\mathit{\theta }-\widehat{\mathit{\xi }}$ in our estimator (3.5) can be rewritten as

$${\widehat{\Sigma }}_{n+N}\mathit{\theta }-\widehat{\mathit{\xi }}=\frac{1}{n+N}\sum _{i=1}^n{X}_i(Y_i+\frac{N}{n}\{Y_i-{\tilde{Y}}_i\}-X_i^T\mathit{\theta })+\frac{1}{n+N}\sum _{i=n+1}^{n+N}{X}_i({\tilde{Y}}_i-X_i^T\mathit{\theta }).$$

We can see that for the samples from the unlabeled data $(i\in \{n+1,\dots ,n+N\})$, we use ${\tilde{Y}}_i$ as the pseudo-outcome. However, for the samples from the labeled data $(i\in \{1,\dots ,n\})$, $Y_i+\frac{N}{n}\{Y_i-{\tilde{Y}}_i\}$ serves as the pseudo-outcome. In comparison, the transductive Dantzig selector (3.9) always uses ${\tilde{Y}}_i$ as the pseudo-outcome for $i\in \{1,\dots ,n+N\}$.

Remark 3.5. (Comparison with Bellec et al. (2018a)). Recently, Bellec et al. (2018a) proposed a modified lasso estimator for prediction in the semi-supervised setting, which can be reformatted as the following Dantzig selector

$${\widehat{\mathit{\theta }}}_U=\text{arg min}\Vert \mathit{\theta }{\Vert }_1,\text{s.t.}{\Vert {\widehat{\Sigma }}_{n+N}\mathit{\theta }-\frac{1}{n}\sum _{i=1}^n{X}_i{Y}_i\Vert }_{\infty }\le {\mathrm{\lambda }}_U,$$ (3.10)

where ${\widehat{\mathbf{\Sigma }}}_{n+N}=\frac{1}{n+N}{\sum }_{i=1}^{n+N}{X}_i^{\otimes 2}$. Compared with our estimator ${\widehat{\mathit{\theta }}}_{SD}$ in (3.5), ${\widehat{\mathit{\theta }}}_U$ turns out to be a special case of ${\widehat{\mathit{\theta }}}_{SD}$ by plugging $\widehat{h}=0$ into (3.4).

Bellec et al. (2018a) showed that if a large number of unlabeled data are used to compute ${\widehat{\mathbf{\Sigma }}}_{n+N}$, it becomes more plausible to assume that the compatibility (or RE) constant is bounded away from zero. The statement also holds true for our semi-supervised estimator ${\widehat{\mathit{\theta }}}_{SD}$ with the use of ${\widehat{\mathbf{\Sigma }}}_{n+N}$. Moreover, they proved that the error bound for the excess risk in prediction remains $O_p(s\log p/n)$ under certain conditions, including $|Y|\le C$ for some constant $C>0$ which indeed implies $\mathrm{\Phi }=O(1)$ and $\sigma =O(1)$ by their proof of Theorem 7. To make a fair comparison of ${\widehat{\mathit{\theta }}}_U$ with our estimator ${\widehat{\mathit{\theta }}}_{SD}$, we show that under the same conditions in our Theorem 3.2,

$$\Vert {\widehat{\mathit{\theta }}}_U-{\mathit{\theta }}^{*}{\Vert }_1=O_p\left(s(\mathrm{\Phi }+\sigma +{({\mathit{\theta }}^{*T}\Sigma {\mathit{\theta }}^{*})}^{1/2})\sqrt{\frac{\text{log}p}{n}}\right).$$ (3.11)

The proof is deferred to Supplement S.4.2. It is seen that ${\widehat{\mathit{\theta }}}_U$ has a slower rate than our estimator ${\widehat{\mathit{\theta }}}_{SD}$ if $\mathrm{\Phi }/\sigma \to \mathrm{\infty }$ or ${\mathit{\theta }}^{*T}\mathbf{\Sigma }{\mathit{\theta }}^{*}/{\sigma }^2\to \mathrm{\infty }$. Again, in Supplement S.4.1, we consider the examples where ${\mathrm{\Phi }}^2\asymp s$ and $\sigma$ is a constant, so that $\mathrm{\Phi }/\sigma \to \mathrm{\infty }$ holds, as $s$ grows with $n$. Perhaps, a more surprising fact is that the convergence rate of ${\widehat{\mathit{\theta }}}_U$ can be even slower than the fully supervised Dantzig selector ${\widehat{\mathit{\theta }}}_D$ in (2.3) if ${\mathit{\theta }}^{*T}\Sigma {\mathit{\theta }}^{*}/({\sigma }^2+{\mathrm{\Phi }}^2)\to \infty$. Indeed, our simulation studies confirm that the estimator ${\widehat{\mathit{\theta }}}_U$ often produces larger estimation error than ${\widehat{\mathit{\theta }}}_{SD}$ and ${\widehat{\mathit{\theta }}}_D$.

4. Safe Semi-Supervised Estimator

Recall from Theorem 3.2 that our semi-supervised estimator ${\widehat{\mathit{\theta }}}_{SD}$ is minimax optimal, if the conditional mean function $f(\cdot )$ can be consistently estimated with a proper rate, e.g., when $f(\cdot )$ belongs to ${ℱ}_{\text{pairwise}}$ or ${ℱ}_{\text{additive}}$ introduced in Section 2. If this does not hold, there is no guarantee that the estimator ${\widehat{\mathit{\theta }}}_{SD}$ attains the minimax lower bound or outperforms the supervised estimator. In particular, when $G_h+b_n\gg \mathrm{\Phi }$, the convergence rate of ${\widehat{\mathit{\theta }}}_{SD}$ can be even slower than the supervised estimator ${\widehat{\mathit{\theta }}}_D$ in (2.2), hence the integration of unlabeled data might fail to improve the estimation accuracy of ${\mathit{\theta }}^{*}$.

To tackle this problem, we develop a safe semi-supervised estimator via a two-step procedure to adapt to the unknown quality of the conditional mean model $h(\cdot )$, which makes the final estimator no worse than the supervised estimator. To facilitate the theoretical analysis of the two-step estimator, we proceed with the lasso type estimators in this section. Recall that the supervised lasso was defined in (2.4) in Section 2. Equivalent to (3.6), we can rewrite our optimal semi-supervised lasso estimator ${\widehat{\mathit{\theta }}}_{SL}$ as

$${\widehat{\mathit{\theta }}}_{SL}=\underset{\mathit{\theta }\in {ℝ}^p}{\text{argmin}}\sum _{j=1}^2\frac{{\sum }_{i\in D_j}{({\widehat{h}}^{-j}(X_i)-X_i^T\mathit{\theta })}^2}{n+N}-\frac{2{\sum }_{i\in D_j^{*}}(Y_i-{\widehat{h}}^{-j}(X_i))X_i^T\mathit{\theta }}{n}+2{\mathrm{\lambda }}_{SL}\Vert \mathit{\theta }{\Vert }_1.$$ (4.1)

We construct the safe semi-supervised estimator in the following two steps. First, for a given estimate of the unknown conditional mean $\widehat{h}$, we compute our semi-supervised lasso estimator ${\widehat{\mathit{\theta }}}_{SL}$ in (4.1) and the supervised lasso estimator ${\widehat{\mathit{\theta }}}_L$ in (2.4). Since the estimator ${\widehat{\mathit{\theta }}}_{SL}$ may not be desirable when the corresponding conditional mean model $h$ is misspecified, in the second step we further improve the initial estimator ${\widehat{\mathit{\theta }}}_{SL}$ by a step of refitting on a suitable estimate of the support set of ${\mathit{\theta }}^{*}$. Denote the support set of two lasso type estimators ${\widehat{\mathit{\theta }}}_L$ and ${\widehat{\mathit{\theta }}}_{SL}$ by ${\widehat{T}}_1$ and ${\widehat{T}}_2$ respectively. We define the safe semi-supervised estimator as

$${\widehat{\mathit{\theta }}}_R={\widehat{\mathit{\theta }}}_{SL}+\widehat{\mathit{\omega }},$$ (4.2)

where $\widehat{\mathit{\omega }}\in {ℝ}^p$ is attained by

$$\widehat{\mathit{\omega }}=\underset{\text{supp}(\mathit{\omega })\subseteq {\widehat{T}}_1\cup {\widehat{T}}_2}{\text{argmin}}\frac{1}{2n}\sum _{i=1}^n{(Y_i-X_i^T({\widehat{\mathit{\theta }}}_{SL}+\mathit{\omega }))}^2+{\mathrm{\lambda }}_{\omega }\Vert \mathit{\omega }{\Vert }_1.$$ (4.3)

In (4.3) we impose $\mathrm{supp}(\mathit{\omega })\subseteq {\widehat{T}}_1\cup {\widehat{T}}_2$, which implies ${\widehat{\mathit{\omega }}}_j=0$ for any $j\notin {\widehat{T}}_1\cup {\widehat{T}}_2$. Thus, the safe semi-supervised estimator ${\widehat{\mathit{\theta }}}_R$ may only differ from ${\widehat{\mathit{\theta }}}_{SL}$ on the estimated support set ${\widehat{T}}_1\cup {\widehat{T}}_2$. Since both the estimands of ${\widehat{\mathit{\theta }}}_{SL}$ and ${\widehat{\mathit{\theta }}}_R$ are ${\mathit{\theta }}^{*}$, by the definition of (4.2) we can treat $\widehat{\mathit{\omega }}$ as an estimator of ${\mathit{\theta }}^{*}-{\mathit{\theta }}^{*}=0$. At a first sight, one may expect that estimating a known vector of 0 by $\widehat{\mathit{\omega }}$ and summing with ${\widehat{\mathit{\theta }}}_{SL}$ will inflate the error of the resulting estimator. However, we show a surprising result that the two step estimator ${\widehat{\mathit{\theta }}}_R$ may outperform the initial estimator ${\widehat{\mathit{\theta }}}_{SL}$.

In the following, we will first explain the intuition behind the estimator ${\widehat{\mathit{\theta }}}_R$. Let us consider two scenarios. First, if our initial estimator ${\widehat{\mathit{\theta }}}_{SL}$ in step 1 is able to achieve a fast rate, one would expect that $\widehat{\mathit{\omega }}\approx 0$ due to the $L_1$ regularization in (4.3). Thus, ${\widehat{\mathit{\theta }}}_R$ will inherit the fast rate from the initial estimator ${\widehat{\mathit{\theta }}}_{SL}$. In the second scenario, if ${\widehat{\mathit{\theta }}}_{SL}$ has a slow rate due to model misspecification, with a suitable choice of tuning parameters, we can obtain $\widehat{\mathit{\omega }}\approx {\widehat{\mathit{\theta }}}_L-{\widehat{\mathit{\theta }}}_{SL}$. As a toy example, if $p$ is fixed and small and we set all tuning parameters in $\widehat{\mathit{\omega }},{\widehat{\mathit{\theta }}}_{SL}$ and ${\widehat{\mathit{\theta }}}_L$ to be 0, it is easily shown from the least square formula that $\widehat{\mathit{\omega }}={\widehat{\mathit{\theta }}}_L-{\widehat{\mathit{\theta }}}_{SL}$. Thus, in this case, the safe semi-supervised estimator will resemble the supervised estimator, since ${\widehat{\mathit{\theta }}}_R={\widehat{\mathit{\theta }}}_{SL}+\widehat{\mathit{\omega }}\approx {\widehat{\mathit{\theta }}}_L$. In summary, the refitting step can simultaneously retain the fast rate of ${\widehat{\mathit{\theta }}}_{SL}$ if it has, and alleviate the potentially unsatisfactory performance of ${\widehat{\mathit{\theta }}}_{SL}$ under model misspecification. The following theorem shows the convergence rate of the safe semi-supervised estimator ${\widehat{\mathit{\theta }}}_R$.

Theorem 4.1. Suppose Assumption 3.1 holds, ${\text{Λ}}_{\text{max}}(\Sigma )\le C_{\text{max}}<\infty$ and $s{(\text{log}p)}^2=o(n)$. Assume that we choose the tuning parameters ${\mathrm{\lambda }}_{SL},{\mathrm{\lambda }}_L$ and ${\mathrm{\lambda }}_{\omega }$ in (4.1), (2.4) and (4.3) as ${\mathrm{\lambda }}_{SL}\asymp K_1(\mathrm{\Phi }\sqrt{\frac{\text{log}p}{n+N}}+(\sigma +b_n+G_h)\sqrt{\frac{\text{log}p}{n}})$ and ${\mathrm{\lambda }}_L\asymp {\mathrm{\lambda }}_{\omega }\asymp K_1(\mathrm{\Phi }+\sigma )\sqrt{\frac{\text{log}p}{n}}$. We have for $q=1,2$,

$${\Vert {\widehat{\mathit{\theta }}}_R-{\mathit{\theta }}^{*}\Vert }_q=O_p(R_{SL}\wedge R_L),$$

where $R_{SL}=s^{1/q}{K}_1\left(\mathrm{\Phi }\sqrt{\frac{\text{log}p}{n+N}}+(\sigma +b_n+\Vert f-h{\Vert }_2)\sqrt{\frac{\text{log}p}{n}}\right)$, and $R_L=s^{1/q}{K}_1(\mathrm{\Phi }+\sigma )\sqrt{\frac{\text{log}p}{n}}$.

Recall that $R_{SL}$ and $R_L$ correspond to the convergence rates of ${\widehat{\mathit{\theta }}}_{SL}$ and ${\widehat{\mathit{\theta }}}_L$, respectively. Theorem 4.1 shows that ${\widehat{\mathit{\theta }}}_R$ attains the faster rate between ${\widehat{\mathit{\theta }}}_{SL}$ and ${\widehat{\mathit{\theta }}}_L$. Thus, the estimator ${\widehat{\mathit{\theta }}}_R$ remains minimax rate-optimal when the initial estimator ${\widehat{\mathit{\theta }}}_{SL}$ is optimal (see Theorem 3.2), and is guaranteed to be no worse than the supervised estimators even if the conditional mean model is misspecified.

To choose the tuning parameters, we note that the magnitude of ${\mathrm{\lambda }}_{SL}$ is the same as ${\mathrm{\lambda }}_{SD}$ in Theorem 3.2. We can apply the same cross-validation method explained after Theorem 3.2 to tune ${\mathrm{\lambda }}_{SL}$. Since ${\mathrm{\lambda }}_L$ can be written as ${\mathrm{\lambda }}_L=C{K}_1{\{𝔼{(Y-X^T{\mathit{\theta }}^{*})}^2\}}^{1/2}\sqrt{\frac{\text{log}p}{n}}$ for some constant $C$, one may use scaled lasso to estimate the noise level $𝔼{(Y-X^T{\mathit{\theta }}^{*})}^2$, and further apply cross-validation to tune ${\mathrm{\lambda }}_L$. In practice, to reduce the computational cost, once we have selected the tuning parameter ${\mathrm{\lambda }}_L$, we can simply set ${\mathrm{\lambda }}_{\omega }={\mathrm{\lambda }}_L$, which works well in our simulations.

Remark 4.2. The two-step approach is inspired by the recent works of Bastani (2020) and Li et al. (2020) in the context of transfer learning. However, our theoretical guarantee in Theorem 4.1 is much stronger than those works. Using our terminology, the theoretical analysis in Bastani (2020) and Li et al. (2020) implied that the estimator ${\widehat{\mathit{\theta }}}_R$ can only attain the worst possible rate between ${\widehat{\mathit{\theta }}}_{SL}$ and ${\widehat{\mathit{\theta }}}_L$, i.e., $\Vert {\widehat{\mathit{\theta }}}_R-{\mathit{\theta }}^{*}{\Vert }_q=O_p(R_{SL}+R_L)$, which is not sufficient to show ${\widehat{\mathit{\theta }}}_R$ to be safe. We obtain a sharper result in Theorem 4.1 because our refitting step (4.3) differs from those works. In particular, unlike their works, we constrain the support of the estimator $\widehat{\mathit{\omega }}$ to be ${\widehat{T}}_1\cup {\widehat{T}}_2$, which guarantees the sparsity of ${\widehat{\mathit{\theta }}}_R$. This is an important intermediate step to prove Theorem 4.1. Finally, we note that, in the context of transfer learning, Li et al. (2020) also considered how to use model aggregation, such as Q-aggregation (Dai et al., 2012), to improve the initial estimator. We refer to Supplement S.4.4 for further discussion on model aggregation.

5. Aggregation of Semi-Supervised Estimators

Recall from Theorem 3.2 that the performance of the semi-supervised estimator depends on the estimator $\widehat{h}$ of the conditional mean model. In practice, it is uncommon for us to find a consistent estimator of the conditional mean function. More commonly, we may face multiple choices of $\widehat{h}$’s, none of which is consistent. We may expect that different $\widehat{h}$’s only capture different aspects of the non-linearity of the conditional mean function. Does aggregating multiple semi-supervised estimators help us explore the non-linearity of the conditional mean function? Here, we generalize the two-step method in Section 4 to answer this question.

Assume that two different estimators of the conditional mean function ${\widehat{h}}_1$ and ${\widehat{h}}_2$ are available. We denote by ${\widehat{\mathit{\theta }}}_{h_1}$ and ${\widehat{\mathit{\theta }}}_{h_2}$ the semi-supervised lasso estimators in (4.1) with ${\widehat{h}}_1$ and ${\widehat{h}}_2$ and tuning parameters ${\mathrm{\lambda }}_{h_1}$ and ${\mathrm{\lambda }}_{h_2}$ respectively. In the following, we will apply the two-step procedure to combine ${\widehat{\mathit{\theta }}}_{h_1}$ and ${\widehat{\mathit{\theta }}}_{h_2}$. We first compute ${\widehat{\mathit{\theta }}}_{h_1}$ and ${\widehat{\mathit{\theta }}}_{h_2}$ in step 1. Their support sets are denoted by ${\widehat{H}}_1$ and ${\widehat{H}}_2$. In step 2, we use ${\widehat{\mathit{\theta }}}_{h_1}$ as the initial estimator and define the aggregated estimator as

$${\widehat{\mathit{\theta }}}_{AH}={\widehat{\mathit{\theta }}}_{h_1}+{\widehat{\mathit{\omega }}}_h,$$ (5.1)

where

$${\widehat{\mathit{\omega }}}_h=\underset{\text{supp}(\mathit{\omega })\subseteq {\widehat{H}}_1\cup {\widehat{H}}_2}{\text{argmin}}\sum _{j=1}^2(\frac{{\sum }_{i\in D_j}{({\widehat{h}}_2^{-j}(X_i)-X_i^T({\widehat{\mathit{\theta }}}_{h_1}+\mathit{\omega }))}^2}{n+N}-\frac{2{\sum }_{i\in D_j^{*}}(Y_i-{\widehat{h}}_2^{-j}(X_i))X_i^T({\widehat{\mathit{\theta }}}_{h_1}+\mathit{\omega })}{n})+2{\mathrm{\lambda }}_H\Vert \mathit{\omega }{\Vert }_1.$$

Unlike the refitting step in (4.3), we also incorporate the unlabeled data to compute ${\widehat{\mathit{\omega }}}_h$ so that ${\widehat{\mathit{\theta }}}_{AH}$ can attain a better convergence rate than ${\widehat{\mathit{\theta }}}_{h_1}$ or ${\widehat{\mathit{\theta }}}_{h_2}$. One may note that the creation of ${\widehat{\mathit{\theta }}}_{AH}$ is not symmetric to $h_1$ and $h_2$; however, our theoretical analysis below shows that the priority order of $h_1$ and $h_2$ does not really matter.

Proposition 5.1. Suppose Assumption 3.1 holds, ${\mathrm{\Lambda }}_{\max }(\mathbf{\Sigma })\le C_{\max }<\mathrm{\infty }$ and $s(\log p{)}^2=o(n+N)$. The estimators ${\widehat{h}}_1$ and ${\widehat{h}}_2$ satisfy $\Vert {\widehat{h}}_1^{-j}-h_1{\Vert }_2=O_p(b_{h_1})$ and $\Vert {\widehat{h}}_2^{-j}-h_2{\Vert }_2=O_p(b_{h_2})$ for $j=1,2$. Denote $G_{h_1}={\Vert h_1-f\Vert }_2$ and $G_{h_2}={\Vert h_2-f\Vert }_2$. Selecting some tuning parameters ${\mathrm{\lambda }}_{h_1}\asymp K_1(\mathrm{\Phi }\sqrt{\frac{\text{log}p}{n+N}}+(\sigma +b_{h_1}+G_{h_1})\sqrt{\frac{\text{log}p}{n}}),{\mathrm{\lambda }}_{h_2}\asymp {\mathrm{\lambda }}_H\asymp K_1(\mathrm{\Phi }\sqrt{\frac{\text{log}p}{n+N}}+(\sigma +b_{h_2}+G_{h_2})\sqrt{\frac{\text{log}p}{n}})$, we can show that for $q=\mathrm{1,\; 2}$,

$${\Vert {\widehat{\mathit{\theta }}}_{AH}-{\mathit{\theta }}^{*}\Vert }_q=O_p(R_{h_1}\wedge R_{h_2}),$$

where $R_{h_j}=s^{1/q}{K}_1\left(\mathrm{\Phi }\sqrt{\frac{\text{log}p}{n+N}}+(\sigma +b_{h_j}+\Vert f-h_j{\Vert }_2)\sqrt{\frac{\text{log}p}{n}}\right)$.

This proposition shows that the aggregated estimator ${\widehat{\mathit{\theta }}}_{AH}$ attains the best possible rate between ${\widehat{\mathit{\theta }}}_{h_1}$ and ${\widehat{\mathit{\theta }}}_{h_2}$. Following the same reasoning, one may use ${\widehat{\mathit{\theta }}}_{AH}$ as the initial estimator and repeat the refitting step, if a third estimator ${\widehat{\mathit{\theta }}}_{h_3}$ is available. The resulting estimator attains the best possible rate among the three estimators ${\widehat{\mathit{\theta }}}_{h_1},{\widehat{\mathit{\theta }}}_{h_2}$ and ${\widehat{\mathit{\theta }}}_{h_3}$. We expect that, in general, our two-step procedure can be applied to aggregate multiple estimators, as long as the number of the candidates is fixed and small. We refer to Section 6 for numerical performance of the aggregated estimators.

6. Simulation Studies

6.1. Data generating models and practical implementation

We first generate a $p$-dimensional multivariate normal random vector $Z\sim 𝒩(0,\text{Σ})$ with ${\text{Σ}}_{jk}={0.3}^{|j-k|}$. We set the covariate $X=(X_1,\dots ,X_p)$ to be $X_1=|Z_1|$ and $X_j=Z_j$ for $1<j\le p$. The reason we take $X_1=|Z_1|$ is that this transformation implies $𝔼(X_1^k{X}_j)=0$ for $j\ne 1$ but the parameter ${\mathit{\theta }}_1^{*}$ for centered $X_1$ is nonzero. We consider the following three data generating models for $Y$. For Model 1, we consider an additive model

$$Y=0.5X_1^2+0.8X_3^3-{(X_4-2)}^2+2{(X_5+1)}^2+2X_6+ϵ,$$

where $ϵ\sim 𝒩(\mathrm{0,1})$. To calculate the corresponding regression parameter ${\mathit{\theta }}^{*}$ under the working linear model, we first center $Y$ and $X_1$ so that their means are 0. By Proposition 4 in Bühlmann and Van de Geer (2015), we know that the support of ${\mathit{\theta }}^{*}$ is $S=\{1,3,4,5,6\}$ and ${\theta }_j^{*}$ for any $j\in S$ is given by the $L_2(\mathbb{P})$ projection in the sub-model only with the variable $X_j$ (e.g, ${\theta }_3^{*}=\text{arg min}𝔼{(0.8X_3^2-{\mathit{\theta }}_3{X}_3)}^2)$. After some calculation, we obtain ${\mathit{\theta }}^{*}=(1.1,0,2.4,4,4,2,0,\dots ,0)$, which is sparse.

For Model 2, we consider

$$Y=0.6{(X_1+X_2)}^2+0.4X_4^3-X_5+2X_6+ϵ,$$

where $ϵ\sim 𝒩(\mathrm{0,1})$. The model is non-additive since it includes an interaction term between $X_1$ and $X_2$. The corresponding regression parameter ${\mathit{\theta }}^{*}$ is $(1.48,1.04,0,1.2,-1,2,0,\dots ,0)$.

Besides, we consider a non-additive Model 3 which includes extra randomly selected support sets $S_1$ and $S_2$ on top of Model 1. We choose $|S_1|=|S_2|=5$ and set

$$Y=0.5X_1^2+0.8X_3^3-{(X_4-2)}^2+2{(X_5+1)}^2+2X_6+2\sum _{k\in S_1}{X}_k+0.05{(\sum _{k\in S_2}{X}_k)}^3+ϵ,$$

where $ϵ\sim 𝒩(0,1)$. In this model, ${\theta }_k^{*}=0.75$ for $k\in S_2,{\theta }_k^{*}=2$ for $k\in S_1$, and the other components of ${\mathit{\theta }}^{*}$ are the same as in Model 1. Under each data generating model, we consider several combinations of $(n,p)$ and vary the ratio $N/n$ in a certain range. We repeat the simulation 100 times and output the average as final results.

Before we proceed to illustrate the results, we list the estimators considered in this section and discuss several practical implementation issues. Since the performance of lasso and Dantzig type estimators are similar, we only consider the Dantzig type estimators here for simplicity.

• The proposed semi-supervised estimator ${\widehat{\mathit{\theta }}}_{SD}$ in (3.5) (SSL1) with a sparse additive model $h_1$. We estimate the sparse additive model by using the group lasso with the spline basis (Huang et al., 2010). To be specific, we use the cubic spline basis with degree of freedom $df=5$. To select the penalty parameter in group lasso and make computation easier, the BIC criterion is used; see Section 4 in Huang et al. (2010) for the definition. After we derive the estimator ${\widehat{h}}_1$ and subsequently $\widehat{\mathit{\xi }}$, we modify the source code in the flare package to compute ${\widehat{\mathit{\theta }}}_{SD}$, where the tuning parameter ${\mathrm{\lambda }}_{SD}$ is selected by 5 fold cross-validation.

• The proposed semi-supervised estimator ${\widehat{\mathit{\theta }}}_{SD}$ in (3.5) (SSL2) with a pairwise interaction model $h_2$. The model $h_2(\cdot )$ corresponds to the linear regression containing all the linear terms, the squares of the variable and the interaction terms (Zhao et al., 2016).

• The supervised Dantzig selector ${\widehat{\mathit{\theta }}}_D$ in (2.2) (Dantzig). We use the flare package to compute the estimator and select the tuning parameter by 5 fold cross-validation.

• The modified Dantzig selector ${\widehat{\mathit{\theta }}}_U$ in (3.10) (U-Dantzig).

• The aggregated estimator ${\widehat{\mathit{\theta }}}_{AH}$ in (5.1) (SSL12) that combines SSL1 and SSL2.

• The safe semi-supervised estimator ${\widehat{\mathit{\theta }}}_R$ (S-SSL1) using SSL1 as the initial estimator. As seen from the discussion after Theorem 4.1, we set the tuning parameter ${\mathrm{\lambda }}_{\omega }$ identical to the tuning parameter for the supervised Dantzig selector.

• The safe semi-supervised estimator ${\widehat{\mathit{\theta }}}_R$ (S-SSL12) using SSL12 (e.g., the aggregated estimator) as the initial estimator.

6.2. Numerical results

Under Model 1 with $p=500$ and $n=200$, the comparisons of the estimation errors of SSL1, SSL2, Dantzig, SSL12, S-SSL1 and S-SSL12 are illustrated in Figure 1, while the results from U-Dantzig are shown in numbers in the caption of the figure due to the scale of plot. Since the true data generating model is additive with some quadratic terms, both SSL1 and SSL2 can leverage the non-linearity of the conditional mean function and their estimation errors are much smaller than Dantzig, which agrees with Theorem 3.2. Besides, we can see that SSL1 outperforms SSL2, since the imposed additive model in SSL1 can better estimate the conditional mean function. By aggregating SSL1 and SSL2, our estimator SSL12 achieves the minimum $L_2$ error among all those methods. For the two safe semi-supervised estimators (S-SSL1 and S-SSL12), they retain the optimal rate in $L_2$ norm from the corresponding semi-supervised estimators (SSL1 and SSL12), and clearly outperform Dantzig, which is consistent with the theoretical property in Theorem 4.1.

One interesting observation is that U-Dantzig performs much worse than the fully supervised estimator Dantzig; see the caption of Figure 1. Thus, using the sample covariance ${\widehat{\mathbf{\Sigma }}}_{n+N}$ from both labeled and unlabeled data in the Dantzig selector may not provide any empirical improvement; see Remark 3.5 for the theoretical justification. In addition, as the size of unlabeled data $N$ increases, the improvement of our semi-supervised estimators (SSL1, SSL2, SSL12, S-SSL1 and S-SSL12) is more overwhelming, whereas the performance of U-Dantzig tends to deteriorate.

The comparisons under Model 2 with $p=500$ and $n=200$ are shown in Figure 2. Since Model 2 includes an interaction term between $X_1$ and $X_2$, the sparse additive model $h_1$ is inconsistent for the true regression function. Thus SSL1 does not improve the estimation accuracy compared with fully supervised Dantzig. However, the safe semi-supervised estimator, S-SSL1, successfully mitigates the undesired performance of SSL1 and its $L_1$ and $L_2$ errors are smaller than Dantzig. This agrees with Theorem 4.1 that the refitting step provides a safe use of unlabeled data even if the imposed conditional mean model is incorrect. On the other hand, since the pairwise interaction model $h_2$ is a correctly specified conditional mean model, the estimators SSL2, SSL12 and S-SSL12, that depend on this model, show small estimation errors.

Figure 2:

The $L_2$ and $L_1$ estimation errors under Model 2 with $p=500$ and $n=200$. The length of the vertical bar represents the magnitude of the sample standard deviations. $L_2$ errors for U-Dantzig are 2.31 (0.08), 2.75 (0.09), 2.87 (0.11) and $L_1$ errors for U-Dantzig are 4.28 (0.10), 4.62 (0.10), 4.66 (0.07).

Under Model 3, since the true conditional mean function differs significantly from the additive model $h_1$, SSL1 yields large estimation errors. Nevertheless, the performance of S-SSL1 is comparable and no worse than the fully supervised Dantzig. While the pairwise interaction model $h_2$ cannot account for the third order interaction terms in the set $S_2$, it can still partially explain the non-linearity of the true conditional mean function. Thus, the performance of SSL2 is still better than Dantzig in $L_2$ norm when $N/n=5$ and also in $L_1$ norm. The comparison results are summarized in Figure 3.

Figure 3:

The $L_2$ and $L_1$ estimation error under Model 3 with $p=500$ and $n=200$. The length of the vertical bar represents the magnitude of the sample standard deviations. $L_2$ errors for U-Dantzig are 11.46(0.23), 13.29(0.25), 13.62(0.26) and $L_1$ errors for U-Dantzig are 14.32(0.16), 15.73(0.24), 15.60(0.20).

The Supplement S.6 contains further simulation results with $p=200$ and $n=100$, and with $p=1000$ and $n=300$, for all the three models considered above, as well as some other numerical results.

7. Real Data Application

In this section, we illustrate our proposed methods in a real data example, derived from the Medical Information Mart for Intensive Care III (MIMIC-III) database (Johnson et al., 2016). MIMIC-III is an openly available electronic health records system developed by the MIT Lab for Computational Physiology. It contains de-identified health-related data for 38,597 adult patients (aged 16 years or above) admitted to intensive care units (ICU) of the Beth Israel Deaconess Medical Center between 2001 and 2012. Some details of the adult patients by their first ICU admissions are available in the Table 1 of Johnson et al. (2016).

Researchers have used the MIMIC-III database to investigate a variety of medical issues such as predicting ICU readmissions (Brown et al., 2012; Tabak et al., 2017; Xue et al., 2019) and associating various clinical biomarkers with mortality (Liu et al., 2020; Du et al., 2021; Jhou et al., 2021; Tang et al., 2021). Our initial motivation for this data analysis is to understand the relation between the albumin level in the blood sample, oftentimes used to screen for liver or kidney disease (Phillips et al., 1989), and all other variables including demographics, chart events, and clinical biomarkers from the laboratory tests.

After all of the data pre-processing steps detailed in Supplement S.7, we are left with a dataset with 4784 patients and each of them has $p=2928$ covariates. The results depicted in Figure 4 are an aggregation from 100 replications. In each replication, we randomly choose 2000 patients and call the first $n=500$ patients the labeled data. We mask the outcome “albumin” for all other 1500 patients and call them the unlabeled data. In each experiment, we gradually increase the sample size of the unlabeled data from $N=500$ to $N=1000$ and to $N=1500$. Since the working model already includes some pairwise interaction terms among biomarkers, we use the random forest to estimate the conditional mean function in our SSL estimator and the corresponding S-SSL estimator. The $L_1$ and $L_2$ estimation errors presented in Figure 4 are computed against the lasso estimator using all of the 4784 samples, which is regarded as the proxy of the underlying true linear coefficients of the working model.

Figure 4:

The $L_2$ and $L_1$ estimation error for real data application. The length of the vertical bar represents the magnitude of the sample standard deviations.

From Figure 4, as the ratio $N/n$ increases from 1 to 3, compared to supervised lasso, the proposed S-SSL estimator could bring the $L_1$ error down around 7.6% to 10.3% and the proposed SSL estimator could bring the $L_1$ error down around 10.1% to 21.7%, respectively. The percentage decrease for the $L_2$ error ranges from 14.9% to 22.4% for the S-SSL estimator and from 23.2% to 28.6% for the SSL estimator. Compared to the S-SSL estimator, the outperformance of the SSL estimator, albeit not as significant as the comparison with the supervised lasso, is still noticeable from Figure 4. The likely reason for this is that the random forest used in our semi-supervised estimators (SSL and S-SSL) has already effectively captured the structure of the conditional mean function. Thus, S-SSL behaves similarly to SSL. All of these observations support the effectiveness of the methods proposed in this paper. In Supplement S.7, similar results can be obtained where the estimation errors are computed against the lasso which exclusively employs samples that are not utilized in the computation of the three comparators.

Finally, Supplement S.7 also contains the detailed data cleaning and data pre-processing procedures for this application, as well as some other results and conclusions. Along the paper, we also submit the programming code for anyone who has interest to reproduce the results.