Prior Adaptive Semi-supervised Learning with Application to EHR Phenotyping

Abstract

Electronic Health Record (EHR) data, a rich source for biomedical research, have been successfully used to gain novel insight into a wide range of diseases. Despite its potential, EHR is currently underutilized for discovery research due to its major limitation in the lack of precise phenotype information. To overcome such difficulties, recent efforts have been devoted to developing supervised algorithms to accurately predict phenotypes based on relatively small training datasets with gold standard labels extracted via chart review. However, supervised methods typically require a sizable training set to yield generalizable algorithms, especially when the number of candidate features, $p$, is large. In this paper, we propose a semi-supervised (SS) EHR phenotyping method that borrows information from both a small, labeled dataset (where both the label $Y$ and the feature set $\mathit{X}$ are observed) and a much larger, weakly-labeled dataset in which the feature set $\mathit{X}$ is accompanied only by a surrogate label $S$ that is available to all patients. Under a working prior assumption that $S$ is related to $\mathit{X}$ only through $Y$ and allowing it to hold approximately, we propose a prior adaptive semi-supervised (PASS) estimator that incorporates the prior knowledge by shrinking the estimator towards a direction derived under the prior. We derive asymptotic theory for the proposed estimator and justify its efficiency and robustness to prior information of poor quality. We also demonstrate its superiority over existing estimators under various scenarios via simulation studies and on three real-world EHR phenotyping studies at a large tertiary hospital.

1. Introduction

Electronic Health Records (EHRs) provide a large and rich data source for biomedical research aiming to further our understanding of disease progression and treatment response. EHR data has been successfully used to gain novel insights into a wide range of diseases, with examples including diabetes (Brownstein et al., 2010), rheumatoid arthritis (Liao et al., 2014), inflammatory bowl disease (Ananthakrishnan et al., 2014), and autism (Doshi-Velez et al., 2014). EHR is also a powerful discovery tool for identifying novel associations between genomic markers and multiple phenotypes through analyses such as phenome-wide association studies (Denny et al., 2010; Kohane, 2011; Wilke et al., 2011; Cai et al., 2018).

Despite its potential, ensuring unbiased and powerful biomedical studies using EHR is challenging because EHR was primarily designed for patient care, billing, and record keeping. Extracting precise phenotype information for an individual patient requires manual medical chart reviews, an expensive process that is not scalable for research studies. To overcome such difficulties, recent efforts including those from Informatics for Integrating Biology and the Bedside (i2b2) (Liao et al., 2015; Yu et al., 2015, e.g.) and the Electronic Medical Records and Genomics (eMERGE) network (Newton et al., 2013; Gottesman et al., 2013) have been devoted to developing phenotyping algorithms to predict disease status using relatively small training datasets with gold standard labels extracted via chart review.

Various approaches to EHR phenotyping have been proposed. Supervised machine learning methods have been shown to achieve robust performance across disease phenotypes and EHR systems (Carroll et al., 2012; Liao et al., 2015). However, supervised methods typically require a sizable training set to yield generalizable algorithms especially when the candidate features, denoted by $\mathit{X}$, is of high dimensionality $p$. One approach to overcome the high dimensionality is to consider unsupervised methods. Unfortunately, standard unsupervised methods such as clustering are likely to fail when the dimension of $\mathit{X}$ is large, but a majority of the features are unrelated to the phenotype of interest and predictive of some other underlying subgroups. Recently, unsupervised methods based on “silver standard labels” have been proposed. These methods leverage a surrogate outcome $S$ that is highly predictive of the true phenotype status $Y$, such as the count of International Classification of Diseases (ICD) billing codes of the disease, to train the phenotyping algorithm against the features $\mathit{X}$. Specifically, Halpern et al. (2016) and Zhang et al. (2020) utilized anchor variables with high positive predictive value as the surrogate $S$ to estimate $Y\mid \mathit{X}$ under the conditional independence assumption $S\perp \mathit{X}\mid Y$. Agarwal et al. (2016) trained penalized logistic regression on $S\sim \mathit{X}$ for phenotyping of $Y$ against $\mathit{X}$. Chakrabortty et al. (2017) provided theoretical justification for this strategy. They showed that a regularized estimator constructed from an unlabeled subset consisting of those with extreme values of $S$ can be used to infer the direction of $\mathit{\beta }$ under single index models $S\sim f\left({\mathit{\alpha }}^{\top }\mathit{X},ϵ\right)$) and $Y\sim g\left({\mathit{\beta }}^{\top }\mathit{X}\right)$. Their method relies on the similarity between the directions of $\mathit{\alpha }$ and $\mathit{\beta }$ to make efficient estimation. However, it is not robust to poor surrogacy resulted from violation of such assumptions. Furthermore, their method cannot be directly used to predict $Y$ using both $S$ and $\mathit{X}$ or accurately recover the scale of $\text{Pr}(Y=1\mid S,\mathit{X})$.

A number of semi-supervised or weakly supervised deep learning procedures have also been proposed recently and shown to attain better performance than the supervised counterparts. For example, Ratner et al. (2017) proposed a weakly supervised approach that trains a deep model with imperfect labels generated from user–specified label functions from sources such as patterns, heuristics, and external knowledge bases. Wang and Poon (2018) developed a framework for weak supervision from multiple sources by composing probabilistic logic with deep learning. McDermott et al. (2018) designed a semi-supervised cycle Wasserstein regression generative adversarial networks (CWR-GAN) approach using adversarial signals to learn from unlabelled samples and improve prediction performance in scarcity of gold-standard labels. However, it remains unclear when and how the surrogate features, along with the unlabeled dataset can improve the prediction performance of these deep models, due to their complex architectures.

In this paper, we propose an semi-supervised (SS) method for estimating $Y\mid \mathbf{W}={\left(S,{\mathit{X}}^{\top }\right)}^{\top }$ that borrows information from both a small labeled dataset with $n$ realizations of ${\left(Y,{\mathbf{W}}^{\top }\right)}^{\top }$ and a much larger unlabeled dataset with $N$ observations on $\mathbf{W}$, under a high dimensional setting with $N\gg p\gg n$. We consider a logistic phenotype model for $Y\mid S,\mathit{X}$, a single index model (SIM) for $S\mid \mathit{X}$, as well as a working prior assumption that $S$ is independent of $\mathit{X}$ given $Y$. We obtain the estimator through regularization with penalty functions reflecting the prior knowledge. When the prior assumption holds exactly, we show that the unlabeled dataset can naturally be used to assist in the estimation of the phenotype model. Allowing the prior assumption to hold approximately or to be highly violated, our prior adaptive semi-supervised (PASS) estimator adaptively incorporate the prior knowledge by shrinking the estimator towards a direction derived under the prior.

The proposed PASS estimator is similar to the prior LASSO (pLASSO) procedure of Jiang et al. (2016) in that both approaches aim to incorporate prior information into the ${\ell }_1$ penalized estimator in a high-dimensional setting. Nevertheless, the differences are substantial and clear. Jiang et al. (2016) assumed that the prior information was summarized into prediction values and contributed to the likelihood term. In contrast, we use prior information to guide the shrinkage and put them into the penalty term. In this sense, PASS and pLASSO complement each other to some extent. However, as shown in both theory and simulations, putting prior information into the likelihood term tends to lead to the “take it or leave it” phenomenon: the usefulness of the prior information is determined based on the overall effect of all predictors. On the other hand, by putting prior information into the penalty term, the PASS approach provides more flexible control: it is able to scrutinize the individual effect of each predictor. This gained flexibility can result in improved theoretical and numerical performances.

The rest of this paper is organized as follows. We discuss the motivation, an important special scenario and the general methodology in Section 2. We analyze the theoretical properties of the proposed approach in Section 3, and access its finite sample performance via simulation studies in Section 4. Furthermore, we illustrate the practical value of the proposed approach on three real EHR datasets in Section 5. Finally, we conclude this paper with some discussions and extensions in Section 6. All technical proofs and additional numerical results are given in the Supplementary Materials.

2. Methodology

2.1. Setup

We assume that the underlying data consists of $N$ independent and identically distributed (i.i.d.) observations $\left\{{\left(Y_i,S_i,{\mathit{X}}_i^{\top }\right)}^{\top }={\left(Y_i,{\mathbf{W}}_i^{\top }\right)}^{\top },i=1,\dots ,N\right\}$, where $Y_i$ is a binary indicator of the disease status of the $i\text{th}$ patient, $S_i$ is a scalar surrogate variable that is reasonably predictive of $Y_i$ chosen via domain knowledge, and ${\mathit{X}}_i$ is a $p$-dimensional feature vector. Examples of $S_i$ includes the total count of ICD codes or mentions for the disease of interest in clinical notes extracted via natural language processing (NLP). Candidate features $\mathit{X}$ may include the ICD9 code counts for competing diagnosis, lab results, as well as NLP mentions of relevant signs/symptoms, medications and procedures. We may also include various transformations of original features in $\mathit{X}$ to account for non-linear effects. While $\left\{{\mathbf{W}}_i,i=1,\dots ,N\right\}$ is fully observed, $Y_i$ is only observed for a random subset of $n$ patients. Hence the observed data are $𝓛\cup 𝓤$, where without loss of generality, the first $n$ observations are assumed fully observed as $𝓛=\left\{{\left(Y_i,{\mathbf{W}}_i^{\top }\right)}^{\top },i=1,\dots ,n\right\}$, and the rest constitute the unlabeled dataset as $𝓤=\left\{{\mathbf{W}}_i,i=n+1,\dots ,N\right\}$.

Throughout, for a $d$-dimensional vector $\mathit{u}$, the ${\ell }_q$-norm of $v$ is $\parallel \mathit{v}{\parallel }_q={\left(\sum _{j=1}^d{\left|v_j\right|}^q\right)}^{1/q}$. The ${\ell }_{\infty }$-norm of $\mathit{v}$ is $\parallel \mathit{v}{\parallel }_{\infty }={\max }_{1\le j\le d}\left|v_j\right|$. The support of $\mathit{v}$ is $\text{supp}(\mathit{v})=\left\{j:v_j\ne 0\right\}$. If $𝓘$ is a subset of $\{1,\dots ,p\}$, then ${\mathit{v}}_{𝓘}$ denotes a $d$-dimensional vector whose $j\text{th}$ element is $v_j{1}_{j\in 𝓘}$, and $1_B$ is the indicator function for set $B$. The independence between random variables/vectors $\mathit{U}$ and $\mathit{V}$ is written as $\mathit{U}⫫\mathit{V}$. We also denote the negative log-likelihood function associated with the logistic model with $\ell (y,\eta )=-y\eta +\log \left(1+e^{\eta }\right)$.

2.2. Model Assumptions

To predict $Y$ using $\mathbf{W}={\left(S,{\mathit{X}}^{\top }\right)}^{\top }$, we assume

$$\mathrm{Pr}(Y=1\mid \mathbf{W})=\sigma \left({\zeta }_0+S{\gamma }_0+{\mathit{X}}^{\top }{\mathit{\beta }}_0\right)=\sigma \left({\vartheta }_0^{\top }\overrightarrow{\mathbf{W}}\right)\text{with}{\vartheta }_0={\left({\zeta }_0,{\gamma }_0,{\mathit{\beta }}_0^{\top }\right)}^{\top },$$ (𝓜_Y)

where for any vector $\mathbf{w}$, $\overrightarrow{\mathbf{w}}={\left(1,{\mathbf{w}}^{\top }\right)}^{\top }$, and $\sigma (t)=e^t/\left(1+e^t\right)$. To leverage the data in $𝓤$, we further assume a single index model (SIM) for $S\mid \mathit{X}$, i.e. there exists ${\mathit{\alpha }}_0\in {ℝ}^p$ such that

$$S=f\left({\mathit{X}}^{\top }{\mathit{\alpha }}_0,ϵ\right),\text{with some}ϵ⫫\mathit{X}\text{and}f\text{satisfying}\mathbb{E}\left\{f^2\left({\mathit{X}}^{\top }{\mathit{\alpha }}_0,ϵ\right)\right\}<\infty ,$$ (𝓜_S)

where ${\mathit{X}}^{\top }{\mathit{\alpha }}_0$ is a single linear combination of the features $\mathit{X}$ and $f$ is an unknown link function. Here ${\zeta }_0$, ${\gamma }_0$, ${\mathit{\beta }}_0$ and ${\mathit{\alpha }}_0$ are parameters to be estimated where only the direction of ${\mathit{\alpha }}_0$ is identifiable and its norm does not affect our construction introduced below. If ${\mathit{\alpha }}_0$ and ${\mathit{\beta }}_0$ are similar in certain ways, one would expect that the unlabeled dataset $𝓤$ may be used to improve upon the standard supervised estimator for ${\mathit{\beta }}_0$ using $𝓛$ alone. For example, if $S$ is a noisy representation of $Y$ with random measurement error, then it is reasonable and common in the EHR literature (Hong et al., 2019; Zhang et al., 2020, e.g.) to assume

$$\mathit{X}⫫S\mid Y.$$ (𝓒^prior)

Note that a similar conditional independence assumption to (${𝓒}^{\text{prior}}$) was imposed between the input and the pretext target given the label, in the context of self-supervised learning to demonstrate its advantage (Lee et al., 2020). Under (${𝓒}^{\text{prior}}$), we have Proposition 1 with proof given in Supplementary Materials.

Proposition 1.

Under $\left({𝓜}_Y\right)$, $\left({𝓜}_S\right)$, $\left({𝓒}^{\mathit{\text{prior}}}\right)$, and assuming $\mathbb{E}\left(\mathit{X}{\mathit{X}}^{\top }\right)$ is positive–definite, and it holds that: $(C1)$ for any two vectors ${\mathit{a}}_1$, ${\mathit{a}}_2$, $\mathbb{E}\left({\mathit{X}}^{\top }{\mathit{a}}_2\mid {\mathit{X}}^{\top }{\mathit{a}}_1\right)$ is linear in ${\mathit{X}}^{\top }{\mathit{a}}_1$, there exist scalars $k_1$, $k_2\in \mathrm{ℝ}$ such that ${\mathit{\alpha }}_0=k_1{\mathit{\beta }}_0$ and ${\mathit{\alpha }}^{\ast }=k_2{\mathit{\beta }}_0$ where

$$\left({\tau }^{\ast },{\mathit{\alpha }}^{\ast }\right)=\underset{\tau ,\mathit{\alpha }}{\text{arg min}}\mathbb{E}{\left(S-\tau -{\mathit{X}}^{\top }\mathit{\alpha }\right)}^2.$$

Remark 1.

Condition (C1) holds for elliptical distributions including multivariate normal. By Diaconis and Freedman (1984) and Hall and Li (1993), this assumption tends to hold for non-elliptical design when the dimensionality is high. Specifically, one can show that under mild regularity conditions, for two projection vectors ${\mathit{a}}_1$ and ${\mathit{a}}_2$ uniformly randomly drawn from ${\mathbb{S}}^{p-1}=\left\{\mathit{v}\in {\mathrm{ℝ}}^{p-1}:\parallel \mathit{v}{\parallel }_2=1\right\}$, the pair $\left({\mathit{X}}^{\top }{\mathit{a}}_2,{\mathit{X}}^{\top }{\mathit{a}}_1\right)$ weakly converges to a bivariate normal distribution with high probability, and thus $\mathbb{E}\left({\mathit{X}}^{\top }{\mathit{a}}_2\mid {\mathit{X}}^{\top }{\mathit{a}}_1\right)$ is at least approximately linear in ${\mathit{X}}^{\top }{\mathit{a}}_1$; see Theorem 1.1 of Diaconis and Freedman (1984) and equation (1.9) of Hall and Li (1993).

Proposition 1 hinges on the main result of Li and Duan (1989) that when the features $\mathit{X}$ satisfy (C1), the direction of the coefficients of a SIM could be estimated using least squares regression for the response against $\mathit{X}$. It suggests that $𝓤$ can greatly improve the estimation of ${\mathit{\beta }}_0$ under (${𝓒}^{\text{prior}}$) because the phenotype model (${𝓜}_Y$) may be rewritten as $\text{logit Pr}(Y=1\mid \mathbf{W})=\zeta +S\gamma +\rho {\mathit{X}}^{\top }\mathit{\alpha }$ for some $\rho$. Under this model, a simple SS estimator for $\zeta$, $\gamma$ and $\mathit{\beta }$ in (${𝓜}_Y$) can be obtained as $\widehat{\zeta }$, $\widehat{\gamma }$ and $\widehat{\rho }\widehat{\mathit{\alpha }}$, where

$${(\widehat{\zeta },\widehat{\gamma },\widehat{\rho })}^{\top }=\underset{\zeta ,\gamma ,\rho }{\arg \min }\sum _{i=1}^n\ell \left(Y_i,\zeta +\gamma S_i+\rho {\mathit{X}}_i^{\top }\widehat{\mathit{\alpha }}\right),{\left(\widehat{\tau },{\widehat{\mathit{\alpha }}}^{\top }\right)}^{\top }=\underset{\tau ,\mathit{\alpha }}{\arg \min }\sum _{i=1}^N{\left(S_i-\tau -\mathit{X}\mathit{\alpha }\right)}^2.$$

By doing so, the direction of the high dimensional vector $\mathit{\beta }$ is estimated based on the entire $𝓛\cup 𝓤$, and only the parameters ${(\zeta ,\gamma ,\rho )}^{\top }$ are estimated using the small labeled dataset $𝓛$. Hereafter we shall refer to this SS estimator derived under (${𝓒}^{\text{prior}}$) as ${\text{SS}}^{\text{prior}}$.

Nevertheless, ${\text{SS}}^{\text{prior}}$ is only valid when (${𝓒}^{\text{prior}}$) and (C1) holds exactly. Our goal is to develop a more robust SS estimator under (${𝓜}_Y$) and (${𝓜}_S$) that can efficiently exploit $𝓤$ when (${𝓒}^{\text{prior}}$) and (C1) may only hold approximately. In this more general setting, a desirable SS estimator should improve upon the standard supervised estimator when the directions of ${\mathit{\alpha }}_0$ and ${\mathit{\beta }}_0$ are similar in their magnitude and/or support. In addition, it should perform similarly to the supervised estimator when the two directions are not close. We shall now detail our PASS estimation procedure which automatically adapts to different cases as reflected in the observed data.

2.3. Prior Adaptive Semi-Supervised (PASS) Estimator

With $𝓛$ only, a supervised estimator for $\mathit{\beta }$ can be obtained via the standard ${\ell }_1$-penalized regression:

$$\breve{\mathit{\vartheta }}={\left(\breve{\zeta },\breve{\gamma },{\breve{\mathit{\beta }}}^{\top }\right)}^{\top }=\underset{\mathit{\vartheta }}{\arg \min }\frac{1}{n}\sum _{i=1}^n\ell \left(Y_i,{\mathit{\vartheta }}^{\top }{\overrightarrow{\mathbf{W}}}_i\right)+\lambda \parallel \mathit{\beta }{\parallel }_1.$$ (1)

With properly chosen $\lambda$, the consistency and rate of convergence for $\breve{\mathit{\vartheta }}$ has been established (van de Geer, 2008). To improve the estimation of $\mathit{\beta }$ through leveraging $𝓤$, we note that when (${𝓒}^{\text{prior}}$) holds approximately, the magnitude of ${\mathit{\beta }}_0-\rho {\mathit{\alpha }}_0$ is small for some $\rho$, and the support of ${\mathit{\beta }}_0-\rho {\mathit{\alpha }}_0$ is of small size as well.

To incorporate such prior belief on the relationship between ${\mathit{\alpha }}_0$ and ${\mathit{\beta }}_0$, we construct the penalty term

$$\underset{\rho }{\min }\left\{{\lambda }_1\Vert {{\left(\mathit{\beta }-\rho {\mathit{\alpha }}_0\right)}_{{𝓐}_0}\Vert }_1+{\lambda }_2{\Vert {\left(\mathit{\beta }-\rho {\mathit{\alpha }}_0\right)}_{{𝓐}_0^c}\Vert }_1\right\},$$

where ${𝓐}_0=\text{supp}\left({\mathit{\alpha }}_0\right)$, and ${\lambda }_1$, ${\lambda }_2>0$ are tuning parameters. Since ${\left({\mathit{\alpha }}_0\right)}_{{𝓐}_0^c}=\mathbf{0}$, the penalty term is equivalent to

$${\lambda }_1\left\{\underset{\rho }{\min }{\Vert {\left(\mathit{\beta }-\rho {\mathit{\alpha }}_0\right)}_{{𝓐}_0}\Vert }_1\right\}+{\lambda }_2{\Vert {\mathit{\beta }}_{{𝓐}_0^c}\Vert }_1.$$ (2)

The first term in the penalty measures how far $\mathit{\beta }$ is from the closest vector along the ${\mathit{\alpha }}_0$ direction, and hence encourages smaller magnitude of $\mathit{\beta }-\rho {\mathit{\alpha }}_0$. The second term shrinks ${\mathit{\beta }}_{{𝓐}_0^c}$ towards $\mathbf{0}$, which reflects our prior that predictors irrelevant to $S$ are likely to be irrelevant to $Y$ as well. The tuning parameters ${\lambda }_1,{\lambda }_2$ control the strength of the belief imposed. When they are sufficiently large, $\mathit{\beta }$ will be forced to be a multiple of ${\mathit{\alpha }}_0$ and thus it ends up with the same estimator as in the case where (${𝓒}^{\text{prior}}$) holds.

Since we have $N\gg p$ samples to estimate ${\mathit{\alpha }}_0$, we use the adaptive LASSO (ALASSO) penalized least square estimator $\widehat{\mathit{\alpha }}$ (Zou, 2006; Zou and Zhang, 2009), where

$$\widehat{\tau },\widehat{\mathit{\alpha }}=\underset{\tau ,\mathit{\alpha }}{\arg \min }\frac{1}{N}\sum _{i=1}^N{\left(S_i-\tau -{\mathit{X}}_i^{\top }\mathit{\alpha }\right)}^2+\mu \sum _{j=1}^p{\widehat{\omega }}_j\left|{\alpha }_j\right|,$$

where ${\widehat{\omega }}_j={\left|{\widehat{\alpha }}_{\text{init},j}\right|}^{-\nu }$ for some constant $\nu >0$, ${\widehat{\mathit{\alpha }}}_{\text{init}}={\left({\widehat{\alpha }}_{\text{init},1},\dots ,{\widehat{\alpha }}_{\text{init},p}\right)}^{\top }$,

$${\widehat{\tau }}_{\text{init}},{\widehat{\mathit{\alpha }}}_{\text{init}}=\underset{\tau ,\mathit{\alpha }}{\arg \min }\frac{1}{N}\sum _{i=1}^N{\left(S_i-\tau -{\mathit{X}}_i^{\top }\mathit{\alpha }\right)}^2+{\mu }_{\text{init}}\parallel \mathit{\alpha }{\parallel }_1,$$

${\mu }_{\text{init}}$ and $\mu$ are tuning parameters that can be chosen via the cross-validation or Bayesian information criterion (BIC). Here, $\widehat{\mathit{\alpha }}$ is actually an estimator of ${\mathit{\alpha }}^{\ast }$, which has the same direction as ${\mathit{\alpha }}_0$ under the conditions in Proposition 1.

Appending the penalty term (2) to the likelihood and replacing ${\mathit{\alpha }}_0$ with its estimate $\widehat{\mathit{\alpha }}$, we propose to estimate ${\mathit{\vartheta }}_0={\left({\zeta }_0,{\gamma }_0,{\mathit{\beta }}_0^{\top }\right)}^{\top }$ by

$$\widehat{\mathit{\vartheta }}={\left(\widehat{\zeta },\widehat{\gamma },{\widehat{\mathit{\beta }}}^{\top }\right)}^{\top }=\underset{\mathit{\vartheta }}{\arg \min }\frac{1}{n}\sum _{i=1}^n\ell \left(Y_i,\mathit{\vartheta }{\overrightarrow{\mathbf{W}}}_i\right)+{\lambda }_1\left\{\underset{\rho }{\min }{\Vert {(\mathit{\beta }-\rho \widehat{\mathit{\alpha }})}_{\widehat{𝓐}}\Vert }_1\right\}+{\lambda }_2{\Vert {\mathit{\beta }}_{{\widehat{𝓐}}^c}\Vert }_1,$$

where $\widehat{𝓐}=\text{supp}(\widehat{\mathit{\alpha }})$. The estimators can be equivalently obtained as

$$\widehat{\rho },\widehat{\mathit{\vartheta }}=\underset{\rho ,\mathit{\vartheta }}{\arg \min }\frac{1}{n}\sum _{i=1}^n\ell \left(Y_i,\mathit{\vartheta }{\overrightarrow{\mathbf{W}}}_i\right)+{\lambda }_1{\Vert {(\mathit{\beta }-\rho \widehat{\mathit{\alpha }})}_{\widehat{𝓐}}\Vert }_1+{\lambda }_2{\Vert {\mathit{\beta }}_{{\widehat{𝓐}}^c}\Vert }_1$$ (3)

The impact of the tuning parameters ${\lambda }_1,{\lambda }_2$ can be understood from a bias-variance tradeoff viewpoint. When ${\lambda }_j$’s are large, $\widehat{\mathit{\beta }}$ tends to be a multiple of $\widehat{\mathit{\alpha }}$ and thus is an estimator with high bias and low variance. In contrast, when ${\lambda }_j$’s are small, the likelihood term based on the labeled dataset $𝓛$ is the dominant part, and hence $\widehat{\mathit{\beta }}$ will have low bias and high variance. By varying the values of ${\lambda }_j$’s, we are able to obtain a continuum connecting these two extremes. In practice, ${\lambda }_1$ and ${\lambda }_2$ can be chosen via standard data-driven approaches such as cross-validation.

2.4. Computation Details

The minimization in (3) can be solved with standard software for LASSO estimation. Let $\mathit{\delta }=\mathit{\beta }-\rho \widehat{\mathit{\alpha }}$. We can re-parametrize the expression above in terms of $\rho$, $\zeta$, $\gamma$, and $\mathit{\delta }$ as

$$\widehat{\zeta },\widehat{\gamma },\widehat{\rho },\widehat{\mathit{\delta }}=\underset{\zeta ,\gamma ,\rho ,\mathit{\delta }}{\arg \min }\frac{1}{n}\sum _{i=1}^n\ell \left(Y_i,\zeta +S_i\gamma +\rho {\mathit{X}}_i^{\top }\widehat{\mathit{\alpha }}+{\mathit{X}}_i^{\top }\mathit{\delta }\right)+{\lambda }_1\left({\Vert {\mathit{\delta }}_{\widehat{𝓐}}\Vert }_1+\kappa {\Vert {\mathit{\delta }}_{𝓟\setminus \widehat{𝓐}}\Vert }_1\right),$$

where $𝓟=\{1,\dots ,p\}$ and $\kappa ={\lambda }_2/{\lambda }_1$. This is a typical LASSO problem with covariates ${\left(1,S_i,{\mathit{X}}_i^{\top }\widehat{\mathit{\alpha }},{\mathit{X}}_i^{\top }\right)}^{\top }$, parameters ${(\zeta ,\gamma ,\rho ,\mathit{\delta })}^{\top }$, and a weighted ${\ell }_1$ penalty on the parameters. Hence it can be solved by essentially any algorithm for ALASSO fitting. In this paper, we use the R package glmnet (Friedman et al., 2010) to compute $\widehat{\zeta }$, $\widehat{\gamma }$, $\widehat{\rho }$, and $\widehat{\mathit{\delta }}$, and construct the final estimator for ${\mathit{\vartheta }}_0$ as $\widehat{\mathit{\vartheta }}={\left(\widehat{\zeta },\widehat{\gamma },{\widehat{\mathit{\beta }}}^{\top }\right)}^{\top }$ with $\widehat{\mathit{\beta }}=\widehat{\mathit{\delta }}+\widehat{\rho }\widehat{\mathit{\alpha }}$.

3. Theoretical Properties

In this section, we present non-asymptotic risk bounds for the PASS estimator. We also make theoretical comparisons with the supervised LASSO estimator to shed light on when PASS outperforms the LASSO and where such improvement comes from.

3.1. Notations

A random variable $V$ is sub-Gaussian $\left({\tau }^2\right)$ if $\mathbb{E}\{\exp (\lambda |V|)\}\le 2\exp \left({\lambda }^2{\tau }^2/2\right)$ holds for all $\lambda >0$. Throughout, we define

$$\mathit{U}={\left({\mathit{X}}^{\top },1\right)}^{\top },\mathit{K}=\mathbb{E}\left(\mathit{U}{\mathit{U}}^{\top }\right),\mathit{\xi }={\left({\mathit{\alpha }}^{\top },\tau \right)}^{\top },{\mathit{Z}}_{\mathit{\alpha }}={\left({\mathit{X}}^{\top },{\mathit{X}}^{\top }\mathit{\alpha },S,1\right)}^{\top },\mathit{G}=\mathbb{E}\left({\mathit{Z}}_{{\mathit{\alpha }}^{\ast }}{\mathit{Z}}_{{\mathit{\alpha }}^{\ast }}^{\top }\right),$$

$$\mathit{\theta }={\left({\mathit{\delta }}^{\top },\rho ,\gamma ,\zeta \right)}^{\top },\mathit{H}=\mathbb{E}\left[\sigma \left({\mathit{Z}}_{{\mathit{\alpha }}^{\ast }}^{\top }{\mathit{\theta }}_0\right)\left\{1-\sigma \left({\mathit{Z}}_{{\mathit{\alpha }}^{\ast }}^{\top }{\mathit{\theta }}_0\right)\right\}{\mathit{Z}}_{{\mathit{\alpha }}^{\ast }}{\mathit{Z}}_{{\mathit{\alpha }}^{\ast }}^{\top }\right],$$

where ${\mathit{\alpha }}^{\ast }$ is given by ${\left({\mathit{\alpha }}^{\ast \top },{\tau }^{\ast }\right)}^{\top }={\mathit{\xi }}^{\ast }={\text{arg min}}_{\mathit{\xi }}\mathbb{E}{\left(S-{\mathit{U}}^{\top }\mathit{\xi }\right)}^2$, and ${\mathbf{\text{Θ}}}_0=\{\mathit{\theta }:\mathit{\delta }+\rho {\mathit{\alpha }}^{\ast }={\mathit{\beta }}_0,\zeta ={\zeta }_0,\gamma ={\gamma }_0\}$. Denote by ${𝓑}_0=\text{supp}\left({\mathit{\beta }}_0\right)$, ${𝓐}^{\ast }=\text{supp}\left({\mathit{\alpha }}^{\ast }\right)$ and $q^{\ast }=\left|{𝓐}^{\ast }\right|$. We assume ${\Vert {\mathit{\alpha }}^{\ast }\Vert }_2=1$ without loss of generality since ${\mathit{\alpha }}^{\ast }$ is used to recover only the direction of ${\mathit{\beta }}_0$ in SIM and one can change $\rho$ correspondingly to make any $\mathit{\beta }=\mathit{\delta }+\rho {\mathit{\alpha }}^{\ast }$ invariant to ${\Vert {\mathit{\alpha }}^{\ast }\Vert }_2$. Note that under (${𝓜}_Y$), any ${\mathit{\theta }}_0\in {\mathbf{\text{Θ}}}_0$ minimizes $\mathbb{E}\left\{\ell \left(Y,{\mathit{Z}}_{{\mathit{\alpha }}^{\ast }}^{\top }\mathit{\theta }\right)\right\}$, and due to perfect multicollinearity in ${\mathit{Z}}_{{\mathit{\alpha }}^{\ast }}$, ${\mathit{\theta }}_0$ is not unique. However, any ${\mathit{\theta }}_0\in {\mathbf{\text{Θ}}}_0$ corresponds to the unique ${\mathit{\beta }}_0={\mathit{\delta }}_0+{\rho }_0{\mathit{\alpha }}^{\ast }$ and thus ${\mathit{Z}}_{{\mathit{\alpha }}^{\ast }}^{\top }{\mathit{\theta }}_0={\zeta }_0+S{\gamma }_0+{\mathit{X}}^{\top }{\mathit{\beta }}_0={\mathit{\vartheta }}_0^{\top }\overrightarrow{\mathbf{W}}$ is well-defined. Moreover, any quantity depending on ${\mathit{\theta }}_0$ through ${\mathit{Z}}_{{\mathit{\alpha }}^{\ast }}^{\top }{\mathit{\theta }}_0$ is well-defined. Since the main results in this section depend on ${\mathit{\theta }}_0$ solely through ${\mathit{Z}}_{{\mathit{\alpha }}^{\ast }}^{\top }{\mathit{\theta }}_0$, we will use ${\mathit{\theta }}_0$ to represent any $\mathit{\theta }\in {\mathbf{\text{Θ}}}_0$ for simplicity.

For $\mathit{\theta }={\left({\mathit{\delta }}^{\top },\rho ,\gamma ,\zeta \right)}^{\top }$, define $\text{Ω}(\mathit{\theta })={\lambda }_0(|\rho |+|\gamma |+|\zeta |)+{\lambda }_1{\Vert {\mathit{\delta }}_{{𝓐}^{\ast }}\Vert }_1+{\lambda }_2{\Vert {\mathit{\delta }}_{𝓟\setminus {𝓐}^{\ast }}\Vert }_1$, ${\text{Δ}}_{\mathit{\alpha }}=2{\mu }_{\text{init}}{q}^{\ast }/{\phi }^2$ and $\text{Π}(\mathit{\theta })=|\rho |$, where $\phi$ is a constant defined by Assumption (A4) in Section S2.1 of the Supplement Material, and ${\lambda }_0=36B{\{\log (6/ϵ)/n\}}^{1/2}$. To introduce the oracle ${\mathit{\theta }}^{\ast }$, we define the oracle risk function as:

$$\begin{gathered} 𝓔\left(\mathit{\theta },{𝓢}_{+},{𝓢}_{-}\right)=\mathbb{E}\ell \left(Y,{\mathit{Z}}_{{\mathit{\alpha }}^{\ast }}^{\top }\mathit{\theta }\right)-\mathbb{E}\ell \left(Y,{\mathit{Z}}_{{\mathit{\alpha }}^{\ast }}^{\top }{\mathit{\theta }}_0\right) \\ +256\frac{\kappa {\left({𝓢}_{+}\right)}^2\left|{𝓢}_{+}\right|}{\varpi \psi \left({𝓢}_{+}\right)}+8{\lambda }_1{\Vert {\mathit{\theta }}_{{𝓢}_{-}\cap {𝓐}^{\ast }}\Vert }_1+8{\lambda }_2{\Vert {\mathit{\theta }}_{{𝓢}_{-}\cap \left(𝓟\setminus {𝓐}^{\ast }\right)}\Vert }_1+8{\lambda }_1{\text{Δ}}_{\mathit{\alpha }}\text{Π}(\mathit{\theta }), \end{gathered}$$ (4)

where

$$\psi \left({𝓢}_{+}\right)=\underset{\mathit{v}:\text{Ω}\left({\mathit{v}}_{{𝓢}_{-}}\right)\le 3\text{Ω}\left({\mathit{v}}_{{𝓢}_{+}}\right)}{\inf }\frac{{\mathit{v}}^{\top }G\mathit{v}}{{\mathit{v}}_{{𝓢}_{+}}^{\top }{\mathit{v}}_{{𝓢}_{+}}},$$

$$\kappa \left({𝓢}_{+}\right)=\{\begin{array}{ll}{\lambda }_0,& \text{if}{𝓢}_{+}\cap {𝓐}^{\ast }=\text{Ø}\text{and}{𝓢}_{+}\cap \left(𝓟\setminus {𝓐}^{\ast }\right)=\text{Ø}\\ {\lambda }_2,& \text{if}{𝓢}_{+}\cap {𝓐}^{\ast }=\text{Ø}\text{and}{𝓢}_{+}\cap \left(𝓟\setminus {𝓐}^{\ast }\right)\ne \text{Ø}\\ +\infty ,& \text{if}{𝓢}_{+}\cap {𝓐}^{\ast }\ne \text{Ø}\end{array}$$

Define ${\mathit{\theta }}^{\ast }={\left({\mathit{\delta }}^{\ast \top },{\rho }^{\ast },{\gamma }^{\ast },{\zeta }^{\ast }\right)}^{\top }$, ${𝓢}_{+}^{\ast }$ and ${𝓢}_{-}^{\ast }$ as the solution to

$$\underset{\left\{\mathit{\theta },{𝓢}_{+},{𝓢}_{-}\right\}:{𝓢}_{+}\cap {𝓢}_{-}=\varnothing ,{𝓢}_{+}\cup {𝓢}_{-}=\text{supp}(\mathit{\theta })\cup \overline{𝓟},{𝓢}_{+}\supseteq \overline{𝓟},\text{and}{\Vert {\mathit{G}}^{1/2}\left(\mathit{\theta }-{\mathit{\theta }}_0\right)\Vert }_2\le \eta ,}{\arg \min }𝓔\left(\mathit{\theta },{𝓢}_{+},{𝓢}_{-}\right)$$

where $\overline{𝓟}=\{p+1,p+2,p+3\}$, and $\eta$ is a constant as defined by Assumption (A3) in Section S2.1 of the Supplement. Let ${𝓢}^{\ast }={𝓢}_{+}^{\ast }\cup {𝓢}_{-}^{\ast }=\text{supp}\left({\mathit{\theta }}^{\ast }\right)\cup \overline{𝓟}$, ${\kappa }^{\ast }=\kappa \left({𝓢}_{+}^{\ast }\right)$, and ${\mathit{\beta }}^{\ast }={\mathit{\delta }}^{\ast }+{\rho }^{\ast }{\mathit{\alpha }}^{\ast }$. Intuitively, one may view ${𝓢}_{+}$ as the union set of unpenalized predictors and the predictors with large coefficients but not recovered by ${𝓐}^{\ast }$. While ${𝓢}_{-}$ can be viewed as the union set of predictors with small nonzero coefficients and the predictors recovered by ${𝓐}^{\ast }$. Partitioning the support of $\mathit{\theta }$ into ${𝓢}_{+}$ and ${𝓢}_{-}$ is inspired by Bühlmann and Van De Geer (2011, Section 6.2.4), which leads to a refined bound.

3.2. Main result

We first establish the risk bounds for the PASS estimator in the following theorem. Its proof can be found in Section S2 of the Supplementary Materials.

Theorem 1.

For any $ϵ>0$, if the assumptions (A1)–(A8) (introduced in Section S2.1 of the Supplementary Materials) hold, the following inequalities hold simultaneously with probability at least $1-10ϵ$:

$$\begin{array}{ll}Excessrisk:& \mathbb{E}\ell \left(Y,{\mathit{Z}}_{\widehat{\alpha }}^{\top }\widehat{\mathit{\theta }}\right)-\mathbb{E}\ell \left(Y,{\mathit{Z}}_{{\alpha }^{\ast }}^{\top }{\mathit{\theta }}_0\right)\le \text{Ξ,}\end{array}$$

$$\begin{array}{ll}Linearpredictionerror:& \mathbb{E}{\left({\mathit{Z}}_{\widehat{\alpha }}^{\top }\widehat{\mathit{\theta }}-{\mathit{Z}}_{{\alpha }^{\ast }}^{\top }{\mathit{\theta }}_0\right)}^2\le \text{Ξ}/\varpi ,\end{array}$$

$$\begin{array}{ll}Probabilitypredictionerror:& \mathbb{E}{\left\{\sigma \left({\mathit{Z}}_{\widehat{\alpha }}^{\top }\widehat{\mathit{\theta }}\right)-\sigma \left({\mathit{Z}}_{{\alpha }^{\ast }}^{\top }{\mathit{\theta }}_0\right)\right\}}^2\le \text{Ξ}/\varpi ,\end{array}$$

where $\varpi$ is a positive constant defined in (A2), $\text{Ξ}=64𝓔\left({\mathit{\theta }}^{\ast },{𝓢}_{+}^{\ast },{𝓢}_{-}^{\ast }\right)$, and $𝓔$ is an oracle risk function as defined in equation (4).

Remark 2.

As detailed in Section S2.1 of the Supplement Material, Assumptions (A1)–(A8) are imposed on tail behaviour of the regression residuals, regularity of the design matrix, minimum signal strength of ${\mathit{\alpha }}^{\ast }$, sample sizes and rates of the tuning parameters. These assumptions are commonly used conditions in the theoretical literature of LASSO, such as the sub-Gaussian variable condition and the restricted eigenvalue condition; see e.g. van de Geer and Bühlmann (2009); Bickel et al. (2009); Bühlmann and Van De Geer (2011).

Remark 3.

The last term of the risk bound $𝓔\left({\mathit{\theta }}^{\ast },{𝓢}_{+}^{\ast },{𝓢}_{-}^{\ast }\right)$ is of order $O\left({\lambda }_1{\text{Δ}}_{\mathit{\alpha }}\left|{\rho }^{\ast }\right|\right)$, which reflects the estimation error in $\widehat{\mathit{\alpha }}$. Following Lemma S8 in the Supplement, one can show that ${\text{Δ}}_{\alpha }=O_p\left(N^{-1/2}\left|{𝓐}^{\ast }\right|\right)$. All the other terms in $\text{Ξ}$ describe the estimation error in $\widehat{\mathit{\theta }}$ as if $\widehat{\mathit{\alpha }}$ is replaced with ${\mathit{\alpha }}^{\ast }$. When $N$ is sufficiently large, $O\left({\lambda }_1{\text{Δ}}_{\alpha }\left|{\rho }^{\ast }\right|\right)$ is typically negligible relative to other terms. Specifically, if $N\gg n{\left|{𝓐}^{\ast }\right|}^2\log (p)$, $O\left({\lambda }_1{\text{Δ}}_{\alpha }\left|{\rho }^{\ast }\right|\right)=O\left({\left\{Nn\right\}}^{-1/2}\log {(p)}^{1/2}\left|{𝓐}^{\ast }\right|\right)=o\left(n^{-1}\right)$. In general, as long as $N\gg \max (n,p)$ and ${\mathit{\alpha }}^{\ast }$ is not much denser than ${\mathit{\beta }}_0$ as in the typical EHR application cases, $O\left({\lambda }_1{\text{Δ}}_{\alpha }\left|{\rho }^{\ast }\right|\right)$ is dominated by the risk of the supervised LASSO estimator and even the supervised oracle estimator obtained under the knowledge of supp(${\mathit{\beta }}_0$).

To gain a better understanding of how the key quantity $\text{Ξ}$ in Theorem 1 changes with respect to the similarity between the prior information ${\mathit{\alpha }}^{\ast }$ and the target ${\mathit{\beta }}_0$, we shall discuss several specific cases in Section 3.3, based on the risk bound derived in Theorem 1.

3.3. Specific Cases

Following Remark 3, we focus our discussions on the settings where $N$ is sufficiently large such that the last term of the risk bound is negligible. We consider three different scenarios as illustrated in Figure 1: (Case 1) ${\mathit{\alpha }}^{\ast }$ recovers both the support and direction of ${\mathit{\beta }}_0$; (Case 2) ${\mathit{\alpha }}^{\ast }$ almost recovers the support of ${\mathit{\beta }}_0$ but has a substantially different direction from ${\mathit{\beta }}_0$; (Case 3) ${\mathit{\alpha }}^{\ast }$ fails to recover the support of ${\mathit{\beta }}_0$ (let alone its direction) and provides poor information. These three cases depict perfect, good, and poor qualities of the prior information ${\mathit{\alpha }}^{\ast }$ in recovering the support and direction of ${\mathit{\beta }}_0$. Next, we rigorously characterize the three cases by properly specifying the parameters $\rho$, $\mathit{\delta }$, ${𝓢}_{+}$, and ${𝓢}_{-}$, and derive the convergence rate of $\text{Ξ}$, the risk bound of the PASS estimator, based on Theorem 1.

Figure 1:

Examples of the coefficients ${\mathit{\beta }}_0$ and ${\mathit{\alpha }}^{\ast }$ in the three cases of Section 3.3. Labels ${\text{S}}_{+}$, ${\text{S}}_{-}$, and ${\text{A}}^{\ast }$ in the diagrams represent ${\overline{𝓢}}_{+}\setminus \overline{𝓟}$, ${𝓢}_{-}$, and ${𝓐}^{\ast }$ as chosen and defined in Cases 1–3. ${\mathit{\beta }}_0$ and ${\mathit{\alpha }}^{\ast }$ are aligned for comparison of their directions and supports. Presented below are scatter plots for ${\sigma }^{-1}\{\mathrm{Pr}(Y=1\mid S,\mathit{X})\}$ against $S$ of the simulated samples generated under Cases 1–3.

Case 1 (presented in the left panel): ${\mathit{\alpha }}^{\ast }$ recovers both the support and direction of ${\mathit{\beta }}_0.{\sigma }^{-1}\{\mathrm{Pr}(Y=1\mid S,\mathit{X})\}$ shows strong collinearity with $S$. PASS largely outperforms supervised LASSO and has the same convergence rate as the low dimensional regression.

Case 2 (middle): ${\mathit{\alpha }}^{\ast }$ (nearly) recovers the support but not the direction ${\mathit{\beta }}_0.{\sigma }^{-1}\{\mathrm{Pr}(Y=1\mid S,\mathit{X})\}$ shows moderate collinearity with $S$. PASS still outperforms both supervised LASSO and pLASSO in terms of convergence rate.

Case 3 (right): ${\mathit{\alpha }}^{\ast }$ fails to recover the support of ${\mathit{\beta }}_0.{\sigma }^{-1}\{\mathrm{Pr}(Y=1\mid S,\mathit{X})\}$ shows weak collinearity with $S$. PASS is of the same convergence rate as supervised LASSO.

Case 1.

Let $\overline{\rho }={\min }_{\rho }{\Vert {\mathit{\beta }}_0-\rho {\mathit{\alpha }}^{\ast }\Vert }_1$, $\overline{\mathit{\delta }}={\mathit{\beta }}_0-\overline{\rho }{\mathit{\alpha }}^{\ast }$, $\overline{\mathit{\theta }}={\left({\overline{\mathit{\delta }}}^{\top },\overline{\rho },{\gamma }_0,{\zeta }_0\right)}^{\top }$, ${\overline{𝓢}}_{+}=\overline{𝓟}$ and ${\overline{𝓢}}_{-}=\text{supp}\left({\mathit{\delta }}_0\right)$. If ${\mathit{\alpha }}^{\ast }$ successfully recovers the support and direction of ${\mathit{\beta }}_0$ (see the left panel in Figure 1), ${\overline{𝓢}}_{-}\approx \text{Ø}$ and ${\Vert {\mathit{\delta }}_0\Vert }_1\approx 0$. Since ${\Vert G^{1/2}\left(\overline{\mathit{\theta }}-{\mathit{\theta }}_0\right)\Vert }_2=0$ and ${\overline{𝓢}}_{+}\cap {𝓐}^{\ast }=\text{Ø}$, we have $\text{Ξ}=O\left\{𝓔\left(\overline{\mathit{\theta }},{\overline{𝓢}}_{+},{\overline{𝓢}}_{-}\right)\right\}$ by the definition of ${\mathit{\theta }}^{\ast }$. Hence by Theorem 1, the excess risk of $\widehat{\mathit{\theta }}$

$$\text{Ξ}=O_p\left({\lambda }_0^2+{\lambda }_1{\Vert {\overline{\mathit{\delta }}}_{{\overline{𝓢}}_{-}\cap {𝓐}^{\ast }}\Vert }_1+{\lambda }_2{\Vert {\overline{\mathit{\delta }}}_{{\overline{𝓢}}_{-}\cap {𝓐}^{\ast c}}\Vert }_1\right)\approx O_p\left({\lambda }_0^2\right)=O_p\left(n^{-1}\right),$$

recalling that ${\lambda }_0=O\left(n^{-1/2}\right)$.

As a standard result (Negahban et al., 2009), the rate of the excess risk of the supervised LASSO estimator is either $O_p\left\{n^{-1}\log (p)\left|{𝓑}_0\right|\right\}$ or $O\left\{n^{-1/2}\log {(p)}^{1/2}{\Vert {\mathit{\beta }}_0\Vert }_1\right\}$. These two rate bounds are established under different sparsity norms of ${\mathit{\beta }}_0$, and generally comparable, e.g. when order of average magnitude of the non-zero entries in ${\mathit{\beta }}_0$ is $n^{-1/2}\log {(p)}^{1/2}$. In comparison with them, $O_p\left(n^{-1}\right)$, the risk rate of PASS in Case 1, is much more refined. Further, $O_p\left(n^{-1}\right)$ dis actually the rate of the estimator of a low (fixed) dimensional logistic regression. Thus, if ${\mathit{\beta }}_0$ is very close to a multiple of ${\mathit{\alpha }}^{\ast }$, PASS could outperform the vanilla LASSO and be comparable with a low dimensional regression in terms of the convergence rate. This big gain is owing to the use of $N$ unlabeled dataset to obtain the direction of ${\mathit{\beta }}_0$, and thus reduce the high dimensional regression to a low dimensional one where only the intercept and the scalar of ${\mathit{\beta }}_0$ need to be estimated.

Case 2.

Consider the same choice of $\overline{\mathit{\theta }}$, ${\overline{𝓢}}_{+}$ and ${\overline{𝓢}}_{-}$ as in Case 1. If ${\mathit{\alpha }}^{\ast }$ recovers the support but not the direction of ${\mathit{\beta }}_0$ (see the middle of Figure 1), we will only have ${\Vert {\overline{\mathit{\delta }}}_{{\overline{𝓢}}_{-}\cap {𝓐}^{\ast c}}\Vert }_1\approx 0$ but not ${\Vert {\mathit{\delta }}_0\Vert }_1\approx 0$. Then by Theorem 1, the excess risk of PASS is

$$\text{Ξ}=O_p\left({\lambda }_0^2+{\lambda }_1{\Vert {\overline{\mathit{\delta }}}_{{\overline{𝓢}}_{-}\cap {𝓐}^{\ast }}\Vert }_1+{\lambda }_2{\Vert {\overline{\mathit{\delta }}}_{{\overline{𝓢}}_{-}\cap {𝓐}^{\ast c}}\Vert }_1\right)\approx O_p\left\{n^{-1/2}\log {\left(q^{\ast }\right)}^{1/2}{\Vert {\overline{\mathit{\delta }}}_{\overline{𝓢}-\cap {𝓐}^{\ast }}\Vert }_1\right\},$$

recalling that ${\lambda }_1=O\left\{n^{-1/2}\log {\left(q^{\ast }\right)}^{1/2}\right\}$.

In Case 2, the convergence rate of the excess risk of PASS is still better than that of the supervised LASSO estimator when $q^{\ast }\ll p$:

$$O\left\{n^{-1/2}\log {\left(q^{\ast }\right)}^{1/2}{\Vert {\overline{\mathit{\delta }}}_{{\overline{𝓢}}_{-}\cap {𝓐}^{\ast }}\Vert }_1\right\}\ll O\left\{n^{-1/2}\log {(p)}^{1/2}{\Vert {\mathit{\beta }}_0\Vert }_1\right\},$$

by ${\Vert {\overline{\mathit{\delta }}}_{{\overline{𝓢}}_{-}\cap {𝓐}^{\ast }}\Vert }_1\le {\min }_{\rho }{\Vert {\mathit{\beta }}_0-\rho {\mathit{\alpha }}^{\ast }\Vert }_1\le {\Vert {\mathit{\beta }}_0\Vert }_1$. Namely, if ${\mathit{\alpha }}^{\ast }$ might not recover the direction of ${\mathit{\beta }}_0$ very well but the prior information ${𝓐}^{\ast }=\text{supp}\left({\mathit{\alpha }}^{\ast }\right)$ is sparse and covers supp (${\mathit{\beta }}_0$) successfully, which is reflected as ${\overline{𝓢}}_{+}=\overline{𝓟}$, the PASS estimator still benefits from the prior information. This is because recovering the support of ${\mathit{\beta }}_0$ reduces the dimensionality of the empirical errors needed to be controlled from $p$ to $q^{\ast }=\left|{𝓐}^{\ast }\right|$. In this case, it is also interesting to compare the proposed PASS estimator with the prior LASSO (pLASSO) procedure of Jiang et al. (2016). When supp (${\mathit{\alpha }}^{\ast }$) and supp (${\mathit{\beta }}_0$) are close but the directions of ${\mathit{\alpha }}^{\ast }$ and ${\mathit{\beta }}_0$ are quite different, the pLASSO procedure is unable to utilize this information and will only result in the same convergence rate as supervised LASSO, as shown to be essentially slower than that of PASS.

Case 3.

Let $\overline{\rho }=0$, $\overline{\mathit{\delta }}={\mathit{\beta }}_0$, $\overline{\mathit{\theta }}={\left({\overline{\mathit{\delta }}}^{\top },\overline{\rho },{\gamma }_0,{\zeta }_0\right)}^{\top }$, ${\overline{𝓢}}_{+}=\overline{𝓟}\cup \left({𝓑}_0\setminus {𝓐}^{\ast }\right)$ and ${\overline{𝓢}}_{-}={𝓑}_0\setminus {\overline{𝓢}}_{+}$. If ${\mathit{\alpha }}^{\ast }$ fails to recover the support of ${\mathit{\beta }}_0$, i.e. ${𝓐}^{\ast }\cap {𝓑}_0\approx \text{Ø}$ and ${\Vert {\mathit{\beta }}_{0,{𝓐}^{\ast }\cap {𝓑}_0}\Vert }_1\approx 0$, we have ${\Vert {\overline{\mathit{\delta }}}_{{\overline{𝓢}}_{-}}\Vert }_1\le {\Vert {\overline{\mathit{\delta }}}_{{𝓐}^{\ast }}\Vert }_1\approx {\Vert {\mathit{\beta }}_{0,{𝓐}^{\ast }\cap {𝓑}_0}\Vert }_1\approx 0$. Then again using Theorem 1,

$$\text{Ξ}=O\left({\lambda }_2^2\left|{\overline{𝓢}}_{+}\right|+{\lambda }_1{\Vert {\overline{\mathit{\delta }}}_{{\overline{𝓢}}_{-}\cap {𝓐}^{\ast }}\Vert }_1+{\lambda }_2{\Vert {\overline{\mathit{\delta }}}_{{\overline{𝓢}}_{-}\cap {𝓐}^{\ast c}}\Vert }_1\right)\approx O_p\left\{n^{-1}\log (p)\left|{𝓑}_0\right|\right\},$$

recalling that ${\lambda }_2=O\left\{n^{-1/2}\log {(p)}^{1/2}\right\}$.

In Case 3, the excess risk of the PASS is of the same order as that of supervised LASSO. Therefore the PASS approach is robust against low-quality prior information that recovers neither the direction nor the support of ${\mathit{\beta }}_0$. This benefit is a result of using a data-adaptive parameter $\rho$ to control the influence of the prior information on the estimator.

4. Simulation Studies

4.1. Main setups

We conducted extensive simulation studies to examine the finite-sample performance of the PASS estimator and to compare it with existing approaches. We first considered the case where the logistic model for $Y\mid S,\mathit{X}$ is correctly specified, $S\mid \mathit{X}$ follows an SIM, and $\mathit{X}$ is near elliptical, but the similarity between ${\mathit{\alpha }}_0$ and ${\mathit{\beta }}_0$ varies. Since EHR features are often zero inflated and skewed count variables, we generated ${\mathit{X}}_{500\times 1}$ from

$${\mathit{X}}_i=h\left({\mathit{Z}}_i\right),{\mathit{Z}}_i\sim N\left(\mathbf{0},{\mathbf{\text{Σ}}}_{\mathit{Z}}\right),h(t)=\log \left(1+\left[e^t\right]\right),$$

where $[u]$ denotes the integer nearest to $u$, ${\mathbf{\text{Σ}}}_{\mathit{Z}}={\left({\sigma }_{i,j}\right)}_{i,j=1}^p$ and ${\sigma }_{i,j}=4{(0.5)}^{|i-j|}$. Here $\left[e^{Z_{ij}}\right]$ mimics the skewed raw EHR feature, which is typically transformed via $t\to \log (1+t)$ prior to model fitting. We then generated the surrogate $S$ from a SIM of $\mathit{X}$:

$$S_i=h\left(1+{\mathit{X}}_i^{\top }{\mathit{\alpha }}_0+{ϵ}_i\right),\text{with}{ϵ}_i\sim N\left(0,2^2\right).$$

Following the model assumption (${𝓜}_Y$), the disease status $Y_i$ was generated from

$${\sigma }^{-1}\left\{\text{Pr}\left(Y_i=1\mid {\mathbf{W}}_i\right)\right\}=-4+0.5S_i+{\mathit{X}}_i^{\top }{\mathit{\beta }}_0.$$

To mimic different qualities of the prior information one could encounter in practice, we design six scenarios with different similarities between the true ${\mathit{\beta }}_0$ and ${\mathit{\alpha }}_0$:

$$\begin{array}{lll}\text{I:}& {\mathit{\alpha }}_0={\left({\mathit{a}}_1^{\top },{\mathit{a}}_2^{\top },{\mathbf{0}}_{p-10}^{\top }\right)}^{\top },& {\mathit{\beta }}_0=1.5{\left({\mathit{a}}_1^{\top },{\mathit{a}}_2^{\top },{\mathbf{0}}_{p-10}^{\top }\right)}^{\top };\end{array}$$

$$\begin{array}{lll}\text{II:}& {\mathit{\alpha }}_0={\left({\mathit{a}}_1^{\top },{\mathit{a}}_2^{\top },{\mathbf{0}}_{p-10}^{\top }\right)}^{\top },& {\mathit{\beta }}_0=1.5{\left({\mathit{a}}_1^{\top }+{\mathit{d}}_1^{\top },{\mathit{a}}_2^{\top }+{\mathit{d}}_2^{\top },{\mathbf{0}}_{p-10}^{\top }\right)}^{\top };\end{array}$$

$$\begin{array}{lll}\text{III:}& {\mathit{\alpha }}_0={\left({\mathit{a}}_1^{\top },{\mathit{a}}_2^{\top },{\mathit{a}}_2^{\top },{\mathit{a}}_2^{\top },{\mathbf{0}}_{p-20}^{\top }\right)}^{\top },& {\mathit{\beta }}_0=1.5{\left({\mathit{a}}_1^{\top }+{\mathit{d}}_1^{\top },{\mathit{a}}_2^{\top }+{\mathit{d}}_2^{\top },{\mathbf{0}}_{p-10}^{\top }\right)}^{\top };\end{array}$$

$$\begin{array}{lll}\text{IV:}& {\mathit{\alpha }}_0={\left({\mathit{a}}_1^{\top },{\mathbf{0}}_{p-5}^{\top }\right)}^{\top },& {\mathit{\beta }}_0=1.5{\left({\mathit{a}}_1^{\top }+{\mathit{d}}_1^{\top },{\mathit{a}}_2^{\top }+{\mathit{d}}_2^{\top },{\mathbf{0}}_{p-10}^{\top }\right)}^{\top };\end{array}$$

$$\begin{array}{lll}\text{V}:& {\mathit{\alpha }}_0={\left({\mathit{a}}_1^{\top },{\mathit{a}}_2^{\top },{\mathbf{0}}_{p-10}^{\top }\right)}^{\top },& {\mathit{\beta }}_0=1.5{\left({\mathit{a}}_2^{\top },{\mathit{a}}_1^{\top },{\mathbf{0}}_{p-10}^{\top }\right)}^{\top };\end{array}$$

$$\begin{array}{lll}\text{VI}:& {\mathit{\alpha }}_0={\left({\mathit{a}}_1^{\top },{\mathit{a}}_2^{\top },{\mathbf{0}}_{p-10}^{\top }\right)}^{\top },& {\mathit{\beta }}_0=1.5{\left({\mathit{a}}_2^{\top },{\mathbf{0}}_5,{\mathit{a}}_1^{\top },{\mathbf{0}}_{p-15}^{\top }\right)}^{\top }.\end{array}$$

where

$$\begin{array}{ll}{\mathit{a}}_1={(0.5,1,-0.8,0.6,0.2)}^{\top },& {\mathit{d}}_1={(-0.05,-0.5,1.4,0.5,-0.6)}^{\top },\end{array}$$

$$\begin{array}{ll}{\mathit{a}}_2={(0.1,-0.2,-0.2,0.2,0.7)}^{\top },& {\mathit{d}}_2={(0.02,0.05,0.02,-0.02,-0.05)}^{\top },\end{array}$$

Our specifications of ${\mathit{\beta }}_0$ and ${\mathit{\alpha }}_0$ are motivated by the three key specific cases introduced in Section 3.3 and illustrated in Figure 1. Scenario I is the ideal case where ${\mathit{\beta }}_0$ and ${\mathit{\alpha }}_0$ have identical direction. In Scenario II, most of the components of ${\mathit{\beta }}_0$ differ slightly from a scalar multiple of ${\mathit{\alpha }}_0$, while a few components differ substantially. Scenarios I and II are designed to examine the performance of PASS estimator when the prior information is highly or somewhat reliable. In Scenario III, ${\mathit{\alpha }}_0$ is denser than ${\mathit{\beta }}_0$ and contains quite a few weak signals. On the contrary, in Scenario IV ${\mathit{\beta }}_0$ is denser than ${\mathit{\alpha }}_0$. In Scenario V, the magnitude of ${\mathit{\alpha }}_0$ and ${\mathit{\beta }}_0$ are quite different, whereas they still share the same support. Scenarios III, IV and V are designed to examine the performance of PASS estimator with respect to different degrees of accuracy of the support information. In Scenario VI, both the magnitude and the support of ${\mathit{\alpha }}_0$ and ${\mathit{\beta }}_0$ differs substantially, which means the unlabeled dataset provides little information. This scenario allows us to see whether the PASS estimator is robust against unreliable prior information. See Figure 2 for a visualization of ${\mathit{\beta }}_0$ and $\rho {\mathit{\alpha }}_0$ across different scenarios.

Figure 2:

Supports and values of the coefficients ${\mathit{\beta }}_0$ and $1.5{\mathit{\alpha }}_0$ under Scenarios I–VI introduced in Section 4.1. Only those indices $j$ satisfying ${\mathit{\beta }}_{0,j}\ne 0$ or ${\mathit{\alpha }}_{0,j}\ne 0$ are shown in the plots.

We compare PASS to following existing methods: (1) supervised LASSO penalized logistic regression with $n$ training samples (${\text{LASSO}}_n$); (2) supervised ALASSO penalized logistic regression with n training samples, denoted by ${\text{ALASSO}}_n$; (3) the ${\text{SS}}^{\text{prior}}$ estimator as described in section 2.2; and (4) two variants of pLASSO estimators as proposed in Jiang et al. (2016). For pLASSO, we fit a penalized logistic model with an LASSO penalty imposed on predictors outside supp $(\widehat{\mathit{\alpha }})$, as in equation (8) of Jiang et al. (2016), and then use the predicted probability from that model as $Y_i^{\text{p}}$ in equation (7) of Jiang et al. (2016), denoted by pLASSO¹; (2) use the predicted probability given by the ${\text{SS}}^{\text{prior}}$ approach as $Y_i^{\text{p}}$ in equation (7) of their paper, denoted by pLASSO².

Throughout, we let $N=10000$ and let $\nu =1$ in the ALASSO weights. We use Bayesian information criterion (BIC) to select ${\mu }_{\text{init}}$ and $\mu$ in the estimation of $\mathit{\alpha }$ due to large $N$, and use 10-fold cross-validation to select ${\lambda }_1$, ${\lambda }_2$ for the estimation of $\mathit{\beta }$, so that the phenotype model is tuned towards prediction performance. We quantify the average prediction performance of the estimated linear score, ${\tilde{\mathit{\vartheta }}}^{\top }\overrightarrow{\mathbf{W}}$, with $\tilde{\mathit{\vartheta }}$ obtained via different methods in an independent test dataset with size 10000. For each choice of ${\tilde{\mathit{\vartheta }}}^{\top }\overrightarrow{\mathbf{\text{W}}}$, we consider the area under the receiver operating characteristic curve (AUC) for classifying $Y$, the excess risk (ER) as defined in Section 3, and the mean squared error of the predicted probabilities (MSE-P) which is the mean squared differences between the predicted probability and the true probability. We summarize results based on 1000 simulated datasets for each configuration.

In Figures 3, we compare prediction measures for estimators obtained with $n=100$. In Scenario I where the directions of ${\mathit{\beta }}_0$ and ${\mathit{\alpha }}_0$ coincide, the ${\text{SS}}^{\text{prior}}$ approach performs the best as expected, yet the proposed PASS method attained very similar accuracy followed by pLASSO² which performed only slightly worse. When the directions of ${\mathit{\beta }}_0$ and ${\mathit{\alpha }}_0$ are somewhat different as in Scenario II, the ${\text{SS}}^{\text{prior}}$ and the pLASSO estimators deteriorated quickly. In contrast, the PASS estimator maintains high accuracy and outperforms all competing estimators substantially. We observe qualitatively similar patterns for Scenarios III and IV under which ${\mathit{\alpha }}_0$ and ${\mathit{\beta }}_0$ have somewhat different support. No matter whether ${\mathit{\alpha }}_0$ is denser than ${\mathit{\beta }}_0$ as in Scenario IV, or ${\mathit{\beta }}_0$ is denser than ${\mathit{\alpha }}_0$ as in Scenario V, the PASS method consistently outperforms the supervised estimators. Additionally, the performances of the ${\text{SS}}^{\text{prior}}$ and pLASSO approaches are not quite satisfactory. In Scenario V, ${\mathit{\beta }}_0$ and ${\mathit{\alpha }}_0$ have the same support but are quite different in terms of magnitude. The proposed method managed to utilize the same-support information, whereas the pLASSO approaches failed to do so. Finally, the goal of Scenario VI is to examine the robustness of the methods when ${\mathit{\beta }}_0$ and ${\mathit{\alpha }}_0$ differs a lot, possibly due to the use of an inappropriate surrogate. The PASS estimator performs similarly to the supervised estimators, indicating that our procedure is indeed adaptive to how well the data supports the prior assumption. Across all scenarios, the ALASSO approach performs slightly worse than LASSO, possibly due to the presence of some small nonzero coefficients in ${\mathit{\beta }}_0$.

Figure 3:

AUC (left), ER (middle) and MSE-P (right) evaluated on the test set for simulation studies under Scenarios I–VI. Outliers are not drawn. Mean performance of the PASS approach are marked using dashed lines for ease of comparison. The size of the labeled dataset is fixed at $n=100$.

In Figure 4, we present the AUC, ER and MSE-P of the PASS estimator trained with $n=100$ and the supervised LASSO estimator with varying label size. In Scenario I where the prior assumption holds exactly, PASS₁₀₀, the PASS approach with 100 labeled samples, even outperforms LASSO₄₀₀, the LASSO approach with 400 labeled samples. When the prior assumption holds approximately as in Scenarios II through V, PASS₁₀₀ consistently outperforms LASSO₁₅₀, and achieves similar performance as LASSO₂₀₀, which requires twice as many labels. Finally, in Scenario VI where the prior information is highly inaccurate, the PASS method maintains comparable performance against LASSO₁₀₀.

Figure 4:

AUC (left), ER (middle) and MSE-P (right) evaluated on the test set for simulation studies under Scenarios I–VI. Outliers are not drawn. Mean performance of the PASS approach are marked using red dash lines for ease of comparison. The size of the labeled dataset is $n=100$ for PASS, while it varies for LASSO, as indicated in the subscripts.

4.2. Efficiency and Robustness Evaluations under Mis-specifications

We conducted simulation studies under three additional scenarios to further investigate the efficiency and robustness of PASS when the model assumptions and elliptical design assumptions are violated. We again set $p=500$ and generate ${\mathit{X}}_i=2\text{Φ}\left({\mathit{Z}}_i\right)-1$, where $\text{Φ}(\cdot )$ is the cumulative distribution function of the standard normal, ${\mathit{Z}}_i={\left(Z_{i1},\dots ,Z_{ip}\right)}^{\top }\sim N\left(\mathbf{0},{\mathbf{\text{Σ}}}_{\mathit{Z}}^{\prime }\right)$, ${\mathbf{\text{Σ}}}_{\mathit{Z}}^{\prime }={\left({\sigma }_{i,j}^{\prime }\right)}_{i,j=1}^p$, ${\sigma }_{i,j}^{\prime }={(0.5)}^{|i-j|}$. If $i=j$ or both $i$ and $j$ are $\le 20$ or both $i$ and $j$ are $>20$ and ${\sigma }_{i,j}^{\prime }=0$ otherwise. We make ${\mathbf{\text{Σ}}}_{\mathit{Z}}^{\prime }$ block-diagonal for the convenience of obtaining the population solution of $\mathit{\beta }$ and $\mathit{\alpha }$ through the best logistic or least square approximation under model mis-specifications. In real EHR studies, a paradigm of data generation is that the features $\mathit{X}$, e.g. some genetic variants, precedes the disease status $Y$, and $Y$ precedes some clinical surrogate $S$, e.g. the count of ICD codes associated with the disease. To mimic this scenario, we generated $Y_i$ and $S_i$ from the following models:

$$\begin{array}{ll}{Y}_i=I\left\{\left(0.8,1,-1,0.8,0.4,{\mathbf{0}}_{p-5}^{\top }\right){\mathit{X}}_i+{ϵ}_{yi}\ge 0\right\},& {ϵ}_{yi}\sim N(0,1),\end{array}$$

$$\begin{array}{ll}{S}_i=\mu Y_i+{\mathit{\eta }}_1^{\top }{\mathit{X}}_i+Y_i{\mathit{\eta }}_2^{\top }{\mathit{X}}_i+{ϵ}_{si},& {ϵ}_{si}\sim N(0,1).\end{array}$$

Assumptions (${𝓒}^{\text{prior}}$) and (${𝓜}_S$) hold when ${\mathit{\eta }}_1={\mathit{\eta }}_2=\mathbf{0}$, and would be severely violated when ${\mathit{\eta }}_1$ and ${\mathit{\eta }}_2$ are large. We design three scenarios with ${\mathit{\eta }}_1$ and ${\mathit{\eta }}_2$ representing different degrees of violation on the surrogate assumptions:

1. $\mu =1,\text{and}{\mathit{\eta }}_1={\mathit{\eta }}_2=\mathbf{0};$

2. $\mu =1.5,{\mathit{\eta }}_1={\left({\mathit{a}}_3^{\top },{\mathbf{0}}_{p-5}^{\top }\right)}^{\top },\text{and}{\mathit{\eta }}_2={\left({\mathit{d}}_3^{\top },{\mathbf{0}}_{p-5}^{\top }\right)}^{\top };$

3. $\mu =2,{\mathit{\eta }}_1={\left({\mathit{a}}_3^{\top },{\mathit{a}}_3^{\top },{\mathit{a}}_3^{\top },{\mathbf{0}}_{p-15}^{\top }\right)}^{\top },\text{and}{\mathit{\eta }}_2={\left({\mathit{d}}_3^{\top },{\mathit{d}}_3^{\top },{\mathit{d}}_3^{\top },{\mathbf{0}}_{p-15}^{\top }\right)}^{\top },$

where ${\mathit{a}}_3={(0.6,-0.4,0.4,0.5,-0.5)}^{\top }$ and ${\mathit{d}}_3={(0.3,0.4,0.6,-0.5,-0.5)}^{\top }$. Here $\mu$ depict the marginal effect of $Y_i$ on $S_i$, and are set to make the AUC of target models at a similar level across the three scenarios. Across all scenarios, $\text{Pr}\left(Y_i=1\mid S_i,{\mathit{X}}_i\right)$ is no longer a parametric logistic model, i.e. (${𝓜}_Y$) is misspecified. Our goal is to estimate the limiting coefficients ${\zeta }_0$, ${\gamma }_0$, ${\mathit{\beta }}_0$ defined as the minimizor of $\mathbb{E}\ell \left(Y_i,\zeta +\gamma S_i+{\mathit{X}}_i^{\top }\mathit{\beta }\right)$. Benchmark methods, and their implementation, tuning, and evaluation procedures are the same as in Section 4.1, except that we implement supervised LASSO with $n$ ranging from 100 to 700.

In Figure 5, we present AUC, ER and MSE-P of the methods under Scenarios i–iii. In Scenario i, PASS has similar performances as the semi-supervised benchmarks ${\text{SS}}^{\text{prior}}$ and pLASSO, and all the semi-supervised estimators significantly outperform the two supervised estimators since (${𝓒}^{\text{prior}}$) holds and ${\mathit{\alpha }}^{\ast }$ basically recovers the direction of ${\mathit{\beta }}_0$ well. Among the semi-supervised estimators, the ${\text{SS}}^{\text{prior}}$ and pLASSO² estimators have a slight advantage with a smaller variation than expected since both heavily rely on the prior information which is of high quality in this setting. In Scenario ii, the key assumption (${𝓒}^{\text{prior}}$) is violated, which drastically impacts the performance of ${\text{SS}}^{\text{prior}}$ and pLASSO². On the other hand, PASS and pLASSO¹ still effectively leverage the imperfect information from ${\mathit{\alpha }}^{\ast }$ to approximately recover the support of ${\mathit{\beta }}_0$, and thus outperform ${\text{SS}}^{\text{prior}}$, pLASSO², and the supervised methods. In Scenario iii, ${\mathit{\eta }}_1$ and ${\mathit{\eta }}_2$ become denser than those in Scenario ii. This can make the recovery of supp (${\mathit{\beta }}_0$) using supp (${\mathit{\alpha }}^{\ast }$) less accurate, and interestingly, PASS outperforms all methods including pLASSO¹ that also leverages supp (${\mathit{\alpha }}^{\ast }$). In all the three scenarios, PASS significantly outperforms supervised LASSO using the same number or even 2–3 more times the number of samples, which displays a large gain of using the unlabelled dataset to assist the regression. Finally, the results demonstrate that our method can still efficiently leverage the prior information from $S$ in estimating the target parameters when $S\mid Y,\mathit{X}$ highly depends on $\mathit{X}$ so ${𝓒}^{\text{prior}}$ is violated, (${𝓜}_Y$) is misspecified, and the design is non-elliptical.

Figure 5:

Evaluation metrics on the test set for simulation studies under Scenarios i–iii introduced in Section 4.2. Outliers are not drawn. Mean performance of the PASS approach are marked using red dash lines for ease of comparison. On the left panel, we present the evaluation metrics of all methods for comparison when $n=100$. On the right panel, we compare the performance of PASS when $n=100$ with supervised LASSO obtained using labelled samples with various $n$ (from 100 to 700).

5. Application to EHR Phenotying

We examine the performance of PASS along with other approaches in three real world EHR phenotyping studies with the goal of developing classification models for the diseases of interest. All studies are performed at a large tertiary hospital system with EHR spanning over multiple decades. Each study has $n_0$ labeled observations for algorithm training and validation. We consider three choices of training size $n$ no more than $n_0/2$ in all examples. First, we randomly split the labelled samples into four folds of equal sizes. Then we pick each fold as the validation set, sample $n$ training labels from the other three folds for 20 times, train and validate the algorithms, and finally average the evaluation metrics and their standard errors over the validation results on the four folds. We replicate this procedure 10 times and report the average performance.

Data Example 1 (CAD Phenotyping).

The goal of this study is to identify patients with coronary artery disease (CAD) based on their EHR features. The study cohort consists of $N=4164$ patients, out of which a random subset of $n_0=181$ patients have their true CAD status annotated via chart review by domain experts. We use the sum of the counts for the CAD ICD code and NLP mention of CAD as the surrogate. There are $p=585$ additional EHR features consisting of the total count of all ICD codes as a healthcare utilization measure, 10 ICD codes related to CAD, and 574 NLP variables. For the size of training labels, we consider $n=50,70,90$. This de-identified dataset has been analyzed in previous studies (Zhang et al., 2019, e.g.) and is publicly available online: https://celehs.github.io/PheCAP/articles/example2.html.

Data Example 2 (RA Phenotyping).

Similar to the CAD phenotyping study, the goal is to identify patients with rheumatoid arthritis (RA) based on their EHR features. There are $N=46114$ patients in total and out of which, $n_0=435$ patients have their RA status annotated. Again, we choose the sum of the ICD code and NLP mention of RA as the surrogate. The $p=924$ additional EHR features consist of the healthcare utilization and 923 NLP variables potentially predictive of RA. For the size of training labels, we consider $n=50,125,200$.

Data Example 3 (Depression Phenotyping).

The goal is to identify patients with depression based on their codified EHR features. There are $N=9474$ patients in total and $n_0=236$ labeled observations. The surrogate is chosen as the counts of depression ICD code. There are $p=231$ additional EHR features, including the healthcare utilization and 230 codified EHR features on depression related medication prescriptions, laboratory tests and ICD codes. For the size of training labels, we consider 50,85,120.

In the three data examples, $N$ is significantly larger than $p$ with $N/\max (p,n)$ being approximately 7 for CAD, 50 for RA, and 41 for Depression. In all the three studies, we apply $x\to \log (1+x)$ transformation for all count variables. Also, since patients with higher healthcare utilization tend to have higher counts of most features, we orthogonalize all features against the healthcare utilization before regression fitting. Since ${\mathit{\vartheta }}_0$ is unknown in applications, we quantify the performance of an estimator $\tilde{\mathit{\vartheta }}$ based on the AUC and Brier skill score (BSS) of $\sigma \left({\tilde{\mathit{\vartheta }}}^{\top }\mathbf{W}\right)$ for predicting $Y$, where the BSS is defined as $1-{\widehat{\mathbb{E}}}_v[\{Y-{\sigma \left({\tilde{\mathit{\vartheta }}}^{\top }\mathbf{W}\right)\}}^2]/{\widehat{\mathbb{E}}}_v\left[{\left\{Y-{\widehat{\mathbb{E}}}_v(Y)\right\}}^2\right]$, and ${\widehat{\mathbb{E}}}_v$ denotes the empirical expectation on the validation sample. The BSS is essentially a binary version of the R-square.

For comparison, we included PASS, ${\text{SS}}^{\text{prior}}$, pLASSO², supervised LASSO and ALASSO on the three data examples to estimate the phenotyping model (${𝓜}_Y$). We exclude pLASSO¹ since it requires fitting of an unpenalized regression on supp ($\widehat{\mathit{\alpha }}$), which is infeasible when $|\text{supp}(\widehat{\mathit{\alpha }})|>n$. In addition, we compare to the unsupervised LASSO (ULASSO) approach of Chakrabortty et al. (2017), which estimates direction of the logistic coefficients for $Y\sim \sigma \left({\mathit{\beta }}^{\top }\mathit{X}\right)$ by regressing $I\left(S>c_u\right)$ against $\mathit{X}$ on the subset whose $S$ is either greater than $c_u$ or smaller than $c_l$, for some pre-specified $c_u$ and $c_l$ typically chosen such that $\text{Pr}\left(S>c_u\right)$ and $\text{Pr}\left(S<c_l\right)$ are small. Since the ULASSO approach only provides an estimate $\tilde{\mathit{\beta }}$ to optimize the prediction of ${\mathit{\beta }}^{\top }\mathit{X}$ for $Y\mid \mathit{X}$ without using $S$ explicitly as an additional predictor, we also derive a semi-supervised variant of ULASSO, denoted by ${\text{SS}}^{\text{ULASSO}}$, by regressing the labeled $Y$ against ${\tilde{\mathit{\beta }}}^{\top }\mathit{X}$ and $S$ as for ${\text{SS}}^{\text{prior}}$.

As shown in Figure 6, PASS significantly outperforms the supervised LASSO and ALASSO when $n=50$ in all three examples. As the label size $n$ increases, their performances get closer. Compared with the semi-supervised benchmarks, PASS has slightly or moderately better performance on the CAD and RA studies. For Depression, PASS substantially outperforms them, especially ${\text{SS}}^{\text{prior}}$ and ${\text{SS}}^{\text{ULASSO}}$. For example, when $n=50$, PASS attained average AUC in classifying depression about 0.1 higher than that of ${\text{SS}}^{\text{prior}}$ and ${\text{SS}}^{\text{ULASSO}}$ and 0.05 higher than pLASSO. The gap becomes smaller when $n$ increases as expected. Interestingly, the supervised estimators outperform pLASSO, ${\text{SS}}^{\text{prior}}$, and ${\text{SS}}^{\text{ULASSO}}$ on the Depression dataset as well but has similar or worse performance than these semi-supervised approaches on the other two examples. This could in part be attributed to the relatively poor quality of the surrogate information, which makes existing semi-supervised approaches fail. In contrast, PASS could utilize such prior information more effectively and robustly, and still preserves better performance than the supervised estimators. Thus, we can conclude that incorporating prior information from the unlabeled dataset could improve and stabilize the prediction performance of phenotyping models in EHR applications, and PASS is more robust and efficient in leveraging the prior information compared with existing semi-supervised methods. In addition, ULASSO shows much worse performance than the other supervised and semi-supervised methods in all examples. This illustrates the importance of collecting labels and including the surrogate in the regression models for EHR phenotyping.

Figure 6:

Out of sample AUC and BSS on the data examples 1–3, with various sizes of labelled training samples denoted as $n$. Median performance of PASS are marked using red dash line for ease of comparison.

6. Discussion

In this paper, we propose PASS, a high dimensional sparse estimator adaptively incorporating the prior knowledge from surrogate under a semi-supervised scenario commonly found in application fields like EHR analysis. Compared to the supervised approaches, the proposed PASS approach can substantially reduce the required number of labeled samples when the model assumptions (${𝓜}_S$) and (${𝓒}^{\text{prior}}$) and the elliptical design assumption (C1) hold exactly or approximately, and thus the prior information ${\mathit{\alpha }}^{\ast }$ is trustworthy. Compared to existing pLASSO and ${\text{SS}}^{\text{prior}}$ approaches that also incorporates prior information, the PASS approach is robust against unreliable prior information ${\mathit{\alpha }}^{\ast }$, which might be the case when the surrogate model assumptions are violated or the design $\mathit{X}$ is highly non-elliptically distributed.

One of the main challenges in our theoretical analysis comes from the colinearity of covariates ${\left(1,S_i,{\mathit{X}}_i^{\top }\widehat{\mathit{\alpha }},{\mathit{X}}_i^{\top }\right)}^{\top }$ due to the enrollment of $\rho$ to leverage the prior information in $\widehat{\mathit{\alpha }}$. We overcome this by properly constructing the oracle coefficients ${\mathit{\theta }}^{\ast }$ and the restricted eigenvalue assumption (A6). The formulation of our problem falls into the missing data framework with missing completely at random. However, the missing probability approaches 1 as $N\to \infty$. This together with the high dimensionality of $\mathit{X}$ makes the theoretical justifications more challenging than those used in the standard missing data literature. Without prior assumptions of ${\mathit{\beta }}_0-\rho {\mathit{\alpha }}_0$ being sparse in certain sense, the unlabeled dataset cannot directly contribute to the estimation of ${\mathit{\beta }}_0$. Our proposed PASS procedure hinges on the sparsity of ${\mathit{\beta }}_0-\rho {\mathit{\alpha }}_0$ to leverage the unlabeled dataset.

We have restricted the discussion to a single surrogate variable for simplicity. However, the proposed method can be easily extended to multiple surrogates. Specifically, consider $K$ surrogates, denoted by $S^{[1]},\dots ,S^{[\text{K}]}$. Let ${\widehat{\mathit{\alpha }}}^{[\text{k}]}$ be the ALASSO estimator regressing $S_i^{[k]}$ against ${\mathit{X}}_i$, $\widehat{𝓐}={\cup }_{k=1}^K\text{supp}\left({\widehat{\mathit{\alpha }}}_k\right)$, ${\mathit{S}}_i={\left(S_i^{[1]},\dots ,S_i^{[\text{K}]}\right)}^{\top }$ and $\mathit{\rho }={\left({\rho }^{[1]},\dots ,{\rho }^{[\text{K}]}\right)}^{\top }$. We can obtain an estimator for the model parameters as

$$\widehat{\zeta },\widehat{\gamma },\widehat{\mathit{\rho }},\widehat{\mathit{\beta }}=\underset{\zeta ,\gamma ,\mathit{\rho },\mathit{\beta }}{\arg \min }{n}^{-1}\sum _{i=1}^n\ell \left(Y_i,\zeta +S_i^{\top }\gamma +{\mathit{X}}_i^{\top }\mathit{\beta }\right)+{\lambda }_1{\Vert {\left(\mathit{\beta }-\sum _k{\rho }_k{\widehat{\mathit{\alpha }}}_k\right)}_{\widehat{𝓐}}\Vert }_1+{\lambda }_2{\Vert {\mathit{\beta }}_{{\widehat{𝓐}}^c}\Vert }_1.$$

Theoretical justification and finite sample performance of $\widehat{\mathit{\beta }}$ under this setting warrant further research. In our numerical studies, we only focus on fully simulated datasets and real examples. We are further interested in investigating the performance of our approach through semi-synthetic experiments with various setups for the surrogate variables. In addition, it may be interesting to extend the semi-supervised PASS estimator under a high dimensional sparse parametric regression to semi-parametric settings such as the sparse additive model (Ravikumar et al., 2009) and the sparse varying coefficient model (Noh and Park, 2010). Under semi-parametric models, one could still leverage prior information through shrinking the coefficients to “$\rho \widehat{\mathit{\alpha }}$“ with some sparse penalty function, to gain statistical efficiency. Studying the specific forms and theoretical properties of such approaches via a semi-supervised framework warrants future research.

$\mathbf{R}$ codes for implementing PASS and the benchmark methods, and replicating the simulation results can be found at https://github.com/moleibobliu/PASS.