SAT: a Surrogate-Assisted Two-wave case boosting sampling method, with application to EHR-based association studies

Abstract

Objectives

Electronic health records (EHRs) enable investigation of the association between phenotypes and risk factors. However, studies solely relying on potentially error-prone EHR-derived phenotypes (ie, surrogates) are subject to bias. Analyses of low prevalence phenotypes may also suffer from poor efficiency. Existing methods typically focus on one of these issues but seldom address both. This study aims to simultaneously address both issues by developing new sampling methods to select an optimal subsample to collect gold standard phenotypes for improving the accuracy of association estimation.

Materials and Methods

We develop a surrogate-assisted two-wave (SAT) sampling method, where a surrogate-guided sampling (SGS) procedure and a modified optimal subsampling procedure motivated from A-optimality criterion (OSMAC) are employed sequentially, to select a subsample for outcome validation through manual chart review subject to budget constraints. A model is then fitted based on the subsample with the true phenotypes. Simulation studies and an application to an EHR dataset of breast cancer survivors are conducted to demonstrate the effectiveness of SAT.

Results

We found that the subsample selected with the proposed method contains informative observations that effectively reduce the mean squared error of the resultant estimator of the association.

Conclusions

The proposed approach can handle the problem brought by the rarity of cases and misclassification of the surrogate in phenotype-absent EHR-based association studies. With a well-behaved surrogate, SAT successfully boosts the case prevalence in the subsample and improves the efficiency of estimation.

INTRODUCTION

Real-world evidence on the effectiveness of medical therapies can be derived from the study of health care information collected in the course of delivery and administration of routine medical care. Some typical sources of real-world data (RWD) include electronic health records (EHR), administrative claims, and billing data. Notably, these data are generated as a by-product of the health information ecosystem and are not directly intended to support research. In contrast, clinical trials and comparative effectiveness research studies relying on primary data collection focus on generation of high-quality research data, but, as a result, patient populations and care patterns represented in these studies may not be generalizable to real-world patients and their health care experiences. RWD provides a data resource that directly inherits the characteristics of patients and their care environments as they occur in routine clinical care, and the use of RWD for research has the potential to overcome limits to generalizability of research study data.¹

EHR, one important source of RWD, is a collection of patients’ individual health records that are routinely collected by health care providers and medical devices.² Diagnosis, clinical, and medication information in EHR can serve as important resources for biomedical and clinical research,^3–5 for example, to identify phenotype-genotype associations or improve clinical care quality. EHR-based association studies aimed at identifying risk factors for patient phenotypes have attracted substantial attention.^6,7 By providing data on a large pool of patients, EHR data are uniquely valuable for studying rare diseases. The 1983 Orphan Drug Act defined a rare disease as a condition that affects fewer than 200,000 persons in the United States, and there are over 10,000 rare diseases which together affect 10% of the population.⁸ For example, progressive supranuclear palsy is a rare degenerative neurological disease and is believed to affect at least 20,000 Americans.⁹ Some rare diseases are life-threatening, and patients who suffer from rare diseases often face diagnostic delays or misdiagnoses due to the lack of sufficient data to establish reliable diagnostic standards. Therefore, EHR-based studies investigating risk factors for rare diseases are particularly useful for advancing understanding of disease mechanisms and identifying biomarkers that may facilitate diagnosis.^10,11 Traditional statistical methods are the workhorse of such investigations; however, their performance is sensitive to the quality of the data.^12–18 In this article, we consider two challenges to the validity of conclusions in rare disease research relying on error-prone EHR-derived phenotypes (hereafter referred to as the surrogate phenotype).

EHR-based phenotyping algorithms have been developed to generate surrogates of the true patient phenotype based on EHR data and other ancillary data resources, including disease registries and claims data.^19,20 Surrogate phenotypes can be used to identify cohorts, exposures, and outcomes for studies such as genome- and phenome-wide association studies in which the surrogate phenotype is used in place of the true phenotype in analysis.^21–23 Before a phenotyping algorithm can be put into application, assessment of its accuracy using a validation dataset is necessary to guarantee its validity. However, even after validation, the performance of phenotyping algorithms is still affected by many factors, and imperfect classification accuracy induces bias and degrades the reliability of analysis results.^12–15 One analytic alternative is to use a subsample of patients whose true phenotypes are collected through manual chart review in conjunction with the surrogate information of the full cohort to conduct combined statistical analyses.^16–18 As the size of this subsample may be small due to time and budget constraints, developing methods to select subjects for chart review that lead to the best performance of subsequent analyses has the potential to significantly improve EHR-based association analyses.

The low prevalence of the outcome is another issue that requires special attention in analysis. Classifiers trained with a low prevalence and hence highly imbalanced outcome may have poor performance in the minority class, and the subsample with the natural class distribution inherited from the population commonly does not lead to the best classification performance.^24–26 Low prevalence outcomes also result in larger variances in coefficient estimation.²⁷ Methods for the low prevalence setting include randomly over-sampling (under-sampling) the minority (majority) class or generating synthetic cases.^28,29 Another solution is to form a balanced subsample with a well-designed sampling strategy.^30,31 Thus, a properly tailored sampling method could simultaneously address both the bias due to error in the surrogate phenotype and the efficiency loss caused by imbalanced classes.

There are a number of sampling methods that have been developed to address either the issue of low prevalence outcomes or surrogate phenotypes. For example, in the low prevalence setting, Tan and Heagerty³⁰ proposed a surrogate-guided sampling (SGS) design to enrich the cases in a subsample. Fithian and Hastie³¹ proposed a local case-control sampling strategy to select influential subsamples, that is, subjects whose responses seem to be contradictory to their predicted class. These samples are more difficult to classify conditional on the features, and thus are influential in forming the classification boundary. When the true phenotype is unavailable, inspired by an optimal subsampling procedure motivated by the A-optimality criterion (OSMAC),³² Zhang et al³³ proposed an optimal nearly response-independent sampling scheme (optimal sampling under measurement constraints, abbreviated as OSUMC). OSMAC and OSUMC both target minimizing the asymptotic mean squared error (MSE) of the subsample estimator, which is the trace of the asymptotic variance-covariance matrix of the subsample estimator, by selecting an optimal set of sampling probabilities. OSMAC and OSUMC differ in that OSMAC requires information on true phenotypes, whereas OSUMC does not. This is because the derived variance-covariance matrix of OSMAC involves the true phenotype while for OSUMC it does not. However, none of the above methods are designed to simultaneously handle both low prevalence and error-prone surrogate phenotypes, which motivates the development of the method considered here.

The proposed surrogate-assisted two-wave (SAT) sampling method leverages the strengths of SGS and OSMAC by incorporating the surrogate information into two subsampling waves to control the estimator’s MSE. See Figure  1 for a research pipeline and see Figure  2 for a conceptual diagram of SAT. Specifically, at the first stage, a pilot subsample is obtained by applying SGS to enrich cases, which facilitates a better pilot estimator to approximate the full cohort maximum likelihood estimator (MLE). At the second stage, by plugging the pilot estimator into OSMAC sampling probabilities and replacing the true phenotype by a function of the surrogate, we compute the SAT sampling probabilities and then obtain the second-stage subsample. We then get the final estimator by fitting a weighted logistic regression on the pooled subsample that combines both the pilot subsample and the second-stage subsample. To the best of our knowledge, this is the first sampling method that utilizes surrogate information to address the effects of both low prevalence and measurement error of an outcome in EHR-based association studies. When compared with OSUMC, SAT further exploits the information in the surrogate to aid sampling, while compared with SGS, SAT utilizes an additional MSE minimization rule in the second-stage sampling to improve estimation accuracy.

Figure 1.

The pipeline for association studies from electronic health record (EHR) data resource to final analysis. Based on the EHR data, surrogate phenotypes are generated by a validated well-behaved phenotyping algorithm, and gold standard phenotypes are obtained through manual chart review for a selected subsample of patients. The association analysis is conducted only on this subsample with their true phenotypes and risk factors. Our proposed method is a novel method that focuses on surrogate-assisted subsample selection to include informative samples to improve the subsequent association estimation.

Figure 2.

Surrogate-assisted two-wave (SAT) case boosting sampling. At stage 1, for enriching cases, we apply surrogate-guided sampling (SGS) to obtain a pilot subsample of size $n_1$, for which we collect the true phenotypes and use them to obtain a pilot estimator for the association parameter. At stage 2, based on the pilot estimator, the covariates, and the surrogate phenotype, we compute the SAT sampling probabilities which target controlling mean squared error (MSE) and apply them to obtain the second-stage subsample of size $n_2$, for which the true phenotypes are collected. The final estimator is obtained by fitting a weighted logistic regression in the pooled subsample of size $n_1+n_2$ with their true phenotypes.

MATERIALS AND METHODS

Methods

The method is applied to an EHR-derived dataset to conduct association analysis between a binary response $Y$ with $y_i\in \left\{0, 1\right\}$ and a set of covariates $x_i^{*}\in R^{p-1}$ using logistic regression. For all $N$ patients in the dataset, we collect $\left(x_i^{*}, s_i\right), i=1, \dots ,N$, where $s_i\in \left\{0, 1\right\}$ is the binary algorithm-derived surrogate ($S$). The true phenotype is only available for a small subsample that consists of two parts, the pilot subsample $\left(x_i^{*}, y_i, s_i\right)$ with $i=1, \dots ,n_1$ patients, and the second-stage subsample $\left(x_i^{*}, y_i, s_i\right)$ with $i=1, \dots ,n_2$ patients. Sampling techniques are exploited to facilitate subsample selection, and the related set of sampling probabilities is denoted as $\left\{{\pi }_i\right\}, i=1, \dots , N.$ We write $x_i={\left(1,x_i^{*T}\right)}^T\in R^p$ to include the intercept.

Existing method-OSMAC

OSMAC minimizes the asymptotic MSE of the subsample estimator by selecting an appropriate set of sampling probabilities. A pilot subsample of size $n_1$ is needed to obtain a pilot estimator ${\widehat{\beta }}_{\mathit{pilot}}$ to facilitate the subsequent sampling probability calculation. With imbalanced data, the pilot subsample is selected through a stratified sampling procedure where the binary response is used as a grouping variable and the stage 1 sampling probabilities are: ${\pi }_i=\frac{1}{2\times \mathit{number} of \mathit{cases} in \mathit{data}}$ for $y_i=1$ and ${\pi }_i=\frac{1}{2\times \mathit{number} of \mathit{controls} in \mathit{data}}$for $y_i=0$. The stage 2 sampling probabilities are then computed as

$${\pi }_i^{\mathit{OSMAC}}=\frac{\left||y_i-p_i\left({\widehat{\beta }}_{\mathit{pilot}}\right)\right|\left|\mathrm{ }\right||\mathrm{ }{M}_X^{-1}{x}_i|{|}_2}{{\sum }_{j=1}^N|y_j-p_j\left({\widehat{\beta }}_{\mathit{pilot}}\right)\left|\mathrm{ }\right|\left|M_X^{-1}\mathrm{ }{x}_j\right|{|}_2}$$ (1)

with $M_X={\sum }_{i=1}^N{p}_i\left({\widehat{\beta }}_{\mathit{pilot}}\right)(1-p_i\left({\widehat{\beta }}_{\mathit{pilot}}\right))x_i{x}_i^T$ and $p_i\left({\widehat{\beta }}_{\mathit{pilot}}\right)={\left(1+e^{{{-x}_i^T\widehat{\beta }}_{\mathit{pilot}}}\right)}^{-1}.$ Then the second-stage sampling is conducted with ${\pi }_i^{\mathit{OSMAC}}$ to obtain a subsample of size $n_2$ ($n_2>n_1$). The final estimator is obtained by fitting a weighted logistic regression on the pooled ${n=n}_1+n_2$ subsample as

$${\widehat{\beta }}^{\mathit{OSMAC}}=\arg \underset{\beta }{\min }\frac{1}{n_1+n_2}(\sum _{i\in \mathit{pilot}\hspace{0.17em}\mathit{set}}\frac{1}{{\pi }_i}\left[y_i\log \hspace{0.17em}{p}_i\left(\beta \right)+\left(1-y_i\right)\log \hspace{0.17em}\left(1-p_i\left(\beta \right)\right)\right]+\sum _{j\in \mathit{second}\hspace{0.17em}\mathit{stage}\hspace{0.17em}\mathit{set}}\frac{1}{{\pi }_j^{\mathit{OSMAC}}}\left[y_j\log \hspace{0.17em}{p}_j\left(\beta \right)+\left(1-y_j\right)\log \hspace{0.17em}\left(1-p_j\left(\beta \right)\right)\right]).$$ (2)

Because calculating the sampling probabilities requires knowledge of the true phenotype, $y_i$, this method is not practicable in real-world settings.

Existing method-OSUMC

The idea of OSUMC is very similar to OSMAC, except that its asymptotic results are derived unconditional on $Y$. Thus, the asymptotic variance-covariance matrix of the subsample estimator does not involve $Y$ and the resulting ${\pi }_i$ need no information on $y_i$, that is:

$${\pi }_i^{\mathit{OSUMC}}=\frac{\left||p_i\left({\widehat{\beta }}_{\mathit{pilot}}\right)(1-p_i\left({\widehat{\beta }}_{\mathit{pilot}}\right))\right|\left|\mathrm{ }\right||\mathrm{ }{M}_X^{-1}{x}_i|{|}_2}{{\sum }_{j=1}^N\left|p_i\left({\widehat{\beta }}_{\mathit{pilot}}\right)\right(1-p_i\left({\widehat{\beta }}_{\mathit{pilot}}\right))\left|\mathrm{ }\right|\left|M_X^{-1}\mathrm{ }{x}_j\right|{|}_2}$$ (3)

where $M_X$ and $p_i\left({\widehat{\beta }}_{\mathit{pilot}}\right)$ are similarly defined as in OSMAC. OSUMC does not consider outcome prevalence and uses simple random sampling (SRS) in pilot subsample selection. Only the second-stage subsample is used in the final estimation step to fit a weighted logistic regression. The OSUMC estimator is denoted as ${\widehat{\beta }}_{\mathit{OSUMC}}$.

Existing method-SGS

SGS is a surrogate-assisted case boosting algorithm. Specifically, the whole dataset is divided into two strata according to the surrogate value $s_i=1$ or $s_i=0$, and different sampling probabilities are then applied. Let $z_i=1$ indicate that the$i$-th subject is selected into the subsample and 0 otherwise, also let $n_{s1}$ be the number of positive surrogates in the whole dataset and $n_{s0}$ be the number of negative surrogates, then:

$${\pi }_i^{\mathit{SGS}}=P\left(z_i=1|s_i=s\right)=\mathrm{ }\frac{P(s_i=s|z_i=1\left)P\right(z_i=1)}{P(s_i=s)}$$ (4)

for $s\in \{0, 1\}$. The case proportion parameter $R=P\left(s_i=1|z_i=1\right)$ is decided by the investigator to adjust for the case proportion in the subsample, $P(z_i=1$) can be estimated by $n_1/N$, and $P(s_i=s)$ is approximated by $n_{s1}/N$ or $n_{s0}/N$. After standardizing to achieve ${\sum }_{i=1}^N{\pi }_i^{\mathit{SGS}}=1$, we get ${\pi }_i^{\mathit{SGS}}=\frac{R}{n_{s1}}$ for $s_i=1$ and ${\pi }_i^{\mathit{SGS}}=\frac{1-R}{n_{s0}}$ for $s_i=0$.

Proposed method-SAT

This method is an extension of OSMAC to the scenario where $Y$ is unavailable for the majority of the cohort and has a low prevalence among the study population. The two sampling steps in OSMAC need to be modified accordingly.

Stage 1: The SGS method is employed to boost cases.

Stage 2: As the true phenotype is not available for all subjects, we use OSMAC with surrogate phenotype:

$${\pi }_i^{\mathit{SAT}}=f\left({\widehat{\beta }}_{\mathit{pilot}},x_i,s_i\right)=\frac{\left||y_i^c-p_i\left({\widehat{\beta }}_{\mathit{pilot}}\right)\right|\left|\mathrm{ }\right||\mathrm{ }{M}_X^{-1}{x}_i|{|}_2}{{\sum }_{j=1}^N|y_j^c-p_j\left({\widehat{\beta }}_{\mathit{pilot}}\right)\left|\mathrm{ }\right|\left|M_X^{-1}\mathrm{ }{x}_j\right|{|}_2}$$ (5)

where $y_i^c=E\left(y_i=1|s_i=s\right)=P\left(y_i=1|s_i=s\right)$ is the expectation of the true phenotype conditional on the corresponding surrogate observation. To obtain $y_i^c,$ we write $P\left(y_i=1|s_i=s\right)=\frac{P(s_i=s|y_i=1\left)P\right(y_i=1)}{P(s_i=s)}$, where $P(s_i=1|y_i=1)$ can be computed based on the pilot subsample as the ratio of positive surrogates among all cases and $P\left(s_i=0|y_i=1\right)=1-P(s_i=1|y_i=1)$, $P(y_i=1)$ is approximated by $p_i\left({\widehat{\beta }}_{\mathit{pilot}}\right)$ and $P\left(s_i=s\right)\mathrm{ }\mathrm{is}\mathrm{ }\mathrm{approximated}\mathrm{ }\mathrm{by}\mathrm{ }{n}_{s1}/N\mathrm{ }\mathrm{or}\mathrm{ }{n}_{s0}/N.$ Then a weighted logistic regression is fitted on the pooled subsample to get the final estimator ${\widehat{\beta }}_{\mathit{SAT}}.$ An alternative is to replace $y_i^c$ by $s_i$ in the calculation of ${\pi }_i^{\mathit{SAT}}$, and we call the two methods as SAT-cY and SAT-S, respectively.

Proposed algorithm:

1. With a given case proportion parameter $R$, compute ${\pi }_i^{\mathit{SGS}}=\frac{R}{n_{s1}}$ for $s_i=1$ and ${\pi }_i^{\mathit{SGS}}=\frac{1-R}{n_{s0}}$ for $s_i=0$, and keep sampling with replacement to obtain a pilot subsample of size $n_1$. Collect true phenotypes for the selected samples.

2. Fit a weighted logistic regression on the pilot subsample to get the pilot estimator:

   ${\widehat{\beta }}_{\mathit{pilot}}=\arg \underset{\beta }{\min }\frac{1}{n_1}{\sum }_{i=1}^{n_1}\frac{1}{{\pi }_i^{\mathit{SGS}}}\left[y_i\log {\hspace{0.17em}p}_i\left(\beta \right)+\left(1-y_i\right)\log \hspace{0.17em}\left(1-p_i\left(\beta \right)\right)\right].$

3. If using SAT-cY, compute $P\left(s_i=1|y_i=1\right)\approx \frac{\mathit{number} of \mathit{positive} \mathit{surrogates} in \mathit{cases} }{\mathit{number} of \mathit{cases}}$ and $P\left(s_i=0|y_i=1\right)=1-P\left(s_i=1|y_i=1\right)$ based on the pilot subsample, and then calculate $y_i^c=\frac{P(s_i=s|y_i=1\left)P\right(y_i=1)}{P(s_i=s)}$. Otherwise, skip this step and go to step 4.

4. Obtain ${\pi }_i^{\mathit{SAT}}$ in equation (5) with $y_i^c$ or $s_i$ and conduct the second-stage sampling by sampling with replacement to get a subsample of size $n_2(n_2>n_1)$. Collect true phenotypes for the selected samples.

5. Pool all the subsamples and compute:

   $${\widehat{\beta }}_{\mathit{SAT}}=\arg \underset{\beta }{\min }\frac{1}{n_1+n_2}(\sum _{i\in \mathit{pilot}\hspace{0.17em}\mathit{set}}\frac{1}{{\pi }_i^{\mathit{SGS}}}\left[y_i\log \hspace{0.17em}{p}_i\left(\beta \right)+\left(1-y_i\right)\log \hspace{0.17em}\left(1-p_i\left(\beta \right)\right)\right]+\sum _{j\in \mathit{second}\hspace{0.17em}\mathit{stage}\hspace{0.17em}\mathit{set}}\frac{1}{{\pi }_j^{\mathit{SAT}}}\left[y_j\log \hspace{1em}{p}_j\left(\beta \right)+\left(1-y_j\right)\log \hspace{1em}\left(1-p_j\left(\beta \right)\right)\right]).$$

Simulation study

We compare SAT to the uniform method (obtain $n$ samples by SRS), SGS (obtain $n$ samples by SGS), and OSUMC under four settings. OSMAC is provided as a reference representing the best performance that could be achieved if true phenotypes were available for the full sample. The considered settings are:

1. In this setting, we use a surrogate exhibiting nondifferential sensitivity and specificity. The covariates $x_i^{*}\in R^{p-1}$ with $p=8$ are generated from a multivariate normal distribution $N_{p-1}(-1.51_{p-1}, \mathrm{\Sigma })$ with $\mathrm{\Sigma }=\left({0.5}^{1\left(\left|i\ne j\right|\right)}\right)$ where $1\left(\left|i\ne j\right|\right)=1$ if $i\ne j$ and 0 otherwise, and $1_{p-1}$ is a length $p-1$ vector of 1’s. The true coefficient vector is ${\beta }^{*}={\left(-0.5, 0, 0, 0.5, 0.5, 0.5, \mathrm{0.5,} 0.5\right)}^T$ to get a low case prevalence $p\left(Y=1\right)=5.4\%$. The response $Y$ is generated independently from the Bernoulli distribution, and the surrogate $S$ is generated from a Bernoulli distribution with sensitivity 85% and specificity 95%. We fix $n_1=300$ and increase $n_2$ from 500 to 1,200 by 100 to evaluate the effect of sample size on performance. The case proportion parameter $R$ is set to be 0.5. Moreover, to see the effects of sensitivity and specificity on the performance of SAT, we also fix $\left(n_1, n_2\right)=(300, 800)$ and let sensitivity and specificity take values from $(0.80, 0.85, 0.90, 0.95, 0.99)$ to generate $S$.

2. The surrogates’ sensitivity and specificity depend on the exposure group. The first covariate, ie, the exposure, is generated from a Bernoulli distribution with probability 0.4, and the remaining covariates are generated from $N_{p-2}(-1.51_{p-2}, \mathrm{\Sigma })$ with $\mathrm{\Sigma }=\left({0.5}^{1(\mathrm{i}\ne \mathrm{j})}\right)$. The coefficient vector is ${\beta }^{*}={\left(-0.7, 0.5, 0, 0.5, 0.5, 0.5, \mathrm{0.5,} 0.5\right)}^T$ to make $p\left(Y=1\right)=5.4\%$. When generate the surrogate, we consider two scenarios: A. for the unexposed group, sensitivity is 85% and specificity is 90%; for the exposed group, sensitivity is 80% and specificity is 95%. B. for the unexposed group, sensitivity is 85% and specificity is 90%; for the exposed group, sensitivity is 95% and specificity is 80%. We fix $n_1=300$ and increase $n_2$ from 500 to 1,200 by 100. The $R$ is set to be 0.5.

3. We investigate the effect of pilot ratio $\rho =\frac{n_1}{n}\in \{0.1, 0.15,\mathrm{\dots }, 0.85, 0.9\}$ on the performance of SAT with pooled subsample size $n=800$, and a surrogate with nondifferential sensitivity (85%) and specificity (95%) is used. All the other details are the same as in setting 1.

4. We evaluate the effect of the case proportion parameter $R\in \{0.1, 0.2,\dots ,0.8, 0.9\}$ on the performance of SAT with case prevalence values 1.6% and 5.4%, respectively, to find a range of $R$ that ensures SAT’s usability and robustness. We fix $n_1=300$ and $n_2=800$ and use a surrogate with nondifferential sensitivity (85%) and specificity (95%). To obtain a case prevalence of 1.6%, we let ${\beta }^{*}={\left(-2, 0, 0, 0.5, 0.5, 0.5, \mathrm{0.5,} 0.5\right)}^T$. All the other details are the same as in setting 1.

See Table  1 for a summary of the simulation setting. Under each setting, the whole population is generated once with full sample size $N$ = 100,000, and the sampling procedure is repeated 1,000 times. The methods are compared in terms of their estimation accuracy (MSE), which is computed as $\frac{1}{\mathrm{1,000}}{\sum }_{t=1}^{\mathrm{1,000}}\parallel \widehat{\beta }-{\beta }^{*}{\parallel }_2^2$, where $\widehat{\beta }$ is the estimated coefficient vector, ${\beta }^{*}$ is the true coefficient vector, and ${\parallel \cdot \parallel }_2^2$ is the squared Euclidean distance.

Table 1.

Details of simulation setting

	$n_1$	$n_2$	Misclassification	(Sensitivity, specificity)
Setting 1	300	500–1,200	Nondifferential	(0.85, 0.90)	Baseline setting
		800		(0.80, 0.85, 0.90, 0.95, 0.99)	Change surrogate’s property
Setting 2	300	500–1,200	Differential	A: Exposed (0.80, 0.95) Unexposed (0.85, 0.90)	Differential misclassification with respect to exposure groups
				B: Exposed (0.95, 0.80) Unexposed (0.85, 0.90)
Setting 3	$n_1=800\times \rho ,$ $n_2=800-n_1$ $\rho \in [0.1, 0.9]$		Nondifferential	(0.85, 0.90)	Change pilot size
Setting 4	300	800	Nondifferential	(0.85, 0.90)	Change case proportion

Note: For all the settings, the total sample size is $N=100,000$ and the dimension is $p=8$. For settings 1, 3, and 4, all the covariates are generated from a multivariate normal distribution, and for setting 2, the exposure variable is generated from a Bernoulli distribution with probability 0.4 and the remaining covariates are generated from a multivariate normal distribution

Application to a breast cancer study

We applied these alternative methods to an EHR dataset from the BRAVA study, which focused on identifying risk factors for second breast cancer events (SBCE, defined as recurrent or second breast primary cancer) in a cohort of women with a personal history of breast cancer at Kaiser Permanente Washington,³⁴ an integrated health care delivery system in Washington state. The dataset consists of 3,152 women diagnosed with a primary stage I–IIB invasive breast cancer between 1993 and 2006. The outcome of interest, SBCE, was obtained through manual review of medical records for all cohort members, and was treated as a gold standard. Additionally, an algorithm-derived surrogate phenotype was obtained based on automated EHR and cancer registry data.³⁵ We treat SBCE as the outcome of interest and use characteristics of the primary breast cancer as risk factors, including patient age at primary breast cancer diagnosis, tumor stage (stage I or II), and estrogen receptor status. The study population demonstrates a moderately low prevalence of SBCE. Among the 3,152 patients, 407 (12.7%) experienced SBCE. To investigate the SAT’s performance on EHR datasets with significantly lower phenotype prevalence, we also experimented with synthesized datasets with phenotype prevalence of 2.5% and 5% by oversampling the controls in BRAVA. The availability of the gold standard SBCE phenotype for the full cohort facilitates the estimation of an oracle full cohort MLE, based on which we calculate the empirical MSE and estimation bias across 500 replicates of sampling and estimation as the metrics to compare methods.

Similar to the settings considered in Tong et al,¹⁸ we compare the methods with surrogates generated in three different ways. The first scenario (case 1) uses a previously published SBCE phenotyping-algorithm of high specificity to generate the surrogate, which has 89% sensitivity and 99% specificity.³⁵ The second scenario (case 2) uses a randomly generated surrogate with nondifferential 90% sensitivity and 95% specificity. The third scenario (case 3) uses a surrogate exhibiting differential misclassification. Specifically, for patients with stage I primary cancers, we generated a surrogate with 90% sensitivity and 95% specificity, whereas for stage II patients, the surrogate was simulated with 95% sensitivity and 90% specificity. Under each scenario, different combinations of pilot subsample size $n_1$ (200 and 400) and second-stage subsample size $n_2$ are tested (400, 700, and 1,000). We use $R=0.5$ in the analysis.

RESULTS

Results from simulation studies

Among all the methods under comparison, the uniform method had the worst performance. As OSUMC uses SRS in the pilot sampling stage, the low prevalence of the outcome led to failures in ${\widehat{\beta }}_{\mathit{pilot}}$ estimation, especially when $n_1$ was small. For methods with pilot subsamples obtained by SGS, there were very few cases of failed estimation. The results reported for each method are limited to the cases where the pilot estimation did not fail. Under setting 1 (Figures  3 and 4), SAT-S had comparable performance to SAT-cY, and both achieved a smaller MSE than OSUMC and SGS. Figure  4 displays head-to-head comparisons of MSEs among methods over a grid of sensitivities and specificities. To summarize, OSUMC performed worse than the SAT methods, and a surrogate with a high specificity enabled the SAT methods to beat SGS. Between SAT-cY and SAT-S, once the sensitivity was high enough, SAT-S outperformed SAT-cY even when the specificity was moderate. The results of setting 2 (see Supplementary Figure S3) indicate that the SAT methods outperformed OSUMC and had performance at least as good as SGS with a surrogate exhibiting differential misclassification. The results from both settings 1 and 2 also demonstrate that, with an increasing $n_2$ (thus $n$), the estimation accuracy of all the methods improved. Figure  5 shows how the MSE changes when we fix $n$ and increase the pilot subsample’s proportion. For SAT-cY and SAT-S, when $0<\rho <0.25,$MSE decreased sharply with increasing $\rho$ for both methods because only when the pilot subsample is large enough can the pilot estimator approximate the full cohort MLE reasonably well. After $\rho =0.4$, the curves of SAT-S and SAT-cY show an increasing trend. When $\rho \to 1$, the two curves converge to the same level, the MSE of applying SGS in both sampling stages. This is because, the second-stage subsample is key to controlling MSE, and a larger $\rho$ implies a smaller proportion of the second-stage subsample thus a less effective reduction of MSE. Therefore, there exists an optimal $\rho$ that achieves a balance between the quality of the pilot estimator and the amount of MSE reduction. As for the effect of $R$ on SAT, the results of setting 4 (see Supplementary Figure S4) show that the region of $R$ that corresponds to lower MSEs is [0.3, 0.5] for case prevalence around 4%. To avoid failures in the estimation of ${\widehat{\beta }}_{\mathit{pilot}}$ when the case prevalence is low and $n_1$ is small, a slightly larger $R\in [0.5, 0.6]$ can be used without compromising the performance of SAT. Using $R=0.5$ is a safe choice for most situations.

Figure 3.

Comparison of methods under setting 1 with $n_1=300$ and an increasing second-stage subsample size $n_2\in (500, 600, 700, 800, 900, 1,000, 1,100, 1,200)$. The surrogate exhibits nondifferential misclassification with sensitivity of 85% and specificity of 95%. OSMAC is displayed here as a gold standard for reference. Under setting 1, the SAT methods achieved smaller MSE than the SGS, OSUMC, and the uniform method.

Figure 4.

Comparison of methods under setting 1 with $(n_1, n_2)=(300, 800)$ for different combinations of sensitivities and specificities. The MSE differences between methods over a grid of sensitivities and specificities are displayed. The heatmap (a) represents the difference between the MSE of SGS and the MSE of SAT-cY (ie, MSE[SGS] – MSE[SAT-cY]) for each combination of sensitivity and specificity. The red block denotes that, at the specified sensitivity and specificity, SGS has a larger MSE than SAT-cY, and the purple block denotes that SAT-cY has a larger MSE than SGS. All the subsequent heatmaps can be similarly interpreted. From these plots we observe that, in terms of MSE minimization, OSUMC performed worse than the SAT methods, and a surrogate with a high specificity enabled the SAT methods to outperform SGS.

Figure 5.

Comparison of methods under setting 3 with $n=800\mathrm{ }\mathrm{and}\mathrm{ }\mathrm{an}\mathrm{ }\mathrm{increasing}\mathrm{ }\mathrm{pilot}\mathrm{ }\mathrm{ratio}\mathrm{ }\rho =\frac{n_1}{n}$. The surrogate has a nondifferential misclassification rate, and sensitivity is 85% and specificity is 95%. The pooled subsample size $n=800$ and the ratio $\rho =\frac{n_1}{n}\in \{0.1, 0.15,\mathrm{\dots }, 0.85, 0.9\}$. OSMAC is displayed here as a gold standard for reference. The pattern of variation in MSE for all the two-stage methods (OSMAC, SAT-S, and SAT-cY) as a function of $\rho$ indicates that the pilot subsample size $n_1$ should be relatively small to guarantee a good MSE reduction performance while it cannot be too small as the first stage pilot sample is needed to generate a well-behaved pilot estimator.

Results from BRAVA data analysis

In the analysis with the original BRAVA dataset, as in the simulation, when $n_1=200$there were two failed cases in ${\widehat{\beta }}_{\mathit{pilot}}$ estimation for OSUMC due to the low prevalence of the outcome, and the methods with pilot subsamples obtained by SGS successfully resolved this problem. Figure  6 reports the MSE of SAT-S, SAT-cY, OSUMC, and the SGS method. The results display a similar pattern as in the simulation, that is, for all three surrogate types, OSUMC and SGS perform worse than the SAT methods; SAT-S has comparable performance to SAT-cY. Supplementary Figure S5 shows the estimation bias of the effect (log odds ratio) of tumor stage on the occurrence of SBCE under case 1. All methods appear to be unbiased. The OSUMC has a larger variance than other methods and the other three methods have comparable performance. The results for cases 2 and 3 are similar (see Supplementary Figures S6 and S7). In settings where we varied prevalence by oversampling controls, SAT achieved smaller MSEs than SGS and OSUMC (Supplementary Material, Part I).

Figure 6.

Comparison of methods in terms of MSE using BRAVA breast cancer data. From the first row to the third row, the plots show the results of surrogate cases 1–3, respectively. The first column is for the setting with pilot subsample size $n_1=200,$ and the second column is for the setting with pilot subsample size $n_1=400$. Each panel displays the MSE of all the methods at 3 levels of the second-stage subsample size, that is, $n_2=400, 700$, and $1,000$. The MSE is calculated as $\frac{1}{500}{\sum }_{t=1}^{500}{\parallel \widehat{\beta }-{\beta }^{*}\parallel }_2^2$, where $\widehat{\beta }$ is the estimated coefficient vector, ${\beta }^{*}$ is the oracle full cohort MLE based on all the true phenotypes and ${\parallel \cdot \parallel }_2^2$ is the squared Euclidean distance. OSMAC is displayed here as a gold standard for reference. These plots display a significant superiority of the SAT methods over SGS and OSUMC in terms of MSE reduction.

DISCUSSION

EHR-based association studies of rare diseases using surrogate phenotypes may produce unreliable conclusions. Both the low prevalence of the outcome and the error in the surrogate impede accurate estimation of risk factors' association with the outcome. Numerous techniques have been developed to handle these two challenges, but to our knowledge, there is little literature that considers the two problems simultaneously, although this is a commonly encountered problem in real-world studies of EHR data. Yin et al³⁶ recently proposed a method that handles these two problems with a surrogate-assisted sampling scheme to enrich cases and a projection technique that exploits the correlation between outcome and surrogate to improve estimation efficiency based on a nondifferential misclassification assumption. Different from their idea and with no assumption, we formulated this problem as a subsample selection task and combined SGS and OSMAC to provide subsample selection guidance, the SAT methods. Through simulation studies, we demonstrated that with a proper surrogate whose sensitivity and specificity are reasonably high, SAT could effectively reduce MSE and is better than the methods that only resolve one of the two issues. For example, OSUMC provides a nearly response-independent subsampling procedure. However, failure to consider the phenotype prevalence results in inaccurate pilot estimation, and the method often fails to converge in pilot estimation. The BRAVA data analysis demonstrated the success of applying the SAT methods on real-world datasets. Although the rate of experiencing SBCE in BRAVA was 12.7%, which does not meet a strict definition of rare disease, we also investigated the SAT’s performance in rarer disease settings by oversampling controls. In these rare disease settings, SAT achieved smaller MSEs than SGS and OSUMC. Future research should focus on methods for very low case prevalence, for example, 1% or lower, in which settings even the OSMAC estimator has a large bias. Thus, to conduct association studies for extremely rare diseases, the sampling approach is not the best choice and other methods should be considered.

SAT is proposed for EHR-based association studies based on the assumption that the information contained in EHR is adequate to guarantee the validity of the manual chart review results, that is, a “true phenotype” can be obtained with manual chart review. However, there are situations where the reference is not a gold standard as the completeness of the information required to guarantee the validity of a diagnosis may vary depending on health care utilization, clinical practice, and documentation practices (eg, an NIH clinical center compared with a community hospital). This issue has been discussed extensively in diagnostic test studies^37,38 but is under-researched in EHR data settings.^39,40 Linking EHR data to external databases with additional specialized information (eg, cancer registry data) may improve accuracy in some applications. In general, however, additional research is needed to develop new methods that account for the imperfect manual chart review.

We have several suggestions to apply SAT in practice. First, when only a surrogate of poor-quality (ie, sensitivity or specificity is <0.8) is available, SGS or OSUMC are better choices than SAT. Second, although SAT-S achieves a greater MSE reduction than SAT-cY with a good surrogate in our simulation, SAT-cY is more robust than SAT-S to the accuracy of the surrogate. Unless a phenotyping algorithm has been validated to be of high accuracy, SAT-cY is preferred. Third, the number of cases in the pilot subsample depends on a user-specified case proportion parameter and the surrogate's accuracy, especially its specificity.³⁰ If the case prevalence is very low, we can appropriately increase the case proportion parameter (eg, taking $R$ from [0.5, 0.6]) or use a surrogate of higher specificity to include more cases in the pilot subsample. Finally, the pilot subsample needs to be large enough to provide a good approximation of the full cohort MLE for the subsequent sampling probability calculation.

In essence, SAT is a generalization of OSMAC under the phenotype-absent scenario in rare disease analysis, and it has some limitations. OSMAC is developed to minimize the asymptotic MSE of the subsample estimator whose asymptotic property derivation procedure does not consider the imbalanced class distribution. To our knowledge, there are no explicit results for the asymptotic distribution of the subsample estimator in the imbalanced class situation. Even though OSMAC and SAT successfully control the MSE in our simulations, theoretical development is needed to generalize the theoretical foundation of OSMAC to be compatible with the low prevalence scenario. The second limitation of SAT comes from the requirement of a phenotyping-algorithm of moderate accuracy to guarantee the success of SAT. In some investigations of rare diseases, a case-finding algorithm is commonly employed to capture as many cases as possible. These algorithms guarantee a high sensitivity of the surrogate, but have poor specificity which is a barrier to the application of SAT. The third limitation is in the selection of an optimal $\rho$ with a given $n$. In general, a smaller $\rho$ is preferred to maximize the extent of MSE reduction. However, $n_1$ needs also to be large enough to obtain a good pilot estimator. An optimal $\rho$ is also related to the problem itself, for example, data quality and signal strength, thus it is hard to provide uniform guidance.

An alternative to the proposed sampling method is to directly apply a nearly ‘perfect’ surrogate produced by a highly polished phenotyping algorithm in association studies. Great efforts and advances have been made in developing high-throughput phenotyping algorithms,^41–45 where techniques such as natural language processing tools are exploited to leverage information from multiple resources to improve classification accuracy. However, as discussed before, even after validation, the performance of phenotyping algorithms could still be affected by many factors in specific applications, and classification error is practically unavoidable. Our method approaches the problem in a different way and reduces the consequences of using error-prone surrogates in association studies. Thus, even with a good surrogate, applying SAT methods is still a good choice that leverages both the experience of clinicians and the accuracy of the surrogate to provide an unbiased and efficient estimator.

Going forward, it is necessary to investigate the asymptotic properties of the subsample estimator in the imbalanced class scenario. Once the asymptotic distribution is available, we can follow the approach of OSMAC to derive an optimal set of sampling probabilities from minimizing the asymptotic MSE. Moreover, it is also worthwhile to develop sampling techniques that target improving prediction performance.

CONCLUSION

To conclude, in this article, we introduce a surrogate-assisted two-wave case boosting sampling method to address the issues in estimation caused by low prevalence outcomes and error-prone surrogate phenotypes in association studies. With a well-behaved surrogate, SAT provides a more accurate estimator compared with some benchmark methods. Further theoretical development to rigorously account for the imbalanced class distribution is needed to sharpen the method’s performance in MSE minimization.

FUNDING

Research reported in this publication was supported in part by the National Institutes of Health (NIH) grants 1R01AI130460, 1R01AG073435, 1R56AG074604, 1R01LM013519, 1R56AG069880 (to XL and YC), R21CA227613 (to RH, YC, and JC), and R21CA143242 (JC). Data collection was funded in part by the NIH grants R01CA093772, R01CA120562, and U01CA063731. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. This work was supported partially through Patient-Centered Outcomes Research Institute (PCORI) Project Program Awards (ME-2019C3-18315 and ME-2018C3-14899). All statements in this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors or Methodology Committee.

AUTHORS CONTRIBUTION

XL and YC designed methods and experiments; XL conducted simulation experiments and data analysis; RH and JC provided the dataset from Kaiser Permanente Washington; XL drafted the main article; all coauthors made substantial contributions to revise the article; all authors interpreted the results and provided instructive comments; all authors have approved the article.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.