Enabling phenotypic big data with PheNorm

Abstract

Objective

Electronic health record (EHR)-based phenotyping infers whether a patient has a disease based on the information in his or her EHR. A human-annotated training set with gold-standard disease status labels is usually required to build an algorithm for phenotyping based on a set of predictive features. The time intensiveness of annotation and feature curation severely limits the ability to achieve high-throughput phenotyping. While previous studies have successfully automated feature curation, annotation remains a major bottleneck. In this paper, we present PheNorm, a phenotyping algorithm that does not require expert-labeled samples for training.

Methods

The most predictive features, such as the number of International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes or mentions of the target phenotype, are normalized to resemble a normal mixture distribution with high area under the receiver operating curve (AUC) for prediction. The transformed features are then denoised and combined into a score for accurate disease classification.

Results

We validated the accuracy of PheNorm with 4 phenotypes: coronary artery disease, rheumatoid arthritis, Crohn’s disease, and ulcerative colitis. The AUCs of the PheNorm score reached 0.90, 0.94, 0.95, and 0.94 for the 4 phenotypes, respectively, which were comparable to the accuracy of supervised algorithms trained with sample sizes of 100–300, with no statistically significant difference.

Conclusion

The accuracy of the PheNorm algorithms is on par with algorithms trained with annotated samples. PheNorm fully automates the generation of accurate phenotyping algorithms and demonstrates the capacity for EHR-driven annotations to scale to the next level – phenotypic big data.

INTRODUCTION

New health problems, medications, and regimens emerge on a daily basis. To understand their clinical impact in a timely manner, large amounts of accurate genotypic and phenotypic data must be readily available for research in a cost-effective manner. The advent of high-throughput gene sequencing technologies has reduced the cost of obtaining genomic data exponentially, from 2.7 billion USD for the first human genome (the Human Genome Project)¹ down to 1000 USD per genome in 2016. Currently, million-people-scale sequencing projects are under way to generate genomic data for research.² However, amassing phenotypic data remains a challenge,³ as it traditionally takes human effort to record the phenotypes of patients.

To overcome the scarcity of phenotypic data, genomic and other medical studies have begun to extract phenotypic information from electronic health records (EHRs) to augment existing biorepositories or quickly create new ones.^4,5 Notable efforts include the i2b2 effort led by Harvard University and Partners HealthCare,^6–16 the BioVU effort led by Vanderbilt University,¹⁷ and the multicenter eMERGE Network.^18–20 Typically, EHR-based phenotyping extracts features from the patient’s EHR and assembles them into a classification rule (called a “phenotyping algorithm”) to infer whether the patient has a target phenotype. The features often involve the patient’s demographic information, such as age and sex; codified information, such as diagnosis, medication, lab, and procedure codes (codified features); and information extracted from the narrative notes via natural language processing (NLP features). It has been demonstrated that studies utilizing such data-driven phenotypes can reproduce previously established results based on phenotypic data obtained by traditional means^21,22 and can drive novel studies, such as genome-wide and phenome-wide association studies.^19,23 However, the current approach to algorithm development relies on tremendous domain expert participation and takes many months to complete.

In most settings, a “supervised” machine learning method is employed to estimate a statistical model that outputs the probability or classification of the target phenotype based on a number of input features and a training dataset consisting of a few hundred patients with “gold-standard” phenotype labels. The features are curated and engineered by a panel of clinicians, informaticians, statisticians, and computer scientists, while the labels are obtained by experts via laborious manual chart review. Feature curation and sample annotation, requiring up to months of human effort, are thus the rate-limiting factors in phenotyping, due to the heavy human input required. Research is now under way to automate algorithm development to fully leverage the efficiency of EHR-based research and achieve so-called high-throughput phenotyping.^24,25

Toward the goal of automating feature curation, some studies have attempted to use all available codified and NLP features potentially predictive of the phenotype of interest for algorithm estimation.^26–29 Though this approach eliminates the feature selection step entirely, it tends to be suboptimal when the number of codified and NLP features is substantially larger than the size of the labeled training set. Supervised algorithms trained with a large number of noisy features and a small number of labeled examples can suffer from significant overfitting and instability, leading to suboptimal out-of-sample performance.³⁰ While overfitting can be reduced with estimation procedures that penalize model complexity (penalized regression procedures are a common choice^31–33) a price is paid in terms of sampling variability that reduces the out-of-sample accuracy; the more unnecessary complexity, the heavier the price. It is therefore essential to use only informative features in the model. Wright et al. mined codified EHR data to look for possible associations between problems, medications, and lab tests, which can potentially be used for automated feature selection. Unfortunately, their remaining modeling steps involved a large amount of manual screening and design.^34,35 More recently, fully automated feature selection has been achieved with satisfying results. The automated feature extraction for phenotyping (AFEP) method³⁶ calls on online sources for the knowledge originally provided by domain experts by scanning articles concerning target phenotypes on Wikipedia and Medscape to extract relevant medical concepts as candidate NLP features. Informative features are then identified by analyzing the co-occurrence patterns of the features in the EHR database. The performance of the automatically selected features is comparable to those designed by experts. The surrogate-assisted feature extraction (SAFE) method³⁷ improved upon AFEP, and was able to cut the feature set down to 10–30 highly informative features that outperform the AFEP features significantly for classifying disease phenotypes when the number of training samples is small.

Despite the success in automated feature curation, sample annotation remains a major obstacle to achieving high-throughput phenotyping. In addition to feature selection, feature refinement or selective prioritization of representative samples for annotation can work to reduce the sample size needed for the training.^37–39 However, manual annotation must be entirely removed from the algorithm development process to truly achieve scalable phenotyping. It is therefore desirable to move toward learning associations between the features and the target phenotype without using any gold-standard labels to guide the learning. A feasible approach is to rely on “silver-standard labels” automatically generated from the EHR in place of human-annotated labels for training, such as counts of relevant billing codes or NLP mentions, which are strong but imperfect predictors of true disease status. This approach is known as “distant supervision” in the machine learning community and was employed by Yu et al.³⁷ with SAFE for feature selection but not for model estimation. Recently, Agarwal et al.²⁹ proposed XPRESS to use the silver-standard labels directly for training the phenotyping algorithm in order to completely eliminate the human annotation step. However, the simplistic choice of their silver-standard labels – an indicator of whether the phenotype is positively mentioned in the narrative notes – leads to suboptimal performance of their algorithm when compared to supervised counterparts.

This paper presents PheNorm, a completely annotation-free 2-step classification method for phenotyping involving an initial normalization step of highly predictive features of the target phenotype followed by a denoising step to leverage additional information contained in the remaining candidate features. We validate the performance of our method with 4 phenotypes from the Partners HealthCare EHR and compare the area under the receiver operating curve (AUC) of the PheNorm score to that of the supervised algorithms trained with gold-standard labels.

METHODS

The 2 steps of the PheNorm procedure are outlined in Figure 1. The first step transforms a highly predictive feature of the target phenotype, such as the number of International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes of the target phenotype in the patient records, to resemble a 2-component normal mixture distribution with high accuracy for prediction. The second step involves self-regression with dropout to denoise the transformed feature based on additional candidate features, similarly transformed, to further improve the prediction. The output is a linear combination of all the transformed features.

Figure 1.

Workflow of PheNorm. Top left: density plot (after logarithm transformation) of a highly predictive feature (illustrated here using the ICD-9-CM count of ulcerative colitis from a Partners HealthCare EHR datamart), denoted by x, in patients who do (the right curve) and do not (the left curve) have the phenotype. Top right: Density plot of the ICD-9-CM count after the normal mixture transformation using the total number of notes in the patient’s EHR, denoted by x_note. The densities of the phenotype positive and negative patients are approximately normally distributed, and the 2 populations are separated to a large degree. Bottom right: The transformed feature is denoised by self-regression of the transformed feature, denoted by y, onto the entire transformed and randomly corrupted feature set, denoted by Z˜, with dropout. The transformed features are then combined into a prediction formula for disease status classification based on the estimated regression coefficient. Bottom left: The receiver operating characteristic (ROC) curve of the feature or score in each step, with AUC growing steadily (gray curves are copies of the ROC curves from the previous steps).

Raw feature preparation

The input for the PheNorm algorithm consists of unlabeled data on a set of potentially informative features, either automatically curated or designed by experts. For the purpose of illustration in a high-throughput phenotyping scenario, we use SAFE³⁷ to automatically curate features (listed in the Supplementary Material). Briefly, online articles about the target phenotype from publicly available knowledge sources, such as Wikipedia and Medscape, are scanned with NLP software to extract medical concepts recorded in the Unified Medical Language System.⁴⁰ These concepts are potentially related to the target phenotype. Then, narrative notes in the EHR database are processed with NLP software, which identifies mentions of the above medical concepts. We include only positive mentions, ie, mentions confirming the presence of a condition, the performance of a procedure, the prescription of a medication, etc., in all analyses. The patient-level counts of these concept mentions are assembled as candidate feature data. The SAFE procedure selects a subset of the candidate features via frequency control and repeated fitting of sparse logistic regressions to predict silver-standard labels created from combinations of ICD-9-CM diagnosis codes and NLP counts of the target phenotype. Features predictive of the silver-standard labels are deemed as informative features for further algorithm training. The NLP analyses used for processing the notes are provided in many out-of-the-box software tools,^41–45 and some hospitals and research institutions have their own NLP implementations.

Throughout, we denote by $x_{\text{ICD}}$ and $x_{\text{NLP}}$ the counts of ICD-9-CM codes and free-text positive mentions of the target phenotype, respectively. In addition, we let $x_i$, $i=1,\dots ,p$ be the counts of positive mentions of all remaining concepts selected by SAFE and $x_{\text{note}}$ be the count of narrative notes for the patient. The input data for the PheNorm algorithm is then the patient-level data on ($x_{\text{ICD}},x_{\text{NLP}},x_{\text{note}},x_1,\dots ,x_p)$. We also include a simple derived feature, $x_{\text{ICDNLP}}=x_{\text{ICD}}+x_{\text{NLP}}$.

Normal mixture normalization

Since $x_{\text{ICD}}\text{,}$$x_{\text{NLP}},$ and $x_{\text{ICDNLP}}$ are expected to be fairly predictive of the underlying phenotype, we expect them to follow mixture distributions: the counts of phenotype-positive patients cluster around the upper end and the counts of phenotype-negative patients cluster around the lower end. However, $x_{\text{ICD}}$, $x_{\text{NLP}}$, and $x_{\text{ICDNLP}}$ tend to be higher for patients with more health care utilization regardless of their true underlying phenotype status. Therefore, we propose to normalize each of these 3 main features by $x_{\text{note}}$ based on our observation that, for $x=x_{\text{ICD}}$, $x_{\text{NLP}}$, or $x_{\text{ICDNLP}}$, with appropriate choice of $\alpha$, the distribution of the normalized count

$$z=\log \left(1+x\right)-\alpha \log (1+x_{\text{note}}),$$

approximately follows a normal mixture distribution (see top right of Figure 1) with$z\mathrm{\mid }Y\sim N({\mu }_Y,{\sigma }^2)$, where $Y\in \{1,0\}$ is the true phenotype status. The optimal $\alpha$ is chosen as the minimizer of the difference between the empirical distribution of $z$ and its normal mixture approximation,

$$\underset{0<\alpha <1}{\text{argmin}}\mathbb{D}\left(\alpha \right),\text{with}\mathbb{D}\left(\alpha \right)=\underset{-\infty }{\overset{+\infty }{{\displaystyle \int }}}|F_n^{\alpha }\left(z\right)-\lambda \text{Φ}\left(\frac{z-{\mu }_1}{\sigma }\right)-(1-\lambda )\text{Φ}\left(\frac{z-{\mu }_0}{\sigma }\right)|dz,$$

where $F_n^{\alpha }$ is the empirical cumulative distribution function (CDF) of $z$ under $\alpha$, $\text{Φ}$ is the CDF of the standard normal distribution, $\lambda \in \left(0,1\right)$ representing $P(Y=1)$ is the mixing proportion, and ${\mu }_1$, ${\mu }_0$, and $\sigma$ (shared) are parameters of the 2 components of the normal mixture. For a given $\alpha$, we can find the maximum likelihood estimates of $\lambda$, ${\mu }_1$, ${\mu }_0$, and $\sigma$ with the expectation-maximization (EM) algorithm,^46,47 and then the distribution divergence $\mathbb{D}\left(\alpha \right)$ is calculated by plugging in these estimated parameters. To overcome overfitting and increase stability, we use bootstrap resampling to repeatedly calculate $\mathbb{D}\left(\alpha \right)$ with $\alpha$ increasing from 0 to 1 and record where the divergence starts to increase, and we obtain the final estimate of $\alpha$ as the average of those recorded points. Our numerical studies demonstrate that the normalized $z_{*}$ achieves a much higher AUC than the original $x_{*}$, for * = ICD, NLP, and ICDNLP.

Random corruption denoising

The normalized main ICD-9-CM and NLP count features do not leverage information from other features, such as counts of competing diagnoses and medication prescriptions, which provide additional characterization of the presence of the target phenotype. In fact, some of these remaining features have been shown to possess predictive values beyond the main features in supervised algorithms trained with gold-standard labels.^{7–11,13–16} We thus wish to utilize the additional information contained in the entire candidate feature set to further refine phenotype definition, but in the setting of not using any gold-standard labels.

To this end, we propose to aggregate the information in the entire candidate feature set with denoising self-regression via dropout training, a popular training method in deep learning to control overfitting.^48–50 Let $\mathit{Z}=\left[{\mathit{z}}_{\text{ICD}},{\mathit{z}}_{\text{NLP}},{\mathit{z}}_{\text{ICDNLP}},{\mathit{z}}_1,\dots ,{\mathit{z}}_p\right]$ be a data matrix whose columns are the transformed features using the above normal mixture transformation ($\alpha$ is calculated for each feature separately) and whose rows represent patients randomly sampled from the EHR database. To obtain a stable result from the dropout training, we recommend that $\mathit{Z}$ has at least 10⁵ rows. When the EHR cohort size $n$ is smaller than 10⁵, we can use bootstrap to sample 10⁵ out of $n$ with replacement. We randomly corrupt $\mathit{Z}$ and obtain $\tilde{\mathit{Z}}$, with

$${\tilde{\mathit{Z}}}_{\text{ij}}={\left(Z_{\text{ij}}\right)}^{W_{\text{ij}}}{\left(\text{Mean}\left(Z_{·j}\right)\right)}^{1-W_{\text{ij}}}$$

where $\text{Mean}\left(Z_{·j}\right)$ is the mean of the jth column of $\mathit{Z}$, and $\left\{W_{\text{ij}}\right\}$ are independent and identically distributed Bernoulli random variables, with $P\left(W_{\text{ij}}=0\right)=r$. Our empirical results suggest that a dropout rate of 20–30% works well. We let the response $y$ be one of the strong predictors in the uncorrupted $\mathit{Z}$, that is, $\mathit{y}={\mathit{z}}_{\text{ICD}}$, ${\mathit{z}}_{\text{NLP}}$, or ${\mathit{z}}_{\text{ICDNLP}}$. Then we predict $\mathit{y}$ with $\tilde{\mathit{Z}}$ using ordinary least squares regression and obtain the regression coefficient vector ${\mathit{\beta }}_{*}$, for $*=$$\text{ICD}$, $\text{NLP}$, or $\text{ICDNLP}$, depending on whether $\mathit{y}={\mathit{z}}_{\text{ICD}}$, ${\mathit{z}}_{\text{NLP}}$, or ${\mathit{z}}_{\text{ICDNLP}}$. Since the column in $\tilde{\mathit{Z}}$ that corresponds to $\mathit{y}$ has been corrupted, $\mathit{y}$ cannot be predicted by a single feature. Instead, the regression will utilize the underlying associations among all the features to recover the lost information of $\mathit{y}$. Since the recovered information must be supported by the evidence in other features, the regression is essentially a denoising process. We obtain the final PheNorm score for each patient as an inner product of the patient’s normalized feature vector $\mathit{z}={\left(z_{\text{ICD}},z_{\text{NLP}},z_{\text{ICDNLP}},z_1,\dots ,z_p\right)}^T$ and the coefficient vector

$${\text{PheNorm}}_{*}={\mathit{z}}^T{\mathit{\beta }}_{*},$$

where $\mathit{z}$ in ${\text{PheNorm}}_{*}$ is the uncorrupted version of the normalized features.

Majority vote for robustness

Without labels, it is unclear which of ${\text{PheNorm}}_{\text{ICD}}$, ${\text{PheNorm}}_{\text{NLP}}$, or ${\text{PheNorm}}_{\text{ICDNLP}}$ performs the best. Thus, we use a voting scheme to combine the 3 scores for robustness. By approximating ${\text{PheNorm}}_{\text{*}}$ ($*=$ ICD, NLP, or ICDNLP) as a normal mixture distribution, we classify a patient’s phenotype status as $G_{*}\in \{+,-\}$ according to whether the posterior probability of phenotype positive given ${\text{PheNorm}}_{\text{*}}$ is >0.5. Let $G_{\text{vote}}$ be the majority vote of $\{G_{\text{ICD}},G_{\text{NLP}},G_{\text{ICDNLP}}\}$. We let the final score be:

$${\text{PheNorm}}_{\text{vote}}={\text{Mean}\{\text{PheNorm}}_{*}:G_{*}=G_{\text{vote}}\}$$

Data and metrics for evaluation

To evaluate the performance of the PheNorm algorithm, we used Partners HealthCare EHR datamarts constructed for phenotyping rheumatoid arthritis (RA), Crohn’s disease (CD), ulcerative colitis (UC), and coronary artery disease (CAD).^7,8,13 We aimed to develop algorithms for classifying these 4 phenotypes: CAD, RA, CD, and UC. The RA datamart included the records of 46 568 patients who had at least one ICD-9-CM code of 714.x (Rheumatoid arthritis and other inflammatory polyarthropathies) or had been tested for anticyclic citrullinated peptide. The RA status was annotated by domain experts for a random sample of 435 patients. From the RA datamart, 4446 patients were predicted to have RA by Liao et al.,⁷ and of those, 758 patients who had at least one ICD-9-CM code or a free-text mention of CAD were reviewed for the CAD phenotype. Since PheNorm relies heavily on clinical notes, the records of 17 of 758 patients that did not have a note were removed from the samples. The datamart for inflammatory bowel diseases included 34 033 patients who had at least one ICD-9-CM code of 555.x (Regional enteritis) or 556.x (Ulcerative enterocolitis). For UC and CD, respectively, 600 patients randomly sampled from those with at least one corresponding ICD-9-CM code were reviewed for the target phenotype. The prevalence for CAD, RA, CD, and UC were estimated at 40.1%, 22.5%, 67.5%, and 63.0%, respectively.

For each phenotype, we used PheNorm to generate a score for each patient using $x_{\text{ICD}}$, $x_{\text{NLP}}$, and $x_{\text{ICDNLP}}$ and calculated the AUC as a metric of accuracy. The corruption rate $r$ in the denoising step was set to 0.3. For sensitivity analysis, we also trained PheNorm using a corruption rate of 0.2 and using features selected from AFEP in the denoising step. As benchmarks, we trained supervised algorithms with randomly sampled gold-standard labels of $N$ patients for $N=100$, $200$, and $300$ using the SAFE and AFEP features, and used the remaining samples to estimate the out-of-sample AUC. The algorithms were obtained from fitting adaptive elastic-net penalized logistic regression models.^32,33 The penalty parameters were chosen by the Bayesian information criterion.⁵¹ We repeated the supervised training process 30 times and report the average AUC. We also trained XPRESS algorithms as proposed in Agarwal et al.²⁹ with silver-standard labels defined as $y_{\text{silver}}=1$ if $x_{\text{NLP}}\ge 1$ and $y_{\text{silver}}=0$ if $x_{\text{NLP}}=0$. As suggested in the paper, we manually validated the dictionary for the target phenotype to ensure that there was no ambiguity. Except for CAD, we sampled 750 patients from $y_{\text{silver}}=1$ and $0$, respectively, and trained a logistic regression model with L₁ penalty. The optimal tuning parameter was selected with 5-fold cross-validation. For CAD, since the population was defined as patients from the identified RA patients who had at least an ICD-9-CM code or a free-text mention of CAD, the entire population had only 741 patients. We therefore used all patients for training. The features for the regression included all the SAFE-selected NLP features, the note count, and expert-curated codified features, including related diagnosis, prescription, and procedure codes, as well as demographic information such as age and sex (listed in Supplementary Material). In addition to XPRESS, we also experimented with the Anchor method,⁵² which was originally developed for annotating single visit notes and relies on expert-curated filters to define positive labels with high positive predictive value. Here we adapted the method for the multivisit scenario, using $x_{\text{ICD}}\ge 20$ as the filter to identify positive labels and removing $x_{\text{ICD}}$ from the predictors due to the conditional independence requirement.

We used bootstrap to estimate the standard errors of the difference in the AUC estimates from comparing different algorithms and to obtain the P-value for testing whether the difference is zero.

RESULTS

The out-of-sample AUC estimates for various algorithms are shown in Table 1. Recall that the PheNorm algorithm involves 2 main steps: (1) normalization and (2) denoising via dropout regression. Comparing the AUC of the ICD-9-CM codes before and after normalization, we found that the normalization step substantially improved the accuracy of the codes, with average improvement in AUC of around 0.04 across phenotypes. The denoising step further improved the AUC with varying degrees of magnitude depending on the phenotype and the feature; it substantially improved the AUC of the normalized ICD-9-CM count, but was not as critical for the NLP or ICD + NLP count as it was for the ICD-9-CM. Comparing the PheNorm algorithm applied to the different features, it appears that using the ICD + NLP count gave the most robust results across phenotypes, and the score based on majority voting achieved similar accuracy.

Table 1.

AUCs of the raw feature $x$, the normalized feature $z$, the PheNorm scores using SAFE feature for denoising with a dropout rate of 0.3, ${\text{PheNorm}}_{\text{vote}}$, the supervised algorithms trained with SAFE features with N = 100, 200, or 300 labels, as well as the XPRESS and Anchor algorithms.

	CAD	RA	CD	UC
$x_{\mathit{ICD}}$	0.844	0.868	0.824	0.812	Comparison is with the previous step; asterisk indicates positive increment at the significance level of 0.05.
$z_{\mathit{ICD}}$	${0.875}_{0.010}^{0.031*}$	${0.901}_{0.008}^{0.033*}$	${0.877}_{0.013}^{0.053\text{*}}$	${0.859}_{0.012}^{0.047\text{*}}$
${\mathit{PheNorm}}_{\mathit{ICD}}$	${0.899}_{0.004}^{0.024*}$	${0.929}_{0.009}^{0.028\text{*}}$	${0.911}_{0.005}^{0.033\text{*}}$	${0.900}_{0.005}^{0.041\text{*}}$
$x_{\mathit{NLP}}$	0.840	0.898	0.906	0.904
$z_{\mathit{NLP}}$	${0.864}_{0.011}^{0.025*}$	${0.923}_{0.011}^{0.025\text{*}}$	${0.947}_{0.007}^{0.041\text{*}}$	${0.931}_{0.006}^{0.026\text{*}}$
${\mathit{PheNorm}}_{\mathit{NLP}}$	${0.884}_{0.003}^{0.019*}$	${0.937}_{0.005}^{0.014\text{*}}$	${0.948}_{0.004}^{0.001}$	${0.935}_{0.002}^{0.004\text{*}}$
$x_{\mathit{ICDNLP}}$	$0.865$	$0.903$	$0.902$	$0.901$
$z_{\mathit{ICDNLP}}$	${0.895}_{0.008}^{0.030*}$	${0.935}_{0.009}^{0.032\text{*}}$	${0.944}_{0.008}^{0.042\text{*}}$	${0.933}_{0.007}^{0.032\text{*}}$
${\mathit{PheNorm}}_{\mathit{ICDNLP}}$	${0.899}_{0.002}^{0.004*}$	${0.936}_{0.002}^{0.001}$	${0.945}_{0.002}^{0.001}$	${0.935}_{0.002}^{0.002}$
${\mathit{PheNorm}}_{\mathit{vote}}$	0.899	0.937	0.945	0.933
100 labels	${0.902}_{0.013}^{0.003}$	${0.924}_{0.012}^{-0.012}$	${0.938}_{0.008}^{-0.007}$	${0.941}_{0.010}^{0.006}$	Comparison is with ${\mathit{PheNorm}}_{\mathit{ICDNLP}}$; asterisk indicates difference at the significance level of 0.05.
200 labels	${0.910}_{0.013}^{0.011}$	${0.933}_{0.014}^{-0.003}$	${0.941}_{0.009}^{-0.004}$	${0.946}_{0.010}^{0.011}$
300 labels	${0.913}_{0.015}^{0.014}$	${0.935}_{0.018}^{-0.000}$	${0.943}_{0.011}^{-0.002}$	${0.946}_{0.012}^{0.012}$
XPRESS	${0.836}_{0.012}^{-0.063\text{*}}$	${0.896}_{0.012}^{-0.040\text{*}}$	${0.905}_{0.009}^{-0.039\text{*}}$	${0.913}_{0.008}^{-0.022\text{*}}$
Anchor	${0.863}_{0.040}^{-0.035\text{*}}$	${0.890}_{0.017}^{-0.045\text{*}}$	${0.895}_{0.014}^{-0.050\text{*}}$	${0.866}_{0.017}^{-0.069\text{*}}$

Comparisons are shown in the form ${\text{AUC}}_{\text{se}(\Delta \text{AUC})}^{\Delta \text{AUC}}$, where the superscript is the increment in AUC and the subscript is the standard error of the difference in AUC.

The PheNorm algorithms using ICD + NLP count achieved an AUC comparable to that of the corresponding supervised algorithms when 100 labels were used for CAD and 300 labels were used for RA and CD. The AUC of the PheNorm score of UC appeared to be lower than those of the supervised algorithms, but an AUC of 0.935 is acceptably high when comparing across phenotypes. None of the supervised algorithms attained an AUC significantly higher than the unsupervised ${\text{PheNorm}}_{\text{ICDNLP}}$. The AUCs from the XPRESS and Anchor methods were significantly lower than that of PheNorm. In addition, as reported in the Supplementary Material, the performance of PheNorm was not sensitive to the choice of the corruption rate or feature set.

DISCUSSION

Maturation of high-throughput phenotyping technology is key to enabling phenomics – the next big challenge for the study of precision medicine.⁵³ However, scalable phenotyping relies on the ability to generate an accurate algorithm without intense involvement of clinical experts. Existing automated feature selection methods, including AFEP and SAFE, serve as a step toward automated phenotyping, but ultimately rely on expert-annotated labels to train supervised phenotyping algorithms with the selected features. PheNorm exploits the underlying distribution and association between features to aggregate them into a score for disease status classification without any gold-standard labels.

Though the training of the XPRESS algorithms does not require gold-standard labels either, our analysis indicates that the resulting algorithms have low accuracy. The suboptimal performance is due to the construction of the silver-standard labels as dichotomized versions of one of the model features, $x_{\text{NLP}}$, and hence the AUC of the XPRESS algorithms essentially approximates the AUC of $x_{\text{NLP}}$. It is also important to note that the original implementation of XPRESS used tens of thousands of features without preselection for training,²⁹ leading to further overfitting and decreased out-of-sample performance.

Using $x_{\text{ICD}}\ge 20$ as the anchor filter potentially automates the Anchor method and adapts it for patient-level multivisit phenotyping. However, this leads to algorithms with suboptimal performance. It is unclear whether alternative anchors would yield more accurate algorithms or how to choose better anchor filters that satisfy the conditional independence requirement in the multivisit setting.

The results from our numerical studies indicate that PheNorm achieves the same accuracy as supervised algorithms based on training set sizes between 100 and 300, depending on the phenotype. Current large-scale phenotyping efforts (eg, 10 phenotypes at a time) rarely have the bandwidth to offer more than 200 gold-standard labels for training for each disease, thus illustrating the potential of our method to streamline phenotyping without compromising the accuracy of a supervised approach. Additionally, our results demonstrate that the normalization step ($x\to z$) always significantly improves prediction performance, while the subsequent denoising step contributes in varying degrees. Denoising appears to be critically important for ICD-9–based algorithms, but contributes minimally to ICD + NLP–based algorithms. This suggests that the effectiveness of the denoising step is inversely related to the predictiveness of the normalized feature. In practice, one may wonder if the denoising step is still necessary, since $z_{\text{ICDNLP}}$ is typically highly accurate. We would argue that such a step is still potentially beneficial, particularly in settings where the ICD-9-CM code provides a poor characterization of the desired phenotype or the NLP software fails to accurately capture the description of the phenotype. In this case, $z_{\text{ICDNLP}}$ would benefit from the additional information in the related features offered by denoising.

Our experiments also show that ${\text{PheNorm}}_{\text{ICD}}$ and ${\text{PheNorm}}_{\text{NLP}}$ perform differently across the phenotypes, with the former better for CAD and the latter better for RA, CD, and UC. It is important to note that ${\text{PheNorm}}_{\text{ICDNLP}}$ is not necessarily a tradeoff between ${\text{PheNorm}}_{\text{ICD}}$ and ${\text{PheNorm}}_{\text{NLP}}$, although $x_{\text{ICDNLP}}=x_{\text{ICD}}+x_{\text{NLP}}$ and its accuracy may surpass both. Though ${\text{PheNorm}}_{\text{ICDNLP}}$ consistently performed well for the 4 phenotypes, it is difficult to determine which score to use in practice without gold-standard labels for validation. Our empirical studies indicate that ${\text{PheNorm}}_{\text{vote}}$ is a good hedging policy, as it consistently achieved accuracy close to the best performing one.

The PheNorm score can be converted to a predicted probability of having the disease phenotype using the EM algorithm. If the goal of the phenotyping is to link the phenotype to genomic data, one can directly use the predicted probability as a continuous trait and perform association analysis by fitting a quasi-binomial model. In fact, one could gain power by leveraging the predicted probability, as compared to converting the probability to a binary trait.⁵⁴ When a small number of labels are available for validation, one can use these labels to estimate the receiver operating characteristic (ROC) curve and then select a threshold value optimizing the tradeoff between positive predictive value and sensitivity.¹⁴

Though our results demonstrate the ability of PheNorm to provide accurate phenotyping for 4 different diseases in the absence of gold-standard labels, further work is needed to understand the performance of our method across a diverse range of phenotypes, particularly for phenotypes that have more subtle definitions, in which case a combination of PheNorm and handcrafted rules or regular expressions might be effective. Additionally, while PheNorm eliminates the annotation typically required for algorithm estimation, labeled examples are still needed to evaluate the algorithm’s accuracy. Future research is thus warranted in unsupervised approaches to estimating the ROC parameters where the statistical inference is particularly challenging.

CONCLUSION

In this paper, we introduce PheNorm for training accurate phenotyping algorithms without using gold-standard labels. In our road map to high-throughput phenotyping, we have decomposed the task into automated feature curation and algorithm training without gold-standard labels. The former goal has been achieved by AFEP and SAFE, and the latter goal has now been achieved by PheNorm. PheNorm is easy to implement, and its accuracy is similar to algorithms trained with gold-standard labels. The bandwidth provided by SAFE + PheNorm can potentially reduce the algorithm development process from months to a few days, providing the valuable phenotypic big data necessary for the study of precision medicine.⁵⁵

Funding

This work was supported by US National Institutes of Health grants U54-HG007963, U54-LM008748, R01-HL089778, R01-HL127118, F31GM119263-01A1, and K23-DK097142, the Harold and Duval Bowen Fund, and internal funds from Tsinghua University and Partners HealthCare.

Competing Interests

None.

Contributors

All authors made substantial contributions to: conception and design; acquisition, analysis, and interpretation of data; drafting the article or revising it critically for important intellectual content; and final approval of the version to be published.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.