sureLDA: A multidisease automated phenotyping method for the electronic health record

Abstract

Objective

A major bottleneck hindering utilization of electronic health record data for translational research is the lack of precise phenotype labels. Chart review as well as rule-based and supervised phenotyping approaches require laborious expert input, hampering applicability to studies that require many phenotypes to be defined and labeled de novo. Though International Classification of Diseases codes are often used as surrogates for true labels in this setting, these sometimes suffer from poor specificity. We propose a fully automated topic modeling algorithm to simultaneously annotate multiple phenotypes.

Materials and Methods

Surrogate-guided ensemble latent Dirichlet allocation (sureLDA) is a label-free multidimensional phenotyping method. It first uses the PheNorm algorithm to initialize probabilities based on 2 surrogate features for each target phenotype, and then leverages these probabilities to constrain the LDA topic model to generate phenotype-specific topics. Finally, it combines phenotype-feature counts with surrogates via clustering ensemble to yield final phenotype probabilities.

Results

sureLDA achieves reliably high accuracy and precision across a range of simulated and real-world phenotypes. Its performance is robust to phenotype prevalence and relative informativeness of surogate vs nonsurrogate features. It also exhibits powerful feature selection properties.

Discussion

sureLDA combines attractive properties of PheNorm and LDA to achieve high accuracy and precision robust to diverse phenotype characteristics. It offers particular improvement for phenotypes insufficiently captured by a few surrogate features. Moreover, sureLDA’s feature selection ability enables it to handle high feature dimensions and produce interpretable computational phenotypes.

Conclusions

sureLDA is well suited toward large-scale electronic health record phenotyping for highly multiphenotype applications such as phenome-wide association studies .

INTRODUCTION

Electronic health records (EHRs), often linked with biorepositories, have become an increasingly important data source for translational research.^1,2 Such rich, multidimensional data promise myriad opportunities for translational applications ranging from personalizing care decisions to predicting disease prognosis.³ However, the scarcity of precise phenotype labels has hampered efforts to harness this dataset for many of these objectives. Studies focusing on one or a few phenotypes typically circumvent this problem by predicting labels from diverse EHR features using (1) rule-based algorithms or (2) supervised learning methods trained on a subset of manually anontated gold-standard labels (GLabels).^4–11 Of note, the Phenotype Knowledgebase (PheKB) platform built for the eMERGE (Electronic Medical Records and Genomics) Network has been shown to effectively integrate expertise across sites to yield accurate, transportable phenotyping algorithms.¹¹ However, applying these approaches to new phenotypes requires substantial expert input: manually annotating GLabels to train supervised methods necessitates laborious chart review, and formulating rule-based algorithms involves iteratively devising and validating rules. Thus, these approaches, while effective, are infeasible for highly multiphenotype applications such as phenome-wide association studies (PheWAS) requiring de novo labeling of hundreds to thousands of phenotypes. This signifies an ongoing need for phenotyping methods that can simultaneously annotate many diverse phenotypes.

Currently, studies requiring many EHR phenotypes often utilize billing codes such as International Classification of Diseases (ICD) codes as surrogates for true phenotype labels. For instance, the original PheWAS^12–14 grouped ICD codes into ∼1800 phenotype codes (phecodes) that were subsequently thresholded and screened for associations with various genetic markers. While a trivial function of ICD codes can be reliably used in lieu of true labels for many diseases, others such as rheumatoid arthritis¹⁵ tend to have inprecise codes that can diminish the power of the association study.¹⁶ Thresholding ICD codes at higher counts boosts label positive predictive value and specificity but may significantly diminish sensitivity, especially for episodic conditions such as pseudogout.

To produce more accurate phenotype labels without extensive expert intervention, researchers have recently advocated for label-free (ie, not requring GLabels) computational phenotyping methods.^17–24 The class of “weakly supervised” methods, which train supervised classifiers using noisy labels generated from key surrogate features in the data rather than expensive GLabels, has proven particularly powerful to this end. For instance, the “anchor and learn” approach trains a regularized logistic regression model on imperfect labels derived from “anchor” features with high positive predictive value.^17,18 While this approach avoids GLabels, it requires expert input to identify appropriate anchors. Several other efforts have fully automated the pipeline by using knowledge base–derived labels or standardized “silver standard” surrogates, such as the ICD code or natural language processing (NLP) mention for the target phenotype, to train classifiers. For instance, Levine et al²⁴ demonstrated high concordance between lagged linear regression models trained using a knowledge base–derived standard vs an expert clinician–derived one. Similarly, the XPress method predicts a phenotype by fitting a regularized logistic regression on a noisy label defined as the presence of at least 1 key ICD code.¹⁹ The PheNorm method uses multiple surrogates derived from the main ICD and NLP by (1) fitting log-normal mixtures adjusting for healthcare utilization and (2) performing additional denoising with other features via dropout training.²⁰ Likewise, the multimodal automated phenotyping (MAP) algorithm fits a multimodal clustering ensemble to the ICD and NLP surrogates adjusting for healthcare utilization.²¹

These automated methods often achieve impressive accuracy and can be practically scaled to a large number of phenotypes. However, we hypothesize 2 means of improvement: (1) better incorporating information from nonsurrogate features and (2) jointly predicting phenotypes in order to impose an Occam’s razor²⁵ assumption that fewer concurrent diseases is more likely than more. To achieve these goals, we draw inspiration from the UPhenome algorithm—a modification of the widely used latent Dirichlet allocation (LDA) topic model²⁶ that combines diverse features to infer patients’ joint distributions over unbiased phenotype “topics.”²⁷ Though UPhenome was developed for phenotype discovery rather than annotation of known phenotypes, the grounded phenome (GPhenome) algorithm derived therefrom better reflects known phenotypes by setting topic prior to the pseudocount of clinical terms associated with each phenotype, thus portending potential application to an annotation task.²⁸

In this article, we propose surrogate-guided ensemble LDA (sureLDA), an automated multiphenotype annotation method that employs probabilistic pseudolabels produced by PheNorm to guide topic inception in LDA in a “weakly supervised” manner. As with GPhenome, this guiding of LDA is inspired by 2 well-trodden LDA derivatives, labeled LDA and semisupervised LDA,^29,30 though using only surrogates rather than GLabels to constrain topics. Unlike GPhenome, sureLDA (1) utilizes phenotype probabilities from PheNorm rather than clinical concept counts as “noisy” labels to guide topic formation in LDA, (2) weights features using regression coefficients from the dropout training step of PheNorm, and (3) employs a cluster ensemble approach to combine guided LDA scores with surrogates to yield posterior phenotype probabilities. sureLDA combines desirable properties of PheNorm and LDA, effectively leveraging the typically informative surrogate features using PheNorm while exploiting LDA’s prowess at extracting information from high-dimensional feature spaces to jointly annotate multiple phenotypes. Consequently, we submit that sureLDA is uniquely well suited to high-throughput phenotyping of the EHR for a highly multiphenotype application akin to PheWAS.

MATERIALS AND METHODS

The sureLDA algorithm broadly consists of 4 key steps: (1) assemble informative features, including the main ICD and NLP features as silver-standard surrogates for each target phenotype, as well as a healthcare utilization feature ($H$); (2) use PheNorm to obtain initial probabilities for each target phenotype based on these surrogates together with $H$; (3) fit guided LDA to all features simultaneously using the initial probabilities as Dirichlet hyperparameters for the patient-phenotype distributions, yielding patient-level phenotype “scores”; and (4) perform ensemble clustering of the surrogates and the LDA phenotype scores (again adjusting for $H$) to predict final probabilities for each target phenotype. A schematic illustrating this procedure is displayed in Figure 1. Throughout, we assume that there are a total of $K$ target phenotypes with true binary status denoted by $\mathbf{Y}=(Y_1,\dots ,Y_K)\mathrm{\text{'}}$, which again is not used to train sureLDA. Let $\mathbf{X}$ denote the entire feature vector of dimension $p$ and ${\mathbf{X}}^{\mathit{log}}=\log (\mathbf{X}+1)$. We assume that there are a total of $D$ patients in the EHR study and use subscript $d$ to index the patients.

Figure 1.

Schematic of the surrogate-guided ensemble latent Dirichlet allocation (LDA) algorithm.

Assembling informative features

The main ICD and NLP surrogates are counts of the corresponding ICD code(s) and NLP-curated mentions of the phenotype in a patient’s chart. For the $k$th phenotype, we denote these surrogates as $\mathit{ICD}{\mathrm{ }}_k$ and $\mathit{NLP}{\mathrm{ }}_k$ respectively. Let $\mathit{ICD}{\mathrm{ }}_k^{\mathit{log}}=\log \mathrm{ }(\mathit{ICD}{\mathrm{ }}_k+1)$, $\mathit{NLP}{\mathrm{ }}_k^{\mathit{log}}=\log \mathrm{ }(\mathit{NLP}{\mathrm{ }}_k+1)$, and $\mathit{ICDNLP}{\mathrm{ }}_k^{\mathit{log}}=\log \left(\mathrm{ }\mathit{ICD}{\mathrm{ }}_k+\mathrm{ }\mathit{NLP}{\mathrm{ }}_k+1\right)$. These 2 key features can be mapped automatically as in Liao et al²¹ using existing knowledge sources including the PheWAS catalogue¹² and the Unified Medical Language System. Additional candidate features—including counts of other ICD codes, NLP features, drug prescriptions, lab tests, and procedure codes—can be identified automatically without GLabels via existing methods such as the Surrogate-Assisted Feature Extraction (SAFE) method.²³ Because sureLDA has the capacity to select useful features for each phenotype, it is preferable to be inclusive rather than aiming for specificity when assembling these additional features.

Initializing prior probabilities using PheNorm

We obtain prior probabilities for each of the $K$ target phenotypes, denoted by $\mathit{\pi }=({\pi }_1,\dots ,{\pi }_K)\mathrm{\text{'}}$, using a slightly modified PheNorm algorithm. In brief, standard PheNorm estimates ${\pi }_k$ by (1) normalizing $\mathit{ICD}{\mathrm{ }}_k^{\mathit{log}}$, $\mathit{NLP}{\mathrm{ }}_k^{\mathit{log}}$, and $\mathit{ICDNLP}{\mathrm{ }}_k^{\mathit{log}}$ against $H^{\mathit{log}}=\log \left(H\right)$ via Gaussian mixture regression; and (2) further de-noising these normalized surrogate features using ${\mathbf{X}}^{\mathit{log}}$ via dropout regression.²⁰ We use ensemble clustering to estimate ${\pi }_k$ by fitting 2-class Gaussian mixture models to each of the normalized surrogate features and taking the mean of the predicted probabilities from the 3 models. For most diseases, patients with $\mathit{ICD}{\mathrm{ }}_k=0$—here described as filter negative—very rarely have the disease, so we set ${\pi }_k=0$ for these patients. The one exception in our study is obesity, which has an unusually insensitive ICD code and for which we define filter negative as $\mathit{ICD}{\mathrm{ }}_k+\mathit{NLP}{\mathrm{ }}_k=0$.

Fitting-guided LDA

Model overview

LDA is a fully specified topic model that models $D$ documents (here, patients) as mixtures of $K^{\mathrm{*}}$ topics (here, phenotypes) in which each topic is defined as a distribution over a vocabulary of $\mathrm{p}$ words (here, EHR features); for simplicity, we will henceforth discuss the model in terms of patients, phenotypes, and features. To model $K$ target phenotypes, we set $K^{\mathrm{*}}=K+K_0$ topics where the first $K$ topics are assigned sequentially to target phenotypes $1,\dots ,K$ and the remaining $K_0$ are “agnostic” to model additional structure (ie, nontarget phenotypes) in the data. The generative model of standard LDA iterates over 3 key steps: (1) draw a phenotype mixture ${\mathit{\theta }}_d=({\theta }_{1,d},\dots ,{\theta }_{K^{\mathrm{*}},d})\mathrm{\text{'}}$ for patient $d$ from a Dirichlet distribution with $K^{\mathrm{*}}$-dimensional hyperparameter $\mathit{\alpha }=({\alpha }_1,\dots ,{\alpha }_{K^{\mathrm{*}}})\mathrm{\text{'}}$; (2) draw a feature mixture ${\varphi }_k=({\varphi }_{1,k},\dots ,{\varphi }_{p,k})\mathrm{\text{'}}$ for phenotype topic $k$ from a second Dirichlet with $p$-dimensional hyperparameter $\mathit{\beta }$; and (3) for the $j$th feature, iteratively sample feature-to-phenotype assignment counts ${\mathit{Z}}_{dj}=(Z_{dj,1},\dots ,Z_{dj,K^{\mathrm{*}}})\mathrm{\text{'}}$ from Multinomial$(X_{dj},{\mathbf{P}}_{dj})$ distributions with probability vector ${\mathbf{P}}_{dj}=(P_{dj,1},\dots ,P_{dj,K^{\mathrm{*}}})\mathrm{\text{'}}$ calculated based on the current parameter values, where $X_{dj}={\sum }_{k=1}^{K^{\mathrm{*}}}{Z}_{dj,k}$.²⁶ Our guided LDA algorithm resembles the labeled LDA algorithm,²⁹ except that whereas labeled LDA requires GLabels to guide topic formation, here we guide topics by setting the Dirichlet hyperparameter for the first $K$ phenotype topic distributions to the initial PheNorm probabilities $\mathit{\pi }$.

Feature weighting

A major shortcoming of classical LDA is that it weights all terms equally and thus loses precision in the presence of frequent yet uninformative terms called stop words.²⁶ To discount these uninformative features, researchers have experimented with term weights including term frequency (TF), inverse document frequency (IDF), the product TF$\times$IDF, and pointwise mutual information.³¹ However, these weighting schemes are (1) not reflective of the actual informativeness of a term for a topic and (2) invariable for a term across topics. In our real EHR data applications, we weight features by the coefficients from the dropout regression step of $\mathit{ICDNLP}{\mathrm{ }}_k^{\mathit{log}}$ against ${\mathbf{X}}^{\mathit{log}}$ in PheNorm.²⁰ Features with a negative dropout regression coefficient for a phenotype are assigned a weight of 0 for the corresponding topic. We henceforth denote the weight of feature $j$ for topic $k$ as ${\omega }_{j,k}$.

Implementation and inference

We implemented and trained guided LDA using collapsed Gibbs sampling.³² Collapsed Gibbs sampling is well suited to sureLDA as we initialize feature-phenotype assignments at their expectations under the PheNorm prior, thus nudging the Gibbs sampler toward a (likely local) optimum within the “vicinity” of the PheNorm solution. Moreover, initialized in this way the Gibbs sampler need only be run once, whereas variational Bayes would necessitate multiple runs to guarantee finding a reasonably good solution. Our modification of LDA does not substantively affect its Gibbs conditionals, allowing for efficient implementation of this well-trodden procedure. In collapsed Gibbs sampling $\mathbf{\Theta }=({\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_D)$ and $\mathbf{\Phi }=({\mathbf{\Phi }}_1,\dots ,{\mathbf{\Phi }S}_{K^{\mathrm{*}}})$ are marginalized out, so we directly sample the posterior feature-to-disease assignment counts ${\mathit{Z}}_{dj}$ without first sampling $\mathbf{\Theta }$ and $\mathbf{\Phi }$. The algorithm iteratively updates the weighted count of total features assigned to disease topic $k$ for patient $d$, $N_{dk}={\sum }_{j=1}^p{\omega }_{j,k}{Z}_{dj,k}$, and the weighted count of feature $j$ assigned to disease topic $k$, $M_{j,k}={\omega }_{j,k}{\sum }_{d=1}^D{Z}_{dj,k}$, by generating ${\mathit{Z}}_{dj}\sim \mathrm{ }\mathit{Multinomial}\mathrm{ }(X_{dj},{\mathbf{P}}_{dj})$ where

$$P_{dj,k}=\frac{N_{dk}-{\omega }_{j,k}{Z}_{dj,k}+{\mathcal{P}}_{dk}}{\sum _{k=1}^K\left(N_{dk}-{\omega }_{j,k}{Z}_{dj,k}+{\mathcal{P}}_{dk}\right)}\times \frac{M_{kj}-{\omega }_{j,k}{Z}_{dj,k}+\beta }{\sum _{j=1}^p\left(M_{kj}-{\omega }_{j,k}{Z}_{dj,k}+\beta \right)}$$

See algorithm 1 in the Supplementary Appendix for details of the algorithm. For the choice of $K^{\mathrm{*}}$, we found that including additional “agnostic” topics with uninformative Dirichlet priors (${\mathcal{P}}_{dk}=1$) improved performance up to $K_0=15$ additional topics, beyond which performance remained stable up to 100 additional topics. We thus set $K^{\mathrm{*}}$ to $K+20$.

Obtaining final probabilities using ensemble clustering

When guided LDA converges, we obtain the mean weighted count of total features assigned to topic $k$ for subject $d$, $N_{dk}$, as the sureLDA score. While ${\mathbf{N}}_k=(N_{d,k},\dots ,N_{D,k})\mathrm{\text{'}}$ is predictive of ${\mathbf{Y}}_k$, it is subject to noise from healthcare utilization $H$. We therefore obtain our final sureLDA phenotype probability predictions via clustering ensemble normalized by $H$. Specifically, for each $k$ we perform a clustering of ${\mathbf{N}}_k^{\mathit{log}}=\log ({\mathbf{N}}_k+1)$ by fitting a Gaussian mixture regression model adjusting for $H^{\mathit{log}}$, and then average this clustering probability with the 3 PheNorm clustering probabilities based on $H^{\mathit{log}}$-normalized $\mathit{ICD}{\mathrm{ }}_k^{\mathit{log}}$, $\mathit{NLP}{\mathrm{ }}_k^{\mathit{log}}$, and $\mathit{ICDNLP}{\mathrm{ }}_k^{\mathit{log}}$.

Data and metrics for evaluation

We evaluated the performance of sureLDA using both simulated datasets and real-world EHR data from the Partners HealthCare Biobank.³³

Simulation study

In the simulation study, we considered a 5400-dimensional feature space over $K=27$ disease phenotypes (200 features per disease including 2 surrogate features each) in addition to $H$ for $N=100\mathrm{ }000$ patients. To test the robustness of our method’s performance to different phenotype characteristics, we simulated 27 phenotypes enumerating combinations of low, medium, and high levels in (1) prevalence, (2) surrogate feature informativeness, and (3) nonsurrogate feature informativeness. Mixture models were used to generate $\mathbf{X}$, $H$, and $\mathbf{Y}$. Details of the simulation generative models are given in the Supplementary Appendix. The feature weights ${\omega }_{j,k}$ were uniformly set to 1 in the simulations because no prior knowledge was used in data generation. Results are summarized based on 100 independent and identically distributed simulated datasets.

Real EHR data analysis

The Partners Healthcare Biobank consists of both codified data (ie, ICD codes) and free text from clinical notes. We considered 10 target phenotypes—asthma, breast cancer, chronic obstructive pulmonary disease, depression, epilepsy, hypertension, schizophrenia, ischemic stroke, type 1 diabetes mellitus, and obesity—to validate our algorithm’s predictive accuracy. These target diseases cover a broad range of acuity, prevalence, and diagnostic ambiguity to test our method’s versatility. We included data from $N=38\mathrm{ }023$ patients observed mostly between 1990 and 2015, when the labels were curated. Moreover, we obtained a total of $p=2125$ features consisting of age, body mass index, 110 ICD codes, 225 NLP features counting relevant concepts in clinical notes, all Current Procedural Terminology codes grouped into 264 categories according to the Clinical Classifications Software,³⁴ and 1534 RxNorm drug codes available in the data. NLP features were selected by the SAFE method, while no selection was performed for Current Procedural Terminology groups and drug codes. ICD codes were assembled by domain experts in prior studies. Continuous variables such as body mass index were rounded to the nearest integer and treated as ordinal in the LDA step. Gold standard labels were manually curated via chart review for 585 patients.

Benchmark methods for comparison

For each dataset, in addition to fitting sureLDA we considered multiple benchmark approaches: (1) classical LDA automatically assigning latent topics to phenotypes using Spearman’s rank correlation with the ICD surrogates; (2) UPhenome, again using Spearman’s rank correlation with the ICD surrogates to assign topics to phenotypes (only with the Biobank data—we could not apply UPhenome in our simulations as generated features were not assumed to have different datatypes); (3) GPhenome using $\mathit{ICD}{\mathrm{ }}_k+\mathit{NLP}{\mathrm{ }}_k$ as the Dirichlet prior for phenotype topic k; (4) the XPress algorithm using $I\mathrm{ }(\mathit{ICD}{\mathrm{ }}_k\ge 1)$ as the proxy label for $Y_k$; (5) PheNorm; and (6) MAP. We also tested 2 supervised models, (1) LASSO-penalized logistic regression and (2) random forest, each with $n_t$ training labels where we let $n_t$ vary from 100 to 300 to assess how sureLDA compares with supervised phenotyping methods. In light of Rajkomar et al’s³⁵ recent result that LASSO logistic regression rivals deep learning methods on a set of high-dimensional EHR-based prediction tasks, and given the limited number of GLabels, we did not include more complex supervised methods. Hyperparameters for LASSO and random forest were optimized using 10-fold cross-validation. Two- and 5-fold cross-validation as well as random subsampling cross-validation yielded statistically equivalent optimized hyperparameters and predictive accuracy for all phenotypes in the Partners EHR dataset (data not shown).

Evaluation metrics

We used 3 evaluation metrics to quantify the predictive performance of sureLDA and its comparators: (1) area under the receiver-operating characteristic curve (AUC), (2) F-score, and (3) label rank loss.³⁶ For F-score, we chose the cutoff value to achieve a specificity of 95% among filter-positive samples. AUC and F-score reflect the sensitivity and specificity and precision and recall of predictions, respectively. On the other hand, label rank loss computes the average number of probabilistic label pairs for a patient that are incorrectly ordered weighted by the inverse of the number of ordered pairs of false and true labels, reflecting the degree to which the relative disease probabilities for a given patient align with that patient’s true overall phenotype. Thus, whereas the first 2 evaluation metrics measure the quality of predictions for a phenotype, the third measures accuracy on the patient level and thereby reflects a model’s capacity to jointly predict multiple phenotypes. Standard errors for all measurements were obtained by bootstrapping with 100 bootstrap samples. To evaluate methods’ robustness to diverse phenotypes, we considered the standard deviation of ${\mathrm{\Delta }=\mathit{AUC}}_{\mathit{max}}-{\mathit{AUC}}_{\mathit{method}}$ across Partners Biobank phenotypes, where for each phenotype ${\mathit{AUC}}_{\mathit{max}}$ denotes the maximum AUC achieved across label-free phenotyping methods, and ${\mathit{AUC}}_{\mathit{method}}$ denotes the AUC of the specific method. Finally, for the Partners EHR data, we qualitatively assessed whether the phenotype topics inferred in the guided LDA step make clinical sense by generating feature clouds using the weighted counts of features assigned to each phenotype topic, $\left\{M_{j,k}\right\}$.

RESULTS

Results on simulated datasets

Figure 2 shows mean accuracies across 27 simulated phenotypes with variable generative parameters. Figure 3 shows AUCs and F scores as a function of (1) prevalence, (2) surrogate feature informativeness, and (3) nonsurrogate feature informativeness. sureLDA, classical LDA, and GPhenome’s accuracy (per both AUC and F score) improved significantly with the informativeness of both surrogate and nonsurrogate features. MAP improved only with surrogate informativeness, as expected. PheNorm and Xpress improved markedly with surrogate informativeness and marginally with nonsurrogate informativeness. Finally, increasing phenotype prevalence improved F scores but not AUCs (which, as a metric, is prevalence-agnostic) for all methods except XPress.

Figure 2.

Mean (A) areas under the receiver-operating characteristic curves, (B) F-scores, and (C) label rank losses of phenotype predictions on simulated datasets comparing raw International Classification of Diseases (ICD) and natural language processing (NLP) surrogates (red), fully unsupervised phenotyping methods (blue), alternative weakly supervised methods (green), surrogate-guided ensemble latent Dirichlet allocation (sureLDA) (purple), supervised phenotyping using LASSO regularized logistic regression (orange), and random forest (RandFor) (yellow) with 100-300 true labels. Error bars reflect empiric bootstrapped 95% confidence intervals. GPhenome: grounded phenome; MAP: multimodal automated phenotyping.

Figure 3.

Mean areas under the receiver-operating characteristic curves (ROC-AUCs) (top) and F-scores (bottom) of Xpress (red), classical latent Dirichlet allocation (LDA) (blue), grounded phenome (GPhenome) (green), PheNorm (purple), multimodal automated phenotyping (MAP) (orange), and surrogate-guided ensemble LDA (sureLDA) (yellow) on simulated datasets with variable (A) phenotype prevalence, (B) surrogate feature informativeness, and (C) nonsurrogate feature informativeness. For each plot, the 2 variables not being varied are held at their “medium” levels.

Results on Partners EHR data

Figure 4 shows mean accuracies across 10 diverse diseases in real-world EHR data. sureLDA exhibited statistically significant improvements in mean AUC relative to other label-free alternatives. For example, compared with MAP, PheNorm, Xpress, and GPhenome, sureLDA improved mean AUCs by 0.021 (95% confidence interval [CI], 0.004-0.038), 0.033 (95% CI, 0.008-0.052), 0.131 (95% CI, 0.100-0.170), and 0.054 (95% CI, 0.028-0.084), respectively. Compared with supervised algorithms, sureLDA achieved statistically significant improvements in mean AUC as well as improvements (though not significant) in mean F score relative to both LASSO and random forest with 300 GLabels. sureLDA also exhibited relatively low standard errors in all accuracy metrics. As shown in Supplementary Figure 3, sureLDA’s deviation from the highest AUC achieved per disease had the lowest standard deviation (0.029), with only PheNorm (0.044) and GPhenome (0.046) achieving comparable robustness by this metric. These results were generally consistent with our simulation study.

Figure 4.

Mean (A) areas under the receiver-operating characteristic curves, (B) F-scores, and (C) label rank losses of phenotype predictions on real diseases from the Partners HealthCare Biobank comparing raw International Classification of Diseases (ICD) and natural language processing (NLP) surrogates (red), fully unsupervised phenotyping methods (blue), alternative weakly supervised methods (green), surrogate-guided ensemble latent Dirichlet allocation (sureLDA) (purple), supervised phenotyping using LASSO regularized logistic regression (orange), and random forest (RandFor) (yellow) with 100-300 true labels. Error bars reflect empiric bootstrapped 95% confidence intervals. GPhenome: grounded phenome; MAP: multimodal automated phenotyping.

Per the label rank loss metric, which reflects the accuracy of relative phenotype predictions on the patient level, sureLDA achieved statistically equal loss to MAP, PheNorm, ICD counts, and random forest with 300 labels. GPhenome, UPhenome, and the classical LDA’s losses are confounded by the fact that they output topic-feature counts—which vary in scale between phenotypes—rather than probabilities. These results depart from our simulation, in which sureLDA achieved the lowest loss, statistically equivalent only to PheNorm. This disparity may be attributable to the fact that our simulation has 27 phenotypes, whereas our real-world EHR example has only 10, which may not be sufficient to observe the benefit of jointly modeling phenotypes over predicting each marginally. Nevertheless, sureLDA’s low rank loss in our simulation study, combined with the qualitative observation that sureLDA consistently achieves higher mean AUCs and F scores the more phenotypes we include (data not shown), suggests that sureLDA benefits from jointly modeling phenotypes.sureLDA performed statististically equivalently to or better than MAP and PheNorm for all phenotypes, suggesting that its anchoring to the PheNorm prior ensures reliable baseline accuracy. Consistent with our simulation results, sureLDA generally performed best relative to alternative label-free methods for phenotypes with informative nonsurrogate features and at least 1 noisy surrogate, such as chronic obstructive pulmonary disease, epilepsy, and ischemic stroke (Supplementary Figures 1 and 2 and Supplementary Table 1). However, sureLDA along with MAP and PheNorm were significantly outperformed by GPhenome for obesity, which has unusually insensitive ICD and NLP surrogates. This suggests that sureLDA’s tethering to PheNorm may occasionally hinder it from fully exploiting information distributed across nonsurrogate features.

We present in Figure 5 and Supplementary Figure 5 feature clouds demonstrating the 20 features sureLDA associates most closely with each target phenotype “topic.” Almost all of these features have an intuitive association with their corresponding phenotype. For instance, highly weighted features for the epilepsy topic include the ICD code for epilepsy, anticonvulsive drugs including phenytoin, lamotrigine, and carbamazepine; the diagnostic test electroencephalogram; and various NLP terms including “seizures” and “aura”—a multitype set of features clearly reflecting a diagnosis of epilepsy (Figure 5A). Topics for obesity (Figure 5B), schizophrenia (Figure 5C), and breast cancer (Figure 5D) follow similar patterns. These results underline sureLDA’s robust feature selection properties in the high-dimensional data setting,

Figure 5.

Feature clouds derived from $M_{j,k}$ for 4 representative disease phenotypes: (A) epilepsy, (B) obesity, (C) schizophrenia, and (D) breast cancer. Natural language processing (NLP) terms have the prefix “NLP” and codified data the prefix “ICD.” EGF: Epidermal Growth Factor; ICD: International Classification of Diseases.

DISCUSSION

Automated, label-free phenotyping methods enable EHR studies for which manually annotating phenotypes or designing new rule-based algorithms are impractical, such as a PheWAS requiring a large number of phenotypes to be defined and labeled de novo. To this end, sureLDA enables efficient annotation of accurate, interpretable computational phenotype labels that are robust to phenotype prevalence and feature set properties (Figure 3 andSupplementary Figure 3).sureLDA combines attractive properties of classical LDA and PheNorm. By leveraging PheNorm to guide topic formation in LDA in a weakly supervised manner, sureLDA focuses LDA’s structure identification prowess toward identifying a well-defined set of phenotypes. It also leverages PheNorm’s dropout regression step to enable a data-driven phenotype-specific feature weighting mechanism that outperforms the literature-standard TF-IDF weighting scheme (Supplementary Figure 4).

Likewise, sureLDA augments PheNorm by using LDA to better extract information from nonsurrogate features. LDA is adept at attributing marginally ambiguous features such as “cough” or “headache” to the most patient-relevant phenotype topic. In this way, it implicitly imposes an Occam’s razor assumption that favors fewer concurrent phenotypes over more, reflecting the negative association that arises between phenotypes conditional on ambiguous features. Future work is warranted to better assess the benefit of this property in the real-world EHR setting.

LDA also gives sureLDA its aptitude for feature selection. As the word clouds in Figure 5 and Supplementary Figure 5 demonstrate, the sureLDA “scores” produced by guided LDA reflect consistently meaningful features. Given the noisy and high-dimensional nature of EHR data, sureLDA’s feature selection ability primes it for EHR modeling. Moreover, it enables fast production of interpretable computational phenotypes.

This combination of desirable properties from PheNorm and LDA explains sureLDA’s robustness to a diversity of phenotypes (Figure 3 and Supplementary Figure 3). For phenotypes with highly informative surrogates, the use of PheNorm ensures high baseline accuracy. For phenotypes with noisier surrogates and informative nonsurrogate features, the LDA step allows for significant improvement over PheNorm. This robustness makes sureLDA well suited to simultaneously annotating many diverse phenotypes.sureLDA could also be used to identify subphenotypes similarly to Li et al.³⁷ Once a phenotype (ie, heart failure) cohort is identified (potentially via sureLDA), one could employ sureLDA on these patients to classify subphenotypes. However, because sureLDA uses PheNorm to guide topics, one would need to prespecify a few known subphenotypes (ie, systolic heart failure with renal complications) and use their relevant ICD and NLP features to guide the subphenotype topics. Additionally, sureLDA’s “agnostic” topics could be used to discover unknown (sub)phenotypes, though its advantage over standard LDA for this task is unclear. Phenotype discovery is also limited by what information is documented in the EHR, which inherently reflects known phenotypes. While sureLDA eliminates reliance on expert review for annotation, it does require gold-standard labels for evaluation of its own performance. Continued work is warranted to estimate performance parameters such as AUC and F-score in an unsupervised fashion and thereby make both implementation and evaluation of weakly supervised phenotyping methods like sureLDA fully label-independent.

CONCLUSION

In this article, we introduce sureLDA, a high-throughput automated EHR phenotyping method that combines PheNorm with LDA to jointly annotate multiple phenotypes without using GLabels. Our method produces accurate, interpretable labels for a broad range of phenotypes. sureLDA exhibits particular improvement over existing label-free phenotyping methods for phenotypes without accurate surrogate features. Given these qualities, sureLDA promises to enable more powerful use of EHR data for highly multiphenotype applications such as PheWAS.

FUNDING

This work was supported by National Institutes of Health grants T32-AR5588511 (YA), T32-GM7489714 (YA), and R21-CA242940 (TC).

AUTHOR CONTRIBUTIONS

All authors made substantial contributions to conception and design; acquisition, analysis, and interpretation of data; drafting the article or revising it critically for important intellectual content; and final approval of the version to be published.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

None declared.