A general framework for developing computable clinical phenotype algorithms

Abstract

Objective

To present a general framework providing high-level guidance to developers of computable algorithms for identifying patients with specific clinical conditions (phenotypes) through a variety of approaches, including but not limited to machine learning and natural language processing methods to incorporate rich electronic health record data.

Materials and Methods

Drawing on extensive prior phenotyping experiences and insights derived from 3 algorithm development projects conducted specifically for this purpose, our team with expertise in clinical medicine, statistics, informatics, pharmacoepidemiology, and healthcare data science methods conceptualized stages of development and corresponding sets of principles, strategies, and practical guidelines for improving the algorithm development process.

Results

We propose 5 stages of algorithm development and corresponding principles, strategies, and guidelines: (1) assessing fitness-for-purpose, (2) creating gold standard data, (3) feature engineering, (4) model development, and (5) model evaluation.

Discussion and Conclusion

This framework is intended to provide practical guidance and serve as a basis for future elaboration and extension.

Background and motivation

Computable clinical phenotype algorithms are used to identify a wide variety of health conditions for epidemiologic, clinical, and health services research and for medical product safety surveillance. With appropriate privacy safeguards and stakeholder approval, such algorithms use electronic healthcare data to identify patients with specific clinical conditions (phenotypes). Both the data inputs and the modeling methods used in developing algorithms range from simple to complex. Some phenotypes such as acute pancreatitis¹ and HIV infection^2–4 can be accurately identified by relatively simple, manually crafted rules applied to structured claims data (eg, diagnosis, procedure, and medication codes). Identifying more clinically complex phenotypes may require marshalling more diverse and subtle information from electronic health records (EHRs), introducing additional measurement and modeling challenges. Anaphylaxis is an example of a difficult phenotype,^5,6 requiring synthesis of information about exposures and subjectively assessed symptoms during a short time period.⁷

The emergence of rich and abundant EHR data, powerful natural language processing (NLP) tools,⁸ and data-driven machine learning (ML) methods^9,10 has expanded the quantity, types, and granularity of information available for phenotyping, but better data and methods alone do not eliminate algorithm development challenges. To the contrary, they introduce or exacerbate certain challenges. Which phenotypes that cannot be accurately identified by simpler data and methods are good candidates for algorithm development? What types of data and modeling approaches are minimally necessary to achieve desired performance? Under what conditions is engineering of NLP-derived predictors likely to improve performance? Can algorithms incorporating cutting-edge data mining and machine learning methods be comparably implemented in multiple settings, including those with modest computing infrastructure and expertise?

In our experience, these and myriad other questions arising during algorithm development tend to be resolved parochially with heavy reliance on the habits, intuition, expertise, and tools available to local developers. The informatics literature offers some guidance on selected aspects of algorithm development (eg, cataloging commonly used methods and tools^11–13 and sources of ambiguity¹⁴) but lacks a characterization of key development stages and systematic presentation of strategies, principles, and practical guidelines intended to make algorithm development more successful and efficient.

Our objective is to present a general framework that offers high-level guidance to developers of computable clinical phenotyping algorithms using a wide variety of methods—from simple manually crafted rules to machine learning applied to high-dimensional or unstructured data. This framework is organized around key development stages and articulates broadly applicable principles and practical guidelines focusing on common phenotyping challenges and strategies for addressing them. It is agnostic to phenotype, scientific objective, data sources, and implementation setting.

We derived the content of this framework from 2 sources. First was our collective experience in several multi-site national consortia where computable phenotyping played central roles, including the Electronic Medical Records and Genomics Network,¹⁵ the Strategic Health IT Advanced Research Projects Program,¹⁶,¹⁷ the Health Care Systems Research Network Collaboratory,¹⁸ the Mental Health Research Network,^19,20 the Vaccine Safety Datalink,²¹ and the US Food and Drug Administration’s (FDA) Sentinel Initiative.^22–25 The second and most formative source was a set of projects we conducted to develop algorithms for anaphylaxis,²⁶ acute pancreatitis,²⁷ and COVID-19 disease²⁸ during 2020-2023. This work, supported by the Sentinel Innovation Center,²⁹ was undertaken in part to prospectively identify and articulate principles, strategies, and guidelines that developers of computable algorithms could use to normalize and streamline the development process itself. These are priorities for the Sentinel Active Risk Identification and Analysis (ARIA) system, the backbone of FDA automated medical product safety surveillance.^30,31

We organize algorithm development into 5 stages (Figure 1). The first is assessing fitness-for-purpose, a “pre-mortem”³² exercise, completed before development begins, to assess the feasibility of developing an algorithm that can deliver desired performance. Next is resource-intensive manual chart review to create gold standard data, sometimes required for algorithm training and always required for evaluation. Stage 3 involves the resource-intensive task of engineering features (predictor variables) from various data sources to be used during modeling. Stage 4, model development, applies statistical methods to estimate true functional relationships between predictors and phenotype status. Stage 5 is model evaluation and reporting. All stages address topics of “scalable development”—approaches that enhance efficiency by avoiding high-risk/low-reward efforts, minimizing use of costly/rare expertise, and optimizing methods and algorithms for reusability.

Figure 1.

Flow diagram of 5 stages of computable phenotype algorithm development.

Assessing fitness-for-purpose

The first stage in any phenotyping effort is assessing fitness-for-purpose. Such assessments produce, on a short timeline, judgements regarding the likelihood that the clinical condition in question can be accurately identified by a computable algorithm.

Assessing fitness for purpose is a team effort, drawing on expertise in clinical practice, medicine, statistics, EHR data and systems, NLP, and chart review. It begins with thoroughly understanding the scientific objective of the phenotyping effort, the clinical characteristics and case definition of the phenotype, and the literature on approaches to identifying necessary variables using available data, typically administrative claims and EHR data (including electronic text). For example, Sentinel's scientific objectives for phenotyping include identifying outcomes for safety studies of potential adverse effects of medical products and evaluating disease incidence or prevalence.³³ Clinical outcomes and estimated incidence, study inclusion and exclusion criteria, data setting(s), types of data available, minimum model performance requirements, and a study timeline should be clearly specified. Discussions with those commissioning an algorithm can resolve ambiguities and identify negotiable specifications. Next, a development team with relevant expertise identifies potential barriers to success, a forward-looking type of assessment that has been referred to elsewhere as a project premortem.³² For the next 4 development stages, team members list challenges to project success and strategies for mitigating them. Tools for estimating how much gold standard training data is needed for modeling may be useful.³⁴ Identified challenges may merit a limited, exploratory chart review to confirm or resolve concerns. Throughout assessment, we recommend attentiveness to clinical and data complexity.

Clinical complexity refers to the inherent ambiguity of a clinical condition. Sources include lack of definitive diagnostic tests or consensus about diagnostic criteria, competing diagnoses, and limitations in clinical knowledge and skill, time, and technology (eg, unavailability of advanced medical imaging) (Table 1). Generally, high clinical complexity leads to greater error in diagnosis, limiting the accuracy of gold standard data and complicating feature engineering by necessitating more data capture, from more data types (eg, imaging, labs, clinical text), and using more sophisticated methods (eg, NLP, ML). This increases the time and expertise needed for development.

Table 1.

Sources of clinical complexity and data complexity impacting development of computable phenotype algorithms.

Sources of clinical complexity
1. Lack of definitive diagnostic tests	Definitive diagnostic tests are unavailable for many conditions. Symptoms such as shortness of breath and pain are subjective, reported differently by patients, and present for various health outcomes. In contrast, elevated serum lipase levels are easily and accurately measured making diagnosing acute pancreatitis less clinically complex.1
2. Lack of consensus about diagnostic criteria	Disagreement in the medical community about the defining characteristics of a condition create ambiguity about the definition of the phenotype to be modeled and variability in its documentation (fibromyalgia is an example).35 The need to select or adapt one set of clinical criteria contributes to the complexity of the development effort, and the use of different criteria may impact the accuracy and reproducibility of a computable phenotyping effort.
3. Competing diagnoses	Competing diagnoses exist when multiple clinical conditions have overlapping symptoms. For example, angioedema and anaphylaxis can present similarly. Some events coded as angioedema meet clinical criteria for anaphylaxis.36
4. Limited knowledge, time, or technology	Resource scarcity may introduce time, skill, technology, and/or clinical knowledge constraints that make accurate diagnosis difficult. Examples are non-urgent, non-respiratory chronic health conditions that may be obscured by a focus on acute COVID-19 disease during a pandemic.37–39
Sources of data complexity
5. Data heterogeneity	Heterogeneity of data across settings impacts the availability and meaning of data. Relevant settings include healthcare organizations, time periods, and/or cohorts of patients. Sources of difference include variation in clinical care practice, and data capture priorities and technologies. An example is the use of different diagnosis codes for eye conditions in different healthcare settings.40
6. Data obscurity	Relevant data may be obscured when recorded among voluminous or temporally dispersed observations, or when clinical conditions have signs and symptoms that partly overlap with other conditions (eg, the clinical overlap of heart failure and chronic obstructive pulmonary disease). Obscurity may cause facts to be overlooked or relationships between facts to go unrecognized—even when records are complete (eg, patient histories of suicide attempt mentioned in clinical notes but not documented by standard diagnosis codes*).41–43
7. Data imprecision	Imprecision in clinical data may be caused by inherent subjectivity, measurement error, and imprecise or missing documentation. It results in non-specific or non-sensitive data, rendering its relevance to a particular phenotype ambiguous. Examples include imprecisely reported pain location/severity, use of medication orders instead of fills.
8. Data irregularity	Data irregularity results from the use of locally defined coding schemes or variability across setting and time in the use of standardized coding systems. Shi and colleagues found substantial variation in the use of ICD-10 ophthalmology diagnosis codes across 2 relatively similar healthcare settings.36,44 Valid, alternative documentation of depressive symptoms include diagnosis codes, depression scale scores, clinical notes.
9. Data instability	Data instability results from evolution in clinical terminology and/or coding schemes, and shifts in care delivery practice. Examples include the 2015 switch from ICD-9 to ICD-10 coding, and the rise of virtual care (accelerated by the COVID-19 pandemic).
10. High dimensionality	High dimensionality occurs when information is represented by discrete data elements that are large in number relative to sample size. It makes feature engineering and modeling more complicated, and measurement error more likely. An example is using 843 NLP-derived measures to model anaphylaxis in a cohort of 239 patients.26
11. Lack of structure	Unstructured data, such as narrative text or images, are inherently complex and require special processing—by humans or software—increasing opportunities for error. Examples include the NLP challenges of distinguishing patient clinical history from family history in clinical notes.

Abbreviations: NLP = natural language processing.

Anaphylaxis is illustrative. Diverse combinations of symptoms satisfy diagnostic criteria,⁷ complicating feature engineering and modeling. Diagnostic ambiguity is exacerbated by sporadic EHR documentation of key information (eg, symptom timing) and/or exposure to known allergens (historical and current) and by frequent preemptive epinephrine treatment which can suppress key symptoms. Clinical complexity can lead to difficulty in assigning accurate gold standard labels, for example, when independent expert reviewers reach discordant determinations. A process for resolving such discordances to produce definitive case status labels cannot eliminate the underlying clinical ambiguity of the diagnosis. This may impose upper bounds to algorithm performance. In our anaphylaxis study, 20% of potential events were assigned discordant labels or judged “difficult to determine” by independent physician adjudicators.²⁶

Data complexity—which may be overlapping with or independent of clinical complexity—refers to ambiguity, lack of structure, missingness, or error in documentation of clinical conditions. It may hinder each development stage and reduce model performance. Sources include data heterogeneity, obscurity, imprecision, irregularity, instability, high dimensionality, and lack of structure (Table 1).

Anaphylaxis has high data complexity, involving documentation of numerous signs, symptoms, and symptom severity for multiple organ systems. Our NLP dictionary for anaphylaxis contained >450 entries for a single symptom category, “skin/mucosal involvement.”²⁶ Symptom timing is also a key clinical criterion but is often sparsely documented or difficult to extract using NLP. In contrast, acute pancreatitis has low data complexity. Although diagnostic criteria include subjective and imprecisely documented pain, simple diagnosis codes and serum lipase labs identify acute pancreatitis with 92% positive predictive value (PPV) (95% CI 86%-95%) and 85% sensitivity (95% CI 79%-89%) (Figure 2).¹

Figure 2.

Relationship between clinical complexity, data complexity, and increasing phenotyping difficulty with illustrative phenotypes.

As clinical and data complexity increase, the anticipated difficulty of phenotyping rises (Figure 2). Anaphylaxis, having high clinical and data complexity, required extensive model development effort and yielded modest model accuracy.²⁶ Acute pancreatitis has high clinical complexity but low data complexity, as noted. Opioid overdose has low clinical complexity, especially in emergency departments (EDs) where overdose symptoms are obvious and laboratory testing confirms opioid exposure, but incomplete clinical documentation often creates high data complexity.⁴⁵ Diabetes is easily diagnosed and typically well-documented.⁴⁶

Fitness-for-purpose assessments conclude with decisions about whether development should proceed. A clear “go” decision allows phenotyping to proceed with a better understanding of the tasks. A clear “no-go,” though disappointing, may avoid expensive, failed development. If a clear decision is elusive, the team should consider whether the algorithm’s scientific objectives may be altered to mitigate feasibility challenges. If the lack of a clear clinical case definition is the problem, the project might be delayed until a sufficiently clear definition is identified or developed. If the problem results from data complexity, options include substituting a comparable phenotype with lower data complexity, targeting a phenotype surrogate (eg, severe allergic reaction instead of anaphylaxis), or restricting outcomes to care settings (outpatient, inpatient, or ED) with more complete phenotype data.

Creating gold standard data

Gold standard data indicating whether a phenotype is present is created by applying vetted clinical events criteria^47,48 during manual review of medical records. These data may be used in feature engineering or model training and are essential for evaluating all algorithms—regardless of their complexity. Medical record review may also yield insights regarding a phenotype’s clinical and data complexities, sources of potential predictors (eg, labs, imaging), and documentation practices.

The quantity of gold standard data needed depends on a phenotype’s clinical complexity, case/non-case proportions (≥10 minority-class observations per predictor is recommended),⁴⁹ and whether any data will be used to inform feature engineering (precluding reuse for modeling).

Inherent challenges in creating gold standard data make it time and resource intensive (and, along with feature engineering, a bottleneck in computable phenotyping). Treating clinicians may struggle to diagnose clinically complex conditions in real time; retrospectively creating gold standard determinations for complex conditions based on EHR data that may lack clinical information or contain errors can be even harder. EHR information can also be impacted by local care and documentation practices and EHR interfaces, sometimes complicating data interpretations. While a gold standard review process may yield unambiguous case/non-case labels, it cannot eliminate underlying documentation inadequacies or clinical ambiguity.

To create high-quality gold standard data typically required for model training and/or evaluation, we recommend structuring review tasks according to established clinical criteria, performing periodic quality assurance evaluations, and using full-featured data collection systems (Table 2). Monitoring for extreme case/non-case imbalance and adapting sampling as warranted can enhance the utility of the data generated. For multi-site studies where physician adjudicators may not be available at some sites, or when the number of cases to be validated is large, there may be advantages to conducting reviews by non-clinician abstractors trained and supervised by clinicians.

Table 2.

Guiding principles for efficiently creating high-quality gold standard data via manual chart review with illustrative examples.

Principle	Description/illustration
1. Structure reviews around established clinical criteria	Structure gold standard chart review tasks and written abstraction guidelines around established, widely accepted clinical criteria (eg, from the Brighton Collaboration).48 If suitable criteria are unavailable, modify existing or develop new criteria to facilitate consistent determinations of phenotype status. The Atlanta criteria for diagnosing acute pancreatitis,50 for example, require any 2 of (1) abdominal pain consistent with acute pancreatitis, (2) serum lipase at least 3 times greater than the upper limit of normal, or (3) characteristic imaging findings.
2. Provide comprehensive training to chart reviewers	Training develops a shared understanding among all chart reviewers of the EHR interface, terminology, process, and clinical events criteria. It requires considerable time and effort and requires reviewing actual charts. Reviewer training for a study of acute pancreatitis included independent review and group discussion of 16 actual charts with clinician guidance.
3. Conduct periodic quality assurance evaluations	Periodic inter-reviewer reliability (IRR) analyses51 based on dual, blind reviews of random 5%-10% samples can detect “drift” in outcome determinations and foster communication among reviewers about potential threats to consistency. Alternatively, random samples of reviewer determinations may be verified by clinician reviews, as was done in an acute pancreatitis study.27
4. Use a full-featured data collection system	Data collection systems such as Research Electronic Data Capture (REDCap)52,53 can streamline and secure the review process by embedding review guidelines in data capture forms, pre-populating data fields, managing IRR reviews, and recording determinations and supporting evidence (eg, a REDCap acute pancreatitis form).54
5. Strive for balance in cases and non-cases	Equal proportions of validated cases and non-cases are optimal for model training.55 Designing a sampling scheme that mitigates class imbalance can help avoid challenges due to small minority class size. Training data in our anaphylaxis algorithm study were severely imbalanced, having 153 true cases and 83 true non-cases.26
6. Train local, non-clinician reviewers to conduct chart reviews	An acute pancreatitis study trained local, non-clinician reviewers to conduct 300 chart reviews, reducing the need for relatively scarce, more costly clinician reviewers.56 According to the principles of distributed cognition,57–59 knowledge sharing and discussion during training may enhance chart review quality by (1) formalizing a clear, reproducible decision-making process, (2) identifying where quality assurance is needed and how to monitor it, and (3) revealing local knowledge of the EHR, healthcare operations, and local practices that impact clinical data and its interpretation. Blind, duplicate review by a physician in the acute pancreatitis study achieved 100% agreement with trained reviewers in a 15% random sample (31/205) and 100% agreement in 10 randomly selected determinations by other physicians.60

Abbreviation: EHR = electronic health record.

Feature engineering

Features (also predictor variables or covariates) are structured data representations of information used as inputs in all phenotype algorithms. Features may be based on structured data, such as medical claims codes, or unstructured data, such as indicators derived from chart notes via NLP. Feature engineering uses clinical and informatics domain knowledge (subject-area expertise) to construct operational definitions of features from these “raw” sources. Feature engineering is an art,⁶⁰ though automation has shown early success.⁶¹ Model development may require dozens to hundreds of features.

Feature engineering is an expertise-intensive, time-consuming task⁶² and a major bottleneck in algorithm development. The necessary expertise is expensive and sometimes scarce. For efficiency, initial feature engineering and model development may occur in one institutional setting, extending to others if successful. Although predicting which features will most improve performance is difficult, excessive effort creating or honing features may be wasted effort.

When algorithms will be implemented in multiple settings, incorporating locally idiosyncratic data can also be wasteful because performance gains may not generalize. Engineering for reusability requires favoring simplicity over complexity and using data available across settings.

Table 3 summarizes principles to enhance efficiency, transportability (ease of implementation across settings), and generalizability of feature engineering. Including standard features, engineering diverse sets of features, and leveraging domain expertise (principles 1-3) are common sense but important. Limiting the number of features engineered is neither beneficial nor necessary; determining which features help predict a phenotype occurs during modeling. Engineering generalizable features requires knowledge of clinical diagnostic criteria, informatics, and how EHR systems influence data capture. Useful domain knowledge may also be derived from published clinical knowledge sources^7,50,66,67 or scientific reports of phenotyping efforts (eg, modeling anaphylaxis^6,68).

Table 3.

Guiding principles of feature engineering with illustrative examples.

Principle	Description/illustration
1. Include standard features	Standard features capture information about demographics, how cohort inclusion criteria were met, presentation setting, and time period (eg, the ICD-10 era or COVID-19 era) and may impact model performance. For example, diagnosis codes for anaphylaxis had a PPV of 78% for patients aged <20 years vs 57% for patients aged ≥60 years.
2. Engineer diverse features to capture diverse signals	Diverse features represent information about the same thing in different ways or from different sources (eg, NLP-derived concepts may be engineered as raw or normalized counts, the latter adjusted for chart length). Diverse features may improve model performance by capturing complementary signals (ie, alternative forms of evidence that a phenotype is present).
3. Feature engineering is enhanced by domain knowledge	Feature engineering relies on clinical domain, informatics, and statistical knowledge to identify information that is relevant, extractable, and represented in a way useful for modeling. We used expertise from all 3 domains to define a key predictor of acute pancreatitis: a patient’s maximum lab value for serum lipase normalized by 3x the upper limit of normal within ±14 days of a diagnosis for acute pancreatitis.
4. Consider automated feature engineering approaches	Automated feature engineering uses formulaic, data-driven methods to define features for specific clinical phenotypes. One such approach mines relevant medical concepts from published clinical knowledge articles on the phenotype, then defines one feature per concept as the count of its mentions in a patient’s chart.63 We used this approach to create and use 158 features in a model identifying symptomatic COVID-19 patients.28
5. Use expert-guided manual feature engineering sparingly	Expert-guided manual feature engineering is potentially very costly and may embody idiosyncrasies stemming from local data and/or expert knowledge. It is therefore best used to identify promising feature categories rather than fashion complex operational definitions. Of 45 structured and 468 NLP features we manually engineered for anaphylaxis, our best models retained only 16 structured and 32 NLP features. We are now investigating whether lower cost automated approaches may perform comparably.
6. Design for transportability across settings	Transportability refers to ease of implementation across settings. Feature definitions that are more general (vs specific) are more transportable (eg, NLP of all imaging reports in a time window vs reports with setting-specific titles). Engineering blind to gold standard labels avoids “baking in” feature associations that vary across settings.64
7. Standardize required tailoring	Tailoring refers to changes in algorithm definitions or logic required to accommodate idiosyncratic data. Tailoring is standardized when software code used to engineer features anticipates and accommodates specification of setting-specific lists or selection of options (eg, custom term lists for NLP features, or software options to include/exclude patient after visit summaries as clinical text for NLP processing).
8. Share code and tools publicly	Use of publicly available software-sharing services such as GitHub vastly improves transportability by simplifying sharing of software code/tools and version management. For example, code used in our anaphylaxis algorithm development projects is available on GitHub.65

Abbreviation: PPV = positive predictive value.

Automated engineering (principle 4) may produce features that perform comparably to manually engineered features with dramatically less effort, expertise, and operator-dependent idiosyncrasies. Both structured⁶⁹ and unstructured data^9,61,63,70 have been successfully used to automate feature engineering to identify chronic conditions and an acute health condition.⁷¹ The low cost of automated features makes them an attractive phenotyping starting point.

Expert-guided manual feature engineering should be used sparingly, focusing on strategic guidance rather than fine tuning (principle 5). Engineering effort that more precisely represents the true facts about a patient but does not improve model performance is wasted. Avoiding overengineering can reduce costs and threats to model generalizability.

Principles 6-8 simplify implementing features across settings (transportability). These include designing features to accommodate heterogeneous data inputs, anticipating and accommodating unavoidable local tailoring, and making software tools publicly available.^72–74

Model development

Constructing a useful prediction model depends on (1) the underlying strength of relationships between available features (predictor variables) X and outcome Y; (2) complexity of the true underlying functional form of these relationships; and (3) how closely algorithms under consideration can approximate the true prediction function. We cannot know in advance which ML or parametric modeling approach will most accurately learn the prediction function from the set of features, so we recommend V-fold cross-validation to simultaneously evaluate many algorithms. Modeling choices should be tailored to the prediction task and data characteristics.⁷⁵ These guiding principles can improve the scalability of developing computable phenotype models.

ML is enhanced by incorporating domain knowledge. Domain expertise can focus manual engineering of features known to distinguish cases from non-cases (eg, serum lipase measures for acute pancreatitis, Table 4), caution against using information that is poorly captured or non-specific to the outcome and determine whether heterogeneous feature-outcome associations across sub-populations pose challenges (eg, anaphylaxis in adults vs children). Automated feature engineering also exploits domain knowledge to identify candidate predictors; Yu and colleagues engineered features for clinical concepts mined from clinical knowledge base articles for several clinical conditions.⁶³

Table 4.

Guiding principles of model development and illustrative examples.

Principle	Description/illustration
1. Incorporate domain knowledge in modeling	ML is enhanced by incorporating domain knowledge to define the target population and in the selection of candidate features. When clinical presentations are known to vary by demographic subgroups, eg, anaphylaxis in adults vs children, at small sample size, stratified modeling may improve performance; cross-validated (cv) sensitivity in our adults-only anaphylaxis models improved to 70% from 65% in our all-age-group models (at cv-PPV of 79% in both models).26 Elevated serum lipase labs ≥3 times the upper limit of normal are known to be clinically significant in diagnosing acute pancreatitis1; if available, such labs should be included as candidate predictors.
2. Pre-process the data to reduce dimensionality.	Outcome blind pre-processing of the set of candidate features to remove those that are redundant or highly correlated can reduce the complexity of a prediction task without sacrificing predictive ability. In developing acute pancreatitis models, we created a binary variable indicating if “necrosis” was ever mentioned in any of a patient's imaging reports, and a count variable for the total number of mentions in all reports; but because most patients had only one mention of necrosis, the empirical correlation between these variables was 0.998, and we randomly excluded one variable from the dataset.
3. Consider many algorithms and dimension reduction strategies	Because the true functional form of the relationship between candidate features and the outcome is unknown, considering a diverse set of candidate prediction algorithms is advantageous. A main terms logistic regression model for anaphylaxis with 136 covariates and 236 observations (153 cases, 83 non-cases), achieved a modest cv-AUC of 0.49 because it had overfit the data. Coupling the same logistic model with least absolute shrinkage and selection operator (LASSO) based dimension reduction retained only 48 covariates, increasing the model's cv-AUC to 0.64.
4. Use V-fold cross-validation to evaluate performance	The cross-validation scheme should accurately estimate performance of a model trained on all available data. With a modest size sample of 236 observations with gold standard labels (153 cases, 83 non-cases) we used V-fold cross-validation (V = 20) to develop an anaphylaxis model. We randomly assigned observations to 20 disjoint validation sets stratifying on gold standard outcome, each set having 7 or 8 cases and 4 or 5 non-cases. Thus, each training set contained 95% of all observations with a similar ratio of cases to non-cases, and performance metrics were averaged over all validation sets, ie, all observations in the data.
5. Specify performance metrics relevant to the use case	Cross validation of performance metric(s) relevant for downstream use is vital. Models we developed for anaphylaxis, acute pancreatitis, and symptomatic COVID-19 disease were intended for use in identifying outcomes for FDA safety studies. Beyond evaluating AUC, the ability to discriminate between cases and non-cases, FDA studies often benefit from high PPV (0.80 or above) while maximizing sensitivity; our performance metrics thus included cv-sensitivity and cv-PPV at multiple candidate classification thresholds. In contrast, for a screening task, high sensitivity is the most relevant metric. If the training sample is not a simple random sample from the population, use appropriate weighting in the evaluation.
6. Final algorithm choice is guided by downstream considerations	Although cross-validated model performance is the primary consideration in algorithm selection, replicability and generalizability should be considered. Consider hypothetical models A, based on easily obtained structured data covariates with cv-AUC of 0.89 (95% CI: 0.87, 0.91), and B, based on the same structured covariates plus NLP-derived covariates with cv-AUC of 0.90 (95%CI: 0.88, 0.92). The small, estimated performance gain of B may not be worth the time and effort of using NLP. However, if model B offers better classification accuracy in a region of the AUC curve relevant to the use case, B may be worth the additional effort. Models based on fewer and more easily operationalized and interpreted covariates may be favored due to their ease of implementation and transparency.
7. Train selected algorithm on the entire dataset	In developing a model to identify patients with acute pancreatitis, a LASSO dimension reduction approach coupled with an elastic net algorithm achieved the best tradeoff between cv-PPV (0.90) and cv-sensitivity (0.92) by setting the classification threshold at the 37% quantile. This threshold corresponded to a predicted probability of 0.39 based on a model trained on all observations.

Abbreviation: PPV = positive predictive value.

Outcome-blind pre-processing of the set of candidate features to remove those that are redundant or highly correlated can reduce the complexity of a prediction task without sacrificing predictive ability. Recommended procedures include omitting features with little to no variation in values (eg, 98% identical), retaining only one of several highly correlated features (eg, duplicate values, absolute correlation > 0.98). Features unlikely to be available in other settings should be avoided for model generalizability.

Because the true functional form of relationships between candidate features and the outcome is unknown, considering a diverse set of candidate prediction algorithms is advantageous. These can include multiple variations of parametric models, tree-based algorithms, and neural networks. If features are continuous, consider multiple additive regression splines, or generalized additive models, and tree-based algorithms that avoid imposing monotonic linear relationships between predictor and outcome. An ensemble Super Learner that combines predictions from multiple algorithms can consider up to a polynomial number of algorithms simultaneously, much larger than typically investigated in practice,^76,77 (and “multiple testing” is not a concern because P-values are not evaluated). With high-dimensional feature data, coupling candidate algorithms with dimension reduction screening strategies that consider feature-outcome associations may be advantageous. Each algorithm-screener combination defines a custom ML approach. Cross validation of performance metric(s) relevant for downstream use is vital. For phenotyping, relevant loss functions are 1 − AUC (area under the receiver operating curve) and negative log likelihood (NLL). 1 − AUC captures ability to discriminate between cases and non-cases but ignores accuracy of the predicted probabilities. A similar metric 1 − AUC-PR (area under the precision-recall curve) is sometimes recommended when the minority class is rare.⁷⁸ NLL favors well-calibrated algorithms with high predictive accuracy, but potentially less accurate classification. We recommend V-fold cross validation over other types because an appropriate V-fold cross validation scheme makes most efficient use of the data.^76,77 V-fold cross-validation begins with randomly assigning observations to V disjoint validation sets. Each validation set has a corresponding independent training set containing the remaining observations. Algorithms are fitted/trained on training set data, and loss evaluated on observations in the independent validation set. Cross-validated risk is average loss over all V validation sets. Stratifying assignment to validation folds on the outcome and choosing a large V (eg, 20) ensures that the training data closely approximates the entire dataset, particularly with sparse data or rare outcomes.

Although cross-validated model performance is the primary consideration in algorithm selection, transportability and generalizability (comparable performance across settings) should be considered, particularly when using data from multiple healthcare settings (Table 4, Figure 3).

Figure 3.

Selecting a final model from all models developed based on considerations of model performance, model transportability, and model generalizability.

The production model is produced by training the selected algorithm on all observations. The classification threshold is chosen to optimize cross-validated performance with respect to PPV, sensitivity, or a summary F-measure, further discussed in the next section. The probability threshold is set to the probability at the optimal quantile of the predicted values.

Model evaluation and reporting

Model evaluation aims to estimate the performance of a prediction model applied to novel data from the same setting or a new setting. Three key questions are as follows: What performance metrics are relevant to the potential downstream use case(s)? Where and when will the predictions be applied, considering generalizability to new candidate locations and specific time frames (past, present)? How will the model be used in epidemiologic studies and trade-offs in performance be tailored to application goals?

Criteria for what to measure in evaluating a clinical prediction model are well established and focus on measures such as sensitivity (proportion of true cases identified, or recall) and PPV (proportion of observations classified as cases that are true cases, or precision).⁷⁹ These rates depend upon cut-offs or thresholds that dichotomize model prediction into “positive” and “negative” outcomes and allow evaluation at multiple candidate thresholds. Specifically, a binary phenotype generated this way is defined as $Y^{*}\left(c\right)$, an indicator of whether model prediction, $M$, is larger than cut-off value, $c$, or $[M>c]$. Performance measures are then defined in terms of true outcomes, $Y$, and dichotomous predictions, $Y^{*}\left(c\right)$, such as sensitivity, $P[Y^{*}\left(c\right)=1 |Y=1]$, and PPV, $P [Y=1 |Y^{*}\left(c\right)=1]$. Graphical summaries can plot error metrics as a function of threshold, $c,$ across a range of candidate thresholds. In the ML literature, a precision-recall curve plots PPV (precision) versus sensitivity (recall) for all possible thresholds.⁷⁸ The F-score summarizes sensitivity and PPV into a single summary, F1, which is the harmonic mean, and is sometimes recommended as a metric to optimize in model building or evaluation.⁸⁰ When the model defines a safety outcome, both sensitivity and PPV are key to understanding the implications of measurement errors associated with using a computable phenotype compared to a gold-standard outcome. When sensitivity is more (or less) important than PPV, a weighted F-score can be evaluated. For example, a screening test might evaluate F2, weighting sensitivity twice as heavily as PPV. Cohort inclusion criteria might evaluate F0.5, weighting PPV twice as heavily as sensitivity.

Considering where model predictions will be used directly relates to generalizability and whether performance may vary by context.⁸¹ Clearly defining settings in which predictions are applied aids in interpreting validation assessments. A setting similar to the development data source evaluates reproducibility (internal validity). A setting with different context elements such as time periods for data capture, geographic or healthcare settings, or population characteristics simultaneously evaluates reproducibility, transportability, and generalizability. Honest performance assessment requires evaluation using data independent of the training data.

For downstream epidemiologic analyses using a prediction model, many applications use a simple binary indicator, $Y^{*}\left(c\right)$. The primary drivers for using a derived binary outcome include simplicity of presentation and interpretation, and potential relaxing of the need for local calibration of predictions. However, binary indicators lead to measurement error in computable phenotypes that can impact both the bias and precision of estimated association measures between the phenotype and exposures.⁸² Understanding the implications of measurement error is key to choosing a threshold to define the phenotype. For example, higher PPV leads to lower bias in relative risk estimates and higher sensitivity leads to lower bias in absolute risk difference estimates.⁸³ Furthermore, if discrimination is poor, higher thresholds for $Y^{*}\left(c\right)$ tend to decrease the overall prevalence of computable phenotypes and inflate the variance of association estimates due to lower in-sample information. Therefore, the implication for bias and standard errors should be carefully considered, evaluating potential trade-offs and linking them to downstream analytical goals. For example, narrowly focusing on high PPV may impact statistical uncertainty of estimation and adversely impact power to detect non-null associations. With large samples for which bias dominates variance, high PPV is paramount. However, smaller samples require more nuanced bias-variance trade-off. Analytical expressions for bias and variance of common regression estimators can be useful guides,⁸³ and simulations can summarize bias, variance, and power for alternative choices of $Y^{*}\left(c\right)$. Bias may be reduced by using the predicted probability as the outcome when estimating effects, rather than binary indicators. Calibration is a key contributor to success.

Discussion

This general framework offers guidance to teams with varying levels of expertise and resources in developing computable phenotype algorithms of many types. It is particularly relevant for phenotypes with high clinical and data complexity, and where multi-site implementation is needed. It is intended to encourage efficiency in all aspects of development and to enhance the performance and transportability of algorithms applied in multi-site settings.

Several points merit emphasis. First, fitness-for-purpose assessment before development work begins is critical. Halting when failure is likely is essential for achieving overall scalability and optimizing the impact of precious, sometimes scarce expertise.

Two major bottlenecks in development are creating gold standard data and feature engineering. Careful attention to the recommendations for these stages will enhance the overall efficiency of phenotype development efforts.

Computable phenotypes developed in one setting for reuse in other settings should be guided by awareness of potential data heterogeneity, computing resources, and availability of relevant expertise across settings. Designing phenotype algorithms based on “lowest common denominators” across settings reduces risks of uneven cross-site performance. When setting-specific feature engineering or algorithm tailoring is unavoidable, code and software should be designed to accommodate them. “One-off” solutions that boost performance in one setting may not be transportable to or perform comparably in other settings.

Automated approaches to feature engineering or developing entire phenotyping models should be considered because they require relatively modest effort, may fully achieve performance requirements or, if not, provide a useful baseline for manual curation.

Aspects of methods development that should be prioritized for future research include:

1. Disseminating findings of and lessons learned by applying fitness-for-purpose assessments to many different phenotypes.

2. Assessing feasibility and efficiency gains of machine-assisted chart review interfaces for creating gold standard data.

3. Defining a systematic approach to incorporating automated feature engineering and silver-label model training,²⁸ coupled with sampling strategies to minimize the number of chart reviews required for evaluation.

4. Defining conditions under which reusing high-performing models without additional gold-standard evaluation is reasonable.

Several limitations merit attention. First, general frameworks inevitably contain flaws; we hope this attempt will elicit useful discussion and future elaboration. Second, the framework we present is limited by our own experiences and knowledge. Third, our illustrative models were appropriate for our data and phenotype relationships; other phenotypes and use cases will involve different data and relationships.

Conclusion

This general framework is intended to provide an overall structure and guidance useful to those considering undertaking development of computable clinical phenotype algorithms. Once commitment to development occurs, this framework provides an integrated set of principles, strategies, and practical guidelines intended to enhance the overall efficiency and transparency of the effort as well as the performance, transportability, and reusability of the algorithms developed.

Author contributions

All authors contributed to the conceptualization, organization, and presentation of the principles and guidelines presented in this paper. David S. Carrell, James S. Floyd, Susan Gruber, Patrick J. Heagerty, Brian L. Hazlehurst, Jennifer L. Nelson, and Brian D. Williamson drafted original content for various sections of the paper. All authors critically reviewed and edited the combined manuscript and contributed intellectual value to the article.

Funding

This work was supported by the U.S. Food and Drug Administration via task order number 75F40119F19002 under master agreement number 75F40119D10037. The FDA approved the study protocol and reviewed and approved this manuscript. Coauthors from the FDA participated in the preparation of and decision to submit the manuscript for publication. The FDA had no role in data collection, management, or analysis.

Conflicts of interest

R.B. is an author on US Patent 9,075,796, “Text mining for large medical text datasets and corresponding medical text classification using informative feature selection.” At present, this patent is not licensed and does not generate royalties. All other authors have no competing interests to declare.

Data availability

This work did not entail collection, management, or analysis of any original or secondary data.