Making the Improbable Possible: Generalizing Models Designed for a Syndrome-Based, Heterogeneous Patient Landscape

INTRODUCTION

Although machine learning (ML)-based clinical decision support has seen some successful implementations in radiology^1,2 and ophthalmology,^3,4 its overall presence in health care is modest when compared with its potential. This underutilization remains particularly true in the intensive care unit (ICU). A major hurdle to the widespread deployment of ML models has been their inconsistent performance as a result of several factors, including hospital-dependent operating procedures, patient demographics, and missing data.^5–7 These factors all contribute to data heterogeneity, so an ML prediction model (herein termed “ML model” or “model”) trained on one dataset may exhibit degradation in performance when deployed on another,⁸ resulting in inadequate generalizability. However, as noted by Futoma and colleagues, the colloquial use of the term generalizability in clinical ML literature is broad and not well-defined.⁹ For clinical applications of ML models, a published hierarchy¹⁰ describes it as a set of rules that may apply to internal, temporal, and external applications relative to the original training dataset. An internal application refers to using an ML model to the same patient cohort on which it was trained (eg, the same dataset at the same hospital). A temporal application denotes using this model at the same location but across a different time period. An external application refers to using this model at a separate location during any time period. The ideal model will be able to demonstrate similar levels of performance under any application, notably external ones.^11–14 This makes sense—it is essential to verify that a model for clinical use can provide similar results for any group of patients, especially those it was not explicitly trained on.⁹ Yet due to the current nature of ML and statistics, external generalizability approaches a limit as the applicable population grows because of increasing variation in workflow practices, point-of-care measurement devices, and population characteristics. As such, the current literature suggests that it is infeasible to develop a single universal prediction model. Therefore, our focus should shift toward more clearly defining the “conditions for use” of a model while improving its external generalizability within the intended use population. These “conditions for use” are ideally broad enough so that a prediction model can be impactful for as many patients as possible while minding the aforementioned limitations.

Patient-focused clinical predictive modeling can be classified as diagnostic versus prognostic.¹⁵ Diagnostic tasks rely on a patient’s “true state,” which are typically defined by proxy criteria and construct via laboratory values, imaging, and physical symptoms, among others, to predict a clinical development (eg, sepsis). Prognostic tasks rely on the ordered tests and their results to quantify the probability of certain patient-centered outcomes (such as in-hospital mortality).¹⁶ Both tasks have clinical utility, albeit at varying levels. Early diagnostic tasks may be able to assist clinicians administer timely interventions for a developing condition (eg, early and appropriate antibiotics for sepsis). Within the same context, prognostic tasks may help administrators optimize resource allocation for patients who are most likely to decompensate or are at risk for other adverse outcomes, including mortality.^15,17–19 In either case, existing studies investigate such tasks in critical care environments because the nature of its routine patient monitoring makes for data-rich electronic health records (EHRs) from which models can learn and make predictions.

In this review article, the authors investigate the current challenges of integrating ML models into critical care, using studies centered on early sepsis prediction as examples. The authors (1) explore clinical challenges with syndrome-based conditions, which are commonly diagnosed in the ICU; (2) clarify data science terminology surrounding these studies; (3) examine major barriers to generalizability; and (4) illustrate how current-day ML models address such obstacles via different methods of learning. The authors conclude with a discussion on areas for future research.

CHALLENGES WITH SYNDROMES

A syndrome can be defined as a recognizable complex of findings and symptoms that indicate a specific condition with a poorly understood cause.²⁰ A disease refers to a condition in which a causative agent or process results in a readily identifiable clinical and biological manifestation. Yet, with increased research and study, a condition that was formerly best described as a syndrome can be referred to as a disease. Indeed, Kawasaki disease was initially known as mucocutaneous lymph node syndrome because the underlying pathophysiology was uncertain and clinical manifestations were varied.²¹ An increased understanding of the disease process later described clearly identifiable diagnostic features and treatment responses.

Although this distinction between a disease and syndrome is easily understood by physicians, syndromic conditions can prove challenging for ML models to identify and predict. This is particularly true in the ICU because a multitude of prevalent syndromes (ie, sepsis and the acute respiratory distress syndrome) share similar physiologic and biological derangements. For example, a patient with decompensated heart failure can show vital signs and laboratory findings that mimic septic shock. In addition, critically ill patients are frequently comorbid for these syndromes. Clinicians use contextual clues, historical components, and physical examination findings to help differentiate between possibilities, but contemporary ML models struggle because historical context and examination findings are oftentimes not readily available¹⁴ for such rapidly evolving patients. Complicating this further are multiple definitions for any given syndrome in sepsis, the condition can either be defined by Sepsis-3,²² Severe Sepsis and Septic Shock Management Bundle (SEP-1),²³ or the Center for Disease Control and Prevention (CDC)^24,25 criteria.

CLARIFYING TERMS

Before we begin our review on generalizability and related ML studies, it is necessary to clarify terms and concepts that are commonly used in data science and relevant to its application in critically ill patients. Three recently published ML models, Artificial Intelligence Sepsis Expert (AISE),^11,12 Weight Uncertainty Propagation and Episodic Representation Replay (WUPERR),¹³ and COnformal Multidimensional Prediction Of SEpsis Risk (COMPOSER),¹⁴ and their respective studies are used as several examples throughout this article. Table 1 summarizes definitions and examples of relevant vernacular.

Table 1.

Definitions and examples of common terms used in data science

Term	Definition	Example
Development cohort	The dataset on which an ML model was trained	AISE11,12 was trained on data from the Emory Healthcare System, also known as the development cohort.
Validation cohort	The dataset on which an ML model must perform a task for the first time	AISE11,12 was validated on data from the MIMIC-III cohort collected from the Beth Israel Deaconess Medical Center in Boston, also known as the validation cohort.
Generalizability	The extent to which an ML model can achieve similar performance on external validation tasks vs internal validation tasks	AISE11,12 WUPERR,13 and COMPOSER14 demonstrated comparable performance at their respective validation hospitals, on par with their performance at their respective development hospitals.
Dimension	A characteristic of data and/or a dataset, relating to the number of variables included	EHR data are multidimensional because it includes heart rate, blood pressure, temperature, laboratory values, and so forth.
Missingness	The presence of missing clinical variables and how this is handled/reported	During Patient A’s 10-d ICU stay, he/she may not have regular blood pressure measurements recorded every 4 h due to movements for scans, tests, and so forth.
Omission	A method of handling missingness where patients and/or characteristics with missing data are deleted from the dataset	As Patient A is missing blood pressure data, he/she is excluded from analysis of the dataset, which uses blood pressure.
Imputation	A method of handling missingness that determines a single or multiple value(s) to replace all missing values for a specific variable	Patient A’s missing blood pressure measurements are estimated by averaging blood pressure values of other patients in the ICU taken at the same time (single imputation).

They are summarized here to provide clarity to the following sections.

Missingness describes the presence of missing data and how it is handled, reported. This can negatively impact the predictive accuracy of a model and decrease its clinical utility.⁸ However, not all missing data are created equally. There exist three primary classifications of missing data²⁶: Missing Completely At Random (MCAR), Missing At Random (MAR), and Missing Not At Random (MNAR). MCAR refers to randomly missing data that does not have any distinguishable pattern to it.^27–29 MAR refers to randomly missing data that may be associated with an underlying pattern.^27,30,31 For instance, in a hypothetical case where missing Glasgow Coma Scores (GCS) among trauma patients were more likely to be observed for older patients,³² the mechanism is MAR. MNAR refers to the likelihood of a missing value to seem as a function of the value itself²⁷—following the same hypothetical, the mechanism is MNAR if a missing GCS is known to be associated with mild brain trauma.³² For prediction models, the MNAR mechanism is typically assumed because they use longitudinal data collection, where patterns of missingness are more likely to present (eg, healthy patients routinely have fewer tests as they are not indicated). The differences in how data go missing are key to understanding their handling,^27,31 which are more deeply explored in the section Approaches to Data Missingness.

At its core, ML consists of a set of parameters that are initialized with random weights. Training then takes place, where the model is exposed to a development cohort (eg, retrospective clinical data at Hospital A). This induces key changes in the weights of different rules. The idea is to give more bearing on the final output to rules that are mathematically determined to be “more important.” Prediction error (or the difference between the predicted outcome and the true patient outcome) will tell the model when it produces a correct or incorrect output, prompting changes to these rule weights, a process known as supervised learning.³³ As it is exposed to a validation cohort (eg, retrospective clinical data at Hospital B), performance usually degrades because differences in the underlying data often necessitates different weights being given to different rules; this is the basis of the barriers to generalizability.⁵

ONE SIZE DOES NOT FIT ALL: THE GENERALIZABILITY PROBLEM

Generalizability was previously described as the ability for an ML model to perform similarly well on both development and validation cohorts within a well-defined intended use population. It would ideally maintain this high level of performance as it is applied to additional institutions that fit its “conditions for use.” However, in order to do so, we must acknowledge the challenges it faces at different locales. These include explicit differences between institutions and missing clinical data. The following sections detail these problems in addition to potential methods that ML models might use to overcome them.

Heterogeneity in Health Care

Despite all its regulations, health care remains an area of great heterogeneity at various levels. Shashikumar and colleagues describe these levels in a recent Behind the Paper⁵ on Nature Portfolio Health Community, starting with EHRs: different EHR vendors currently encode information in non-standardized formats which reduce their interoperability. The data are recorded using clinical instruments from different vendors which often use proprietary data processing methods with varying necessity for clinician verification. Local guidelines vary in their frequency for specific clinical measurements to be taken, but this should not be confused with the predictive value that missing data itself can offer (see Approaches to Data Missingness). For diagnostic records, differences in clinical inclusion and exclusion criteria between institutions can introduce label noise and give way to label bias.³⁴ Shifting criteria also lay the foundation for the difficulties in managing syndrome-based conditions, which were discussed in Challenges with syndromes. Temporal changes in data might occur as care and monitoring processes transform, including the disruption of existing clinical workflows resulting from implementing ML models. Taken together, these “systemic factors” all increase the heterogeneity of a clinical dataset which can confound predictive accuracy of a model not trained to recognize and correct for them (see Machine learning-based solutions, for an introduction to such methods). Regular updates to models are necessary as institutions evolve in these respects.

Differences in patient demographics between institutions can further add to the previously described data heterogeneity. However, this aspect is more difficult to handle due to the simultaneous potential for demographics to confound and improve impact predictive accuracy. Consider the following example, recent facial recognition³⁵ and hiring and recruitment³⁶ models were found to unintentionally perpetuate discriminatory harm as a result from overrepresentation of specific racial and ethnic groups during development and validation. Although the ML models themselves were independent of any explicit racial or ethnic bias, a biased data distribution contributed to the skewed outcomes³⁷; this is one of many identified mechanisms behind “unfair” models.³⁴ In health care, large datasets may not be representative of traditionally underrepresented minorities,³⁸ which may similarly lead to bias. However, a demographic-specific biological susceptibility/response might also contribute to unequal distributions of patients,^39,40 and this could help improve prediction. Recent data evaluating clinical use of ML models suggest that a significant number of them did not have any evaluation of racial bias,⁴¹ as there is currently no standardized process to do so. Of the exceptions, studies designed to predict mortality or sepsis in critically ill patients were shown to be free of any bias.⁴² Further investigation into this field is therefore indicated, and careful attention must be given to precisely delineate between the two effects and their degrees of impact on patient data.

Approaches to Data Missingness

A common theme to the obstacles of generalizability thus far is data heterogeneity.⁸ Missingness similarly contributes another dimension to the data, increasing its heterogeneity. Several methods have been proposed and implemented in the clinical prediction models to handle this challenge, including omission, imputation, and physiology-focused solutions. These are applied depending on the type of missingness that is present in each problem. It should also be noted that existing studies on the clinical uses of ML do not agree on a universal practice, even for the same missingness mechanism. Our goal for this section is to therefore present the prominent techniques for handling missingness to provide clarity to the existing literature.

Of the methods that can handle missing data, omission is the simplest and least computationally demanding, an important consideration for processing large health care datasets.⁴³ The most reported process of omission for studies of prediction models that use ML is complete case analysis (CCA).⁶ As its name suggests, CCA only includes patient cases that are not missing any data on the variables of interest.^44,45 However, its use is limited to datasets with MCAR data, as applying CCA to MAR or MNAR studies can introduce bias from nonrandom deletion of patients.^46–48 Consider a dataset with MAR data, younger diabetic patients have a higher rate of missing self-reported blood sugar data versus older diabetic patients as a result from only beginning the recording regimen. If CCA is applied here, we will disproportionately delete data from younger patients, so an ML model trained on this data will be biased toward making predictions for older diabetic patients. Nevertheless, even if CCA is correctly used for MCAR data, it still suffers from decreasing statistical power,⁴⁹ a flaw inherent to data omission.

More complex than methods of omission are those of imputation. Unlike omission, imputation requires added computation to produce more complete datasets, typically with less bias.⁵⁰ Three broad subtypes of imputation can achieve this with varying degrees of success: simple, hot deck, and multiple imputation.⁴⁵ Simple imputation replaces all missing values of a particular parameter with a single value computed from the present data. Hot deck imputation replaces a patient’s missing value with one computed from a subset of patients with similar characteristics. This is repeated for each patient with missing data. Multiple imputation involves multiple rounds of simple imputation with minor changes; for a missing patient value, one is computed from a random subset of the existing dataset. Additional values are then computed from different random subsets, resulting in multiple possible replacement values. An aggregate of these values is taken (eg, mean), which ultimately replaces the missing one. Although imputation usually results in decreased bias of the dataset when compared with methods of omission on MAR or MNAR data, simple imputation may increase bias.²⁷ Specifically, for a large subset of missing data, imputation with a single value can artificially decrease the variability of the dataset. Models trained on such data may subsequently have decreased generalizability to patient data from external cohorts. Hot deck and multiple imputations do not share this immediate weakness and generally result in more balanced datasets. Yet unlike simple imputation, they are more computationally expensive to perform.

Simple, hot deck, and multiple imputation methods provide a collective foundation for increasingly sophisticated variations of imputation. These should be thoughtfully used for predictive tasks because careless estimation of missing values can introduce artifacts that impact model accuracy. Specific factors that influence the type of imputation method used include study objectives, importance of missing data to these objectives, the amount of missing data, and the learning method used in an ML model.⁵¹ As such, Table 2 summarizes major studies of ML use in clinical contexts and their imputation and learning methods.

Table 2.

Major studies of clinical applications of machine learning, method of imputation for handling missingness and learning method used

Study	Missingness Method Used	Learning Method Used
Development and validation of an ICU mortality prediction model (Davoodi et al, 2018)52	Gaussian Imputation by Chained Equation	Deep rule-based fuzzy model
Development and validation of an in-hospital mortality prediction model for AKI (Lin et al, 2019)53	Mean Imputation	Random forest
Derivation and validation of novel sepsis phenotypes (Seymour et al, 2019)54	Multiple Imputation with Chained Equations	Latent class analysis
Development and validation of a volume responsiveness prediction model in oliguric AKI (Zhang et al, 2019)97	Multivariate Imputation by Chained Equation	XGBoost
Sepsis predictive model designed to identify instances of ML prediction uncertainty (Shashikumar, et al 2021)14	Mean Imputation and Sample-and-Hold with a weighted input layer to learn the “hold” duration	Conformal prediction

Abbreviation: AKI, acute kidney injury.

We have thus far discussed statistical methods of handling classic mechanisms of missingness. Although these are useful for completing datasets with minimal bias,⁴⁴ predictive models may actually find additional physiologic value in the missing data itself, a process known as the missing indicator method (MIM).^55,56 A critical distinction must be made here, as omission and imputation methods are advantageous at completing datasets with minimal bias, they optimize for parameter estimation.⁵⁷ Alternatively, MIM may be more suitable for clinical prediction because it encompasses implicit factors that may be relevant to patient outcomes.^58,59 To clarify this concept, consider a dataset where all patients had a white blood count (WBC) ordered but at different times (4 AM and 4 PM). Agniel and colleagues demonstrated that patients with a normal 4 AM WBC were associated with a greater mortality rate than those with an abnormally high or low 4 PM WBC. Here, missingness of the 4 AM WBC is associated with an improved patient outcome, which is likely due to the fact that higher acuity patients require closer monitoring throughout the day and night.¹⁶ As this example demonstrates, the type of missing data should be factored into its missingness pattern because the frequency of clinical and laboratory measurements are dictated by different guidelines and indications—not all missing data are of equal importance. Importantly, caution should be used with MIM approaches as this relies heavily on local practices and behaviors and this may lead to poor generalizability. Although MIM might introduce bias in estimation of causal relationships,⁶⁰ this can effectively improve predictive performance, a phenomenon termed “Stein’s Paradox.”^7,61

When it comes to missingness, the prediction models benefit from statistical methods and MIM. MIM can provide additional insight into the state of the patient. However, it may decrease model generalizability because it strongly assumes that both the validation and development cohorts exhibit the same missingness mechanism. This is likely not the case, as missingness can be highly dependent on context. Over-reliance on MIM also limits the utility of a model as it considers information encoded in ordering clinical tests; they are consequently geared toward quantifying physician thinking instead of suggesting previously unconsidered diagnoses.¹⁵ On the other hand, causal information in predictive models permits counterfactual prediction, a crucial component of models that provide the basis of a clinical decision.⁶² How a prediction model handles missingness accordingly depends on its purpose. A focus on generalizability and counterfactual prediction should prioritize statistical methods (ie, omission/imputation), and attention to a specific environment and risk assessment should prioritize MIM.^63,64 Being that each method has its own use case, many studied models use a combination of the two rather than complete reliance on either. Still, recent reviews of ML models have frequently demonstrated an insufficient or ignored method of handling missing data.^65–70

MACHINE LEARNING-BASED SOLUTIONS

There exist various ML methods which may improve generalizability of predictive models for commonly encountered syndromic conditions in the ICU. They factor in the systemic differences between institutions described in Heterogeneity in Health Care. We focus on three approaches which critical care physicians may encounter when evaluating an ML model: transfer learning, continual learning, and conformal prediction. Each of these approaches has distinct methodologies and benefits which can yield improved performance under various situations.

Transfer Learning

Transfer learning is a technique in ML which has seen select use in health care. The current applications have been largely limited to oncology⁷¹ and medical imaging,⁷² with only recent applications into critical care. Conceptually, understanding transfer learning can be illustrated as follows: a prediction model (eg, to predict delayed septic shock) is developed and validated at a single institution. Although it can immediately be applied to a second institution, subtle differences between the locales (see Heterogeneity in Health Care) often dictate that the ML model’s performance will be inferior to that of the original development institution. The initial development and validation of the ML model on a larger dataset may alleviate this drop in predictive ability, but this is not always possible. Transfer learning offers a solution to these subtle differences by using a small and representative dataset from the new location to optimize model parameters for it⁷³ (Fig. 1). Importantly, this bypasses the significant cost and data required for development and validation of a novel ML model per distinct institution to achieve similar performance. The utilization of a smaller dataset for retraining and fine-tuning of the original model is more computationally efficient and allows smaller hospitals to use such tools. However, being that the model must undergo retraining at each unique location, regulatory concerns arise as novel variants of the original model accumulate on a large scale. Nevertheless, examples of transfer learning applications in critical care include (1) fine-tuning a tracheal intubation prediction model for patients with COVID-19 pneumonia,⁷⁴ (2) adapting a delayed septic shock prediction model for use at various external institutions,¹¹ (3) predicting mortality in patients with end-stage renal disease,⁷⁵ (4) predicting acute kidney injury,⁷⁶ and (5) predicting acute respiratory distress syndrome on radiographs.⁷⁷ Test characteristics in these scenarios were significantly improved with the use of transfer learning.

Fig. 1.

Example of applying transfer learning to a delayed septic shock prediction model. The initial ML model was fine-tuned using data at a second site. The use of transfer learning significantly increased test characteristics (AUCroc) of the delayed septic shock model at the validation site. AUCroc, area under the curve of the receiver operating characteristic curve. (From Wardi, G. et al Predicting Progression to Septic Shock in the Emergency Department Using an Externally Generalizable Machine-Learning Algorithm. Ann. Emerg. Med. 77, 395–406 (2021); with permission.)

Continual Learning

Continual learning (also referred to as lifelong learning, incremental learning, or sequential learning) describes a model that continuously learns and evolves based on increasing data, fed over time, with retention of previously gained knowledge.^78–81 In this way, it is intuitively appealing and like human cognition. One well-known use of continual learning is found in recommender systems of companies such as Amazon and Netflix—the model is continually updated with labeled data from interactions with the end-user to reflect changes in personal preference over time.⁸² Yet unlike transfer learning, research into its clinical applications has been meager in the critical care domain, and no implementations currently exist⁸³ because of the considerable potential for “catastrophic forgetting.” This describes a phenomenon in which new information interferes with previously learned patterns, resulting in a paradoxic decrease in model performance. There also exist privacy concerns as the single model is continuously fed sensitive clinical data from various institutions. Observational data suggest this approach may improve predictive models over time, such as in the early prediction of sepsis,⁸⁴ medication dosing,⁸⁵ or augmentation of imaging studies performed in critically ill patients.⁸⁶ Nevertheless, these have not yet been translated into clinical tools.

Conformal Prediction

Conformal prediction refers to a model’s assessment of the uncertainty of a prediction based on the past experience. Intuitively, when a model encounters a scenario like its training dataset, the confidence in prediction is high. However, when a model encounters a scenario where input data are significantly different (non-conformal) from training data, the utility and confidence of the prediction are uncertain. Conformal prediction is therefore a mathematical approach that quantifies the uncertainty of an ML prediction. In effect, this allows the model to say “I don’t know” for inputs that are foreign from training data.^87–90 Applications of conformal prediction have been used in various nonmedical fields, including facial recognition, financial risk, and language recognition. In health care, it has been used to augment breast cancer diagnosis⁹¹ and prediction assessment of stroke risk,⁹² albeit primarily under research applications. Conformal prediction has more recently been described to assist in sepsis prediction, as shown in Fig. 2.¹⁴ In this example, potential sepsis cases in which an ML model had low certainty in prediction were identified and resulted in a significant decrease in false alarms. The ideal application of conformal prediction in the ICU would follow a similar pattern: alerting clinicians to scenarios in which a predictive model has low certainty of a prediction may increase their trust in ML predictive scores.

Fig. 2.

Example of applying conformal prediction to a sepsis prediction model. If the model does not recognize the input data from a patient, the conformal prediction layer “rejects” the data, and the sepsis prediction layer alerts the end-user that there exists a high degree of uncertainty. This resulted in a significant decrease in false alarms. (From Shashikumar SP, Wardi G, Malhotra A, Nemati S. Artificial intelligence sepsis prediction algorithm learns to say “I don’t know”. NPJ Digit Med. 2021;4(1):134. Published 2021 Sep 9. doi:10.1038/s41746–021-00504–6.)

DISCUSSION

ML models have the potential to significantly improve care of critically ill patients by leveraging the data-rich nature of the ICU. However, despite promising research, their real-world implementations in critical care are presently scant; this unfortunately growing chasm between what has been developed and what has been implemented is a phenomenon referred to as the “implementation gap.”⁹³ Although there are various reasons behind this discrepancy, inherent challenges with generalizability in a syndrome-based, heterogeneous patient landscape have significantly limited the utility of ML models in critical care. Our review explores many of these obstacles. Various syndrome-based conditions with overlapping clinical characteristics are difficult for ML models to delineate due to a critical delay in patient information availability and ambiguity in syndrome diagnosis. These effects are especially pronounced in the ICU, where syndrome-based conditions are common, and patients rapidly evolve. Compounding the challenge is heterogeneity in health care data itself. Data recording and storage, local health care guidelines, and temporal shifts in data necessitate correction in models themselves. Although patient demographics and missing data can further contribute heterogenous dimensions, they can also convey valuable information that might help improve prediction model accuracy.

Prevailing research on ML models use novel methods of learning to overcome these challenges and ultimately maximize external validity: transfer learning, continual learning, and conformal prediction.^11–14 Transfer learning involves retraining a model with limited data at a new deployment site so that it learns the specific nuances of that particular location.⁷³ Continual learning describes a central model that constantly adjusts its set of rules as it is applied to more data all while maintaining acceptable performance on prior applications.^79–81 Conformal prediction will only allow the model to predict from data that it deems “conformant” with the training data—that is, if it detects dissimilar data that may result in poor performance, it will refrain from making predictions.^87–90

It is important to note that although we primarily demonstrate these in the context of sepsis prediction, ML models have been similarly studied for a variety of other prediction tasks, including respiratory failure in COVID-19 patients,^74,94–96 complications for critically ill patients,^{76,77,97–99} and in-hospital mortality risk.^75,100–105 Other uses for clinical ML models outside of predictive tasks include identification of health factors related to patient outcomes, novel intervention design, and allocation of resources.¹⁰⁶ Future research into the applications of ML for critically ill patients should focus on prospective implementations across multiple centers to demonstrate clinical value. Many studies thus far are retrospective, and only a few of them undergo prospective validation; an even fewer number undertake randomized ML trials.¹⁰⁷ Indeed, many clinical ML models are currently developed and validated to collect dust only then in the “model graveyard.”⁹³ Other deployed models, such as the Epic Sepsis Score (ESS), paint a cautionary tale to hospital systems that fail to perform rigorous testing and optimization during development: researchers at the University of Michigan described a significant performance drop and increased rate of false alarms of the ESS when implemented at their institution.¹⁰⁸ To conduct the multi-center trials necessary for demonstrating clinical value, models may therefore undergo local optimization. This can be accomplished via transfer learning or continual learning and further improved through conformal prediction or similar methods. Such approaches may help alleviate the problem of generalizability and improve test characteristics. To our knowledge, this has not yet been done and should thus be emphasized in future trial designs involving prospective validation of ML models.

SUMMARY

We believe that clinicians must understand the basics of ML and its major challenges to evaluate current and future models. This is a vital step for their successful implementation into clinical practice. Likewise, we believe that data scientists interested in the health care applications of ML must understand its unique clinical challenges; the barriers to generalizability can presently be overcome with solutions offered by transfer learning, continual learning, and conformal prediction. With increased attention, we are optimistic that a future with accurate and fair ML-based clinical aids is not far.

KEY POINTS.

• Barriers to the use of machine learning in critically ill patients include challenges with recognition of syndromic conditions, data missingness, and the underlying heterogeneity of health care systems which may limit the generalizability of machine-learning algorithms.

• Recent advances in machine-learning applications, such as transfer learning and conformal prediction, can overcome barriers to improve generalizability across various institutions.

• Future studies are required to confirm the benefit of these strategies in both experimental and routine clinical care.

CLINICS CARE POINTS.

• Syndromic conditions in the ICU are easy for clinicians to grasp, but many challenges exist for machine-learning models, which has thus far limited generalizability.

• Recent advances in machine-learning approaches may alleviate these concerns, although we still lack large, prospective trials demonstrating benefit in critically ill patients.