Predictive Modeling for Readmission to Intensive Care: A Systematic Review

OBJECTIVES:

To evaluate the methodologic rigor and predictive performance of models predicting ICU readmission; to understand the characteristics of ideal prediction models; and to elucidate relationships between appropriate triage decisions and patient outcomes.

DATA SOURCES:

PubMed, Web of Science, Cochrane, and Embase.

STUDY SELECTION:

Primary literature that reported the development or validation of ICU readmission prediction models within from 2010 to 2021.

DATA EXTRACTION:

Relevant study information was extracted independently by two authors using the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies checklist. Bias was evaluated using the Prediction model Risk Of Bias ASsessment Tool. Data sources, modeling methodology, definition of outcomes, performance, and risk of bias were critically evaluated to elucidate relevant relationships.

DATA SYNTHESIS:

Thirty-three articles describing models were included. Six studies had a high overall risk of bias due to improper inclusion criteria or omission of critical analysis details. Four other studies had an unclear overall risk of bias due to lack of detail describing the analysis. Overall, the most common (50% of studies) source of bias was the filtering of candidate predictors via univariate analysis. The poorest performing models used existing clinical risk or acuity scores such as Acute Physiologic Assessment and Chronic Health Evaluation II, Sequential Organ Failure Assessment, or Stability and Workload Index for Transfer as the sole predictor. The higher-performing ICU readmission prediction models used homogenous patient populations, specifically defined outcomes, and routinely collected predictors that were analyzed over time.

CONCLUSIONS:

Models predicting ICU readmission can achieve performance advantages by using longitudinal time series modeling, homogenous patient populations, and predictor variables tailored to those populations.

KEY POINTS

Question: What are characteristics of high-performing, generalizable, and transparent models for predicting ICU readmission?

Findings: Higher-performing ICU readmission prediction models used homogenous patient populations, specifically defined outcomes, and routinely collected predictors that were analyzed over time.

Meanings: Models predicting ICU readmission can achieve performance advantages by using longitudinal time series modeling, homogenous patient populations, and predictor variables tailored to those populations.

More than 4 million patients are admitted to ICUs in the United States each year (1). On average, 7% of all patients discharged from an ICU will be readmitted to the ICU (2). Episodes of readmission are associated with an approximately sevenfold increase in risk-adjusted mortality (3). With intensive care accounting for 25%–40% of all healthcare expenditures, intensivists seek to identify patients that can be safely transferred out of the ICU as early as possible (4). Yet, premature ICU discharge and unplanned readmission is associated with additional costs and prolonged hospital and ICU lengths of stay (5).

Accurately predicting ICU readmission has the potential to identify patients that would benefit from further medical optimization prior to transfer out of the ICU or continuous vital sign monitoring on hospital wards. Traditionally, clinical approaches to ICU discharge decisions involve examining the patient and performing a time-consuming, manual review of health records under strict time constraints. Under these conditions, clinicians tend to rely on heuristics, which can lead to cognitive errors (6–9). Not surprisingly, clinician predictions of a patient’s need for future ICU resources are variable and often inaccurate, to the detriment of patients (10, 11). Data-driven clinical decision-support tools can potentially minimize harm from cognitive errors by providing clinicians with accurate, precise predictions by modeling what has happened for similar patients in the past. Unfortunately, many lack accuracy, reproducibility, and clinical workflow integration. Better methods are needed to support ICU discharge decision-making.

To develop accurate and effective decision-support platforms that augment ICU discharge decision-making, we aim to 1) evaluate the methodologic rigor and predictive performance of models predicting ICU readmission and 2) understand the characteristics of high-performing, generalizable, and transparent prediction models.

MATERIALS AND METHODS

PubMed, Web of Science, Cochrane, and Embase were systematically searched for articles related to the development or validation of ICU readmission prediction models within the past 11 years (2010–2021) on April 1, 2021. Searches were tailored to each database to use the annotated concepts specific to each database, such as PubMed’s medical subject heading terms (see Supplemental Digital Content, Appendix A, http://links.lww.com/CCX/B130). The search terms were batched into three groups: ICU, readmission, and prediction/modeling.

Two authors independently reviewed the abstracts for article inclusion and exclusion criteria, with any disagreements settled by a third investigator. Full texts of the articles that passed the screening were reviewed for relevance to review objectives independently by two authors, resulting in a final set of articles. Study information was extracted from each of the relevant articles using the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies checklist; bias was evaluated using the Prediction model Risk Of Bias ASsessment Tool (PROBAST) (12, 13).

RESULTS

One-hundred sixteen unique articles were identified from the 181 articles returned by the four database searches (Fig. 1). Title/Abstract screening reduced the total number of articles to 66, which was further reduced to the final number of 33 after full-text screening. The primary reasons for exclusion were non-ICU readmission prediction model (21), abstract only (4), did not evaluate prediction model (4), non-English full text (3), nonprimary literature (1).

Figure 1.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram.

Thirty-three studies were included in the final analysis, as summarized in Supplemental Table 1 (http://links.lww.com/CCX/B130) (14–46). Twenty-five studies recruited patients from a single center; eight studies (14–19, 29, 42) reported results from two or more centers. The most common source of patient data was Medical Information Mart for Intensive Care (MIMIC), which was used in nine studies (14, 19, 21–25, 30, 43) (Supplemental Table 2, http://links.lww.com/CCX/B130). ICUs with more specific patient populations (e.g., neuro ICU only or postoperative cardiac ICU only) sourced 12 studies (16, 17, 20, 25, 26, 29, 31, 32, 34, 37, 38, 44). Twenty-one studies used ICU readmission within windows that ranged from six hours to 30 days as the primary endpoint. Thirteen studies used a combined endpoint that included death (14, 15, 18, 22, 24–29, 42, 44, 45), cardiac arrest (14, 29), or rapid response team deployment (27) in addition to ICU readmission. One study evaluated ICU readmission as both a single and combined endpoint (26). One study derived two models to predict ICU readmission and death following ICU discharge separately (18). Seven studies used time series models that accounted for longitudinal patient trends (21–25, 30, 44).

Details of study methodology, validation, and sensitivity analyses are summarized in Supplemental Table 2 (http://links.lww.com/CCX/B130). Six studies accounted for reasons behind ICU readmission (20, 31–33, 36, 37), with a single study limiting results to only readmissions directly related to the initial ICU admission (32). In 22 studies, exclusion criteria comprised variations of the following: outbound facility transfer, death during index ICU admission, do not resuscitate (DNR) or withdrawal of invasive physiologic support, and an ICU stay less than 24 hours. Other exclusion criteria, particularly for the surgical or postoperative ICU studies, were surgeries not pertaining to the index ICU stay, missing data, death in surgery, or severe illness such as congenital valvular disease or brain tumor. Rationale for predictor variable selection was stated in 18 studies and was based on clinical experience (six studies), routinely available variables (four), previously unvalidated models (five), hypothesis testing of specific predictor variables (five), or restricted readmission predictors previously identified in the literature (two). Two studies selected predictor variables from a larger pool based on the greatest differences between readmitted and nonreadmitted patient cohorts (18, 21). Validation was performed in 25 studies—internal validation in 19 articles and both internal and external validation in six articles (14, 17, 21, 29, 42, 45). Seventeen studies used discretization to transform continuous variables into categorical variables.

The most common modeling technique was logistic regression. Twenty-three studies evaluated at least one logistic regression model (15–18, 23, 24, 26–29, 31–40, 42, 43, 46). Other model techniques included recurrent neural networks (22, 30), gradient boosting machines (14, 25), and artificial neural networks (23). Model performance was commonly assessed by area under the receiver operating characteristic curve (AUROC), a metric that shows the tradeoff between the true positive rate and false positive rate across different decision thresholds, where 0.5 represents random chance (e.g., a coin flip) and 1 represents perfect discrimination. The AUROC of a test with acceptable discriminating capabilities lies between 0.70 and 0.80. AUROC ranged from 0.51 (26) to 0.94 (21). Models that did not exceed an AUROC of 0.75 tended to 1) use existing clinical risk or acuity scores such as Acute Physiologic Assessment and Chronic Health Evaluation (APACHE) II (26, 34) or Stability and Workload Index for Transfer as the sole predictor (28, 30, 31) and 2) predict readmission based on singular laboratory values or findings (36, 40). The best performing models did one of the following: 1) used models with longitudinal temporal components that can analyze trends over time (21, 22), 2) used a more homogenous, post-surgical patient population (16, 17, 20, 29, 31, 32), or 3) limited ICU readmissions to those directly related to original admission (32).

Model sensitivity and specificity were reported in 15 studies. Sensitivity ranged from 18% to 91% (21, 40). Specificity ranged from 22% to 95% (14, 28). One study reported a range of sensitivities and specificities that varied from 0% to 99% based on the predicted likelihood of readmission or death. Only four models reported both sensitivities and specificities above 70% (18, 21, 29, 32). The models with high sensitivity followed the same three principles as above (22, 30, 31), whereas the models with high specificity used novel predictor variables, such as eosinophilia (14, 40). Models with high AUROC and sensitivity/specificity adhered to the patterns described above and also frequently employed some form of data imputation, internal validation, and ICU readmission timing (18, 21, 22, 29, 32, 34).

Six studies had a high overall risk of bias as indicated by PROBAST (26, 33, 34, 42–44) due to improper inclusion criteria (33, 34, 42, 43) or omission of critical analysis details (26, 44). Four other studies had an unclear overall risk of bias due to lack of detail describing the analysis (31, 33, 34, 41). Overall, the most common (50% of studies) source of bias was the filtering of candidate predictors via univariate analysis (15, 26–28, 31–34, 37–40). The remaining studies used appropriate cohort definitions and analytical methods and were assessed as having a low risk of bias.

DISCUSSION

We identified methodological characteristics that enhanced performance, generalizability, and transparency of ICU readmission prediction models. We did not unequivocally identify the “best” performing ICU prediction model in our review due to the limitations of the metric reported across all studies (AUROC) and the nuance and complexity of adjudicating what describes the “best” ICU prediction model.

Clearly Define a Homogenous Patient Population

A single prediction model that is highly accurate and generalizable to any ICU patient would be quite useful, but the authors found no such model. Instead, it may be preferable to identify patient subgroups with distinct baseline characteristics and physiologic signatures of illness and use separate models for each subgroup. This is consistent with other research trends, such as New York State’s Report Cards on Cardiac Surgeons, which succeeded in generating credible comparisons between the mortality rates of individual providers performing coronary artery bypass grafting; similar projects, such as the Healthcare Financing Administration’s annual report on mortality rates among hospitalized Medicare patients, failed in part due to the population heterogeneity (47). For example, models designed and validated exclusively among cardiac surgery patients performed well (16, 17, 29, 31, 38). Even among models that used a similar patient subgroup, models trained and evaluated in a single center tended to outperform multicenter models.

One counter-interpretation of our results is that the higher performance of homogenous models was a result of overfitting and, thus, a sign of poor external generalizability. In the present review, the likelihood of overfitting was deemed low. Eight of the 11 high-performing studies had large sample sizes (> 1,000), and 10 involved some form of internal or external validation.

A likely contributing factor that helps to explain the increase in performance observed between homogenous single center studies and heterogenous multicenter studies is the strong and often overriding influence that operational and administrative factors play in ICU bed allocation and level of care determination. This includes differences in resource availability (e.g., number of beds, presence of step-down units, and staffing) as well as care protocols that can vary widely both inter- and intra-institutionally (e.g., medical vs surgical services) and can decouple ICU readmission decisions from the underlying patient physiology.

A likely example of this is the increase in performance (AUROC, sensitivity, specificity) observed between the separate logistic regression models for ICU readmission and mortality developed by Badawi and Breslow (18). Death is a physiologic phenomenon that escapes many of the operational and administrative hurdles encountered when admitting a patient to the ICU. Two additional contributing factors, which may help to explain this difference, are that the patients who died represented a smaller, but possibly more physiologically extreme, pole of the dataset and that the model predicting mortality relied on fewer admission variables and included more last-day physiologic variables reflecting patient state at the time of discharge.

Last, it is important to distinguish between palliative care, DNR, or withdrawal of invasive physiologic support patients from the general population as their goals for care tend to differ significantly, which can impact the training of data-driven models.

The importance of homogeneity was substantiated by several studies with a low risk of bias (Supplemental Table 3, http://links.lww.com/CCX/B130) (14–25, 27–30, 32, 37–43, 45, 46). While developing distinct models for each patient subgroup may prove cumbersome if performed to the extreme, limiting future studies to a single surgery type (e.g., cardiac surgery) or a single initial complaint (e.g., respiratory distress) may lead to better overall model performance.

Be Specific When Defining Outcomes

We found that few models controlled for the reasons behind ICU readmission. ICU readmission is often due to multiple factors not necessarily related to patient stability upon discharge, including nosocomial infection, administrative causes, or pathologic causes unrelated to stability during initial discharge. A multicenter cohort study found that less than half of ICU readmissions within the same hospitalization pertained to the initial complaint, and almost half experienced new complications, particularly related to respiratory diseases (48). Furthermore, ICU readmission with the same diagnosis as the initial admission may not represent patient acuity during their initial stay but deterioration at home or within the ward. As time increases between the initial ICU stay and readmission, the likelihood of confounding factors leading to readmission increases. Only seven articles restricted ICU readmissions to within 48 hours or less, and the only model to restrict readmissions to those related to initial admission was one of the highest performing. Restricting readmission causes to physiologic exacerbations related to initial admission and restricting to earlier readmission may be valuable tools that allow a closer snapshot of patient state during their initial ICU stay and should be further considered.

Incorporate Temporality

Models that incorporate time series data generally exhibit greater accuracy than static models. This is likely attributable to the nature of human pathophysiology. Illness severity evolves and snapshot assessments can misrepresent risk for decompensation. Fortunately, electronic health record data are typically recorded with date and time values that can capture trends in illness severity over time. This approach has been successfully used to predict mortality and specific conditions, such as acute kidney injury (49–52).

Use Routinely Collected Predictors

Models should be developed using select, routinely available predictors. Overly complex algorithms can hamper clinicians in practice, where time-sensitive treatments and diagnosis can be critical to patient survival. We found that model simplicity could be improved, as only six studies deliberately limited predictors to those routinely collected, and only one deliberately restricted the number of predictors.

Transparently Report Predictive Performance

AUROC can serve as a valuable indicator of discriminative performance in balanced datasets; however, in the prediction of ICU readmission, a relatively low frequency event, performance can appear deceivingly high for models that are overly biased toward the majority class (no readmission). Alternative performance metrics such as F1 score ($\frac{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}}{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}+\frac{1}{2}\left(\mathrm{F}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}+\mathrm{F}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e}\mathrm{N}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\right)}$) and area under the precision-recall curve ($\frac{\frac{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}}{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}+\mathrm{F}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}}}{\frac{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}}{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}+\mathrm{F}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}}}$), which were reported in a handful of included studies, are robust against dataset imbalance and may paint a more clear and fair picture of how one model compares to another. Additionally, calibration metrics such as Brier score, Hosmer-Lemeshow statistic, and average absolute error can further elucidate the overall performance of models in a robust manner (53, 54). In our review, 18 (55%) of studies reported calibration measurements, the majority of which used the Hosmer-Lemeshow statistic. Further complicating the evaluation of superior model performance are the similar yet distinct values and priorities of stakeholders in the healthcare delivery system (e.g., physicians, patients, and healthcare administrators), which often vary both across and within roles.

Transparently Report Methodology

Approximately half of the reported studies omitted key details, which hampered our ability to critically evaluate the methodology employed in the creation of the models described. This highlights the importance of using reporting guidelines such as Transparent Reporting of multivariate prediction model for the Individual Prognosis Or Diagnosis (TRIPOD) that ensure essential elements for the appraisal of methodological rigor are presented clearly and consistently across all studies (55). None of the included studies reported compliance with the TRIPOD reporting guidelines (Supplemental Table 3, http://links.lww.com/CCX/B130).

Six studies had a high overall risk of bias as indicated by PROBAST (26, 33, 34, 42–44) due to improper inclusion criteria (33, 34, 42, 43) or omission of critical analysis details (26, 44). Four other studies had an unclear overall risk of bias due to lack of detail describing the analysis (31, 33, 34, 41).

Previous systematic reviews of ICU readmission prediction models focused primarily on the outcome (i.e., ICU readmission) rather than model characteristics. Nevertheless, the major findings from other systematic reviews are consistent with findings from the present review. Rosenberg and Watts (2) found that unstable vital signs at the time of ICU discharge were the strongest predictor of ICU readmission. Similarly, Wong et al (56) reported that greater illness severity, as measured with APACHE or Simplified Acute Physiology Score scales at the time of ICU admission or discharge, was associated with increased risk for ICU readmission (57). Apart from the classic approach to using illness severity to predict ICU admission, Vollam et al (58) performed a systematic review to evaluate associations between time of ICU discharge and outcomes. The authors found that evening and nighttime ICU discharge transfer times were associated with an increased prevalence of ICU readmission and in-hospital mortality, thus, underscoring the importance of health system factors in patient triage decisions and outcomes. In a novel assessment of prediction models themselves, Markazi-Moghaddam et al (59) examined the performance and validity of five ICU readmission prediction models that met relatively stringent quality criteria, finding not only heterogeneous and often suboptimal methods but also weak external validity. All of the studies evaluated by Markazi-Moghaddam et al (59) were among those included in the present review. There was general agreement between reviews regarding the content and quality of these three common studies. Collectively, other systematic reviews had different objectives and breadth of included studies but drew conclusions consistent with the present review.

The major findings from the present review are that transparent, high-performing, and generalizable models utilizing clearly defined homogenous patient populations with few ICU readmission confounders, specifically defined outcomes, incorporating routine datapoints collected over time, and presented transparently in terms of methodology and model performance, can inform future investigations in building effective models primed to augment decision-making when applied clinically. To achieve homogeneity, it is necessary to identify patient subgroups with distinct baseline characteristics and physiologic signatures of illness and use separate models for each subgroup. To avoid overfitting, models should incorporate a sufficient sample size, enumerate differences between subgroups and the patient population as a whole, and perform both internal and external validation. Most models presented in this review were published prior to the dissemination of reporting guidelines from editors of critical care journals (60); future investigations should adhere to these guidelines for selecting predictor variables, operationalizing variables, dealing with missing data, appropriate validation, model performance measures and their interpretation, and reporting practices. Future prediction models should also seek to represent and include clinical decisions for treatments, continuous vital sign monitoring, and other relevant aspects of clinical care (61, 62). To determine whether additional observations contribute significantly to predictive performance, future investigations should generate model learning curves in which the y-axis plots a performance metric (ideally the F1 score, representing the weighted average of precision and recall) and the x-axis plots sequential epochs of observations during training. When the slope plateaus, additional observations (e.g., patients or ICU admissions) will not improve performance, but additional input variables (e.g., demographics or biomarkers) might. If the training set is exhausted before the slope plateaus, performance might improve with additional observations. Model learning curves also allow future research to build upon previously reported models in an evidence-based fashion.

This study has several limitations. First, the included studies demonstrate significant heterogeneity, which precludes the performance of a meta-analysis. Although a meta-analysis would help elucidate relationships between appropriate triage decisions and patient outcomes, this objective has been met by previous studies. Meta-analysis was not necessary to complete the other two objectives of this review: critically evaluating the methodologic rigor and predictive performance of models that predict ICU readmission and understanding the characteristics of ideal prediction models. Second, while most included articles used unique datasets generated at their institutions, many used common datasets, namely MIMIC. As MIMIC data are drawn from a single institution, this introduces possible systemic findings into our conclusions. Third, eight of our articles combined mortality with readmission as a single endpoint, which can change the frequency of the outcome in each dataset and introduce bias. Thus, care should be practiced when interpreting the results of the mixed outcome studies, and a more balanced approach using AUROC, sensitivity, and specificity should be employed.

CONCLUSIONS

Among models predicting ICU readmission, models with the highest transparency, performance, and generalizability used clearly defined homogenous patient populations with few ICU readmission confounders, specifically defined outcomes, incorporated routine datapoints collected over time, and presented methodology and model performance in a forthright and unambiguous manner. These findings were substantiated by several studies that had a low risk of bias. Future investigations regarding the prediction of ICU readmission should consider longitudinal time series modeling using homogenous patient populations, with specific outcomes, and routinely collected predictor variables tailored to those populations. Their methodologies should be reported in accordance with consensus guidelines (e.g., TRIPOD) and include sufficient performance and calibration metrics in order to fairly assess model quality.