Risk Prediction Models for Hospital Mortality in General Medical Patients: A Systematic Review

Article Highlights

• •The performance and validity of hospital mortality prediction models in general medical patients have not been well appraised.
• •Hospital mortality prediction models were either developed or validated in all general medical patients or those with infection.
• •Both general and infection models had high risk of bias with variable performance. No models have been well validated to ensure generalizability and stability of performance.
• •Further validated models are required to predict mortality and guide mortality reduction interventions.

Abstract

Objective

To systematically review contemporary prediction models for hospital mortality developed or validated in general medical patients.

Methods

We screened articles in five databases, from January 1, 2010, through April 7, 2022, and the bibliography of articles selected for final inclusion. We assessed the quality for risk of bias and applicability using the Prediction Model Risk of Bias Assessment Tool (PROBAST) and extracted data using the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist. Two investigators independently screened each article, assessed quality, and extracted data.

Results

From 20,424 unique articles, we identified 15 models in 8 studies across 10 countries. The studies included 280,793 general medical patients and 19,923 hospital deaths. Models included 7 early warning scores, 2 comorbidities indices, and 6 combination models. Ten models were studied in all general medical patients (general models) and 7 in general medical patients with infection (infection models). Of the 15 models, 13 were developed using logistic or Poisson regression and 2 using machine learning methods. Also, 4 of 15 models reported on handling of missing values. None of the infection models had high discrimination, whereas 4 of 10 general models had high discrimination (area under curve >0.8). Only 1 model appropriately assessed calibration. All models had high risk of bias; 4 of 10 general models and 5 of 7 infection models had low concern for applicability for general medical patients.

Conclusion

Mortality prediction models for general medical patients were sparse and differed in quality, applicability, and discrimination. These models require hospital-level validation and/or recalibration in general medical patients to guide mortality reduction interventions.

Introduction

The global burden of hospital mortality from preventable and nonpreventable causes is high.¹ Recent studies estimated that 3.1% of hospital deaths were preventable² and that 134 million adverse events annually in low- and middle-income countries resulted in 2.6 million hospital deaths.^1,3 Despite these and other studies, there is sparse information on risk prediction models for preventable hospital mortality, limiting the development of mortality reduction interventions. Furthermore, predicting nonpreventable deaths may facilitate earlier discussion of advanced care directives or transition to palliative or hospice care.

In the United States, from 2000 to 2010, the annual overall hospital mortality rate declined from 2.5/100 patients to 2.0/100 patients.⁴ While reassuring, the overall rate concealed changes in mortality were attributed to different conditions, such as kidney disease (–65%), heart disease (–16%), and septicemia (+17%).⁴ In this context, disease-specific models have been developed to predict risk of hospital mortality for conditions including acute myocardial infarction, stroke, heart failure, pancreatitis, and pneumonia.5, 6, 7, 8, 9 In contrast to disease-specific models, there are fewer models for general medical patients, who frequently present with undifferentiated and/or multiple medical conditions and experience an unpredictable hospital course. Furthermore, general medical patients have large amounts of evolving biopsychosocial data that are challenging to integrate in prediction models for mortality. Therefore, early identification of general medical patients at risk of hospital mortality may improve decision making, guide escalation in care, and reduce the risk of preventable deaths. Despite this, there is sparse information on risk prediction models for hospital mortality for general medical patients, and specifically, no systematic appraisal of model quality and performance.

To address these knowledge gaps, we conducted a systematic review of risk prediction models for hospital mortality in general medical patients. The objective of the study was to evaluate models that predicted acute decompensation, focusing on deaths in general medical patients. We reviewed model characteristics and performance and critically appraised their quality and performance. These models may be validated, recalibrated, or improved to guide interventions to reduce hospital mortality.

Methods

The study was conducted by the Hospital Experiences to Advance Goals and Outcomes Network (HEXAGON) group at Mayo Clinic.10, 11, 12 The systematic review is reported following the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines and the protocol was prospectively registered (PROSPERO CRD42020176054).^13,14 The search strategy for all databases is provided in Supplemental Table 1 (available online).

Data Sources and Search Methods

We searched the peer-reviewed literature for articles on models to predict hospital mortality in general medical patients. We searched Ovid/Medline, Embase, Evidence-Based Medicine Reviews, Scopus, and Web of Science from January 1, 2010, through April 7, 2022. We restricted the search period to focus on models reflecting contemporary cohorts and hospital practice. The bibliography of articles selected for final inclusion was screened by one investigator.

Study Selection

We defined hospital mortality as death occurring in the hospital following admission to a general medical ward. We included original English language articles that reported at least one prediction model for mortality in adults (age ≥ 18 years) hospitalized on general medical wards. We focused on models designed to predict acute decompensation, focusing on mortality. We excluded studies with any of the following: patients admitted from the emergency department to the intensive care unit (ICU); patients on surgical, palliative care, oncology, or cardiology services; and patients with coronavirus disease from severe acute respiratory syndrome coronavirus 2 (COVID-19). We excluded studies that focused on one diagnosis/condition (eg, pneumonia). To avoid over-fitting of models, we excluded studies with fewer than 500 mortality events. We also excluded studies that focused on specific time horizons for hospital mortality (eg, 7-day mortality), as predicting mortality at any point during hospitalization may guide overall hospital care and mortality reduction interventions. After pilot screening, 2 investigators independently screened titles and/or abstracts (primary screening) and full-text (secondary screening) using Covidence.¹⁵

Data Extraction and Quality Assessment

We categorized the models as “general models” if developed and/or validated in all general medical patients and as “infection models” if developed and/or validated in patients admitted to medical wards with suspected infection. Data from articles selected for final inclusion were extracted at the model level using the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist.¹⁶ Quality was assessed using Prediction Model Risk of Bias Assessment Tool (PROBAST).¹⁷ Using PROBAST, we assessed risk of bias using 4 domains (Participants, Predictors, Outcome, and Analysis), and applicability using 3 domains (Participants, Predictors, and Outcome) (Supplemental file 2).

Conflict Resolution

Screening, data extraction, and quality assessment were conducted independently by two investigators, with conflicts resolved by discussion and consensus between the investigators or, if needed, by a third investigator.

Ethics Approval

The study used publicly available, deidentified data and forms and therefore did not require Institutional Review Board approval.

Results

Of 20,424 unique articles, 8 studies on 15 prediction models met selection criteria (Figure 1). The studies, published since 2017, were based in 10 countries across 4 world regions (Table 1 and Supplemental Table 2 [available online]). The 8 studies on 280,793 patients included 128,017 women (45.6%) and 19,923 hospital deaths (7.1%) (Table 1). Of the 15 models, 10 models were based on general medical patients, 7 models on general medical patients with infection, and 2 models were studied in both groups. Out of the 15 models, 7 were novel or modified,18, 19, 20, 21, 22, 23, 24 while the others^19,20,24,25 were preexisting models externally validated in general medical patients. Major findings of the systematic review are summarized in Figure 2. Heterogeneity in variables and models precluded a metaanalysis.

Figure 1.

Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) 2020 flowchart of articles included for data extraction. The search strategy is provided in Table 1 in the Supplement (available online).

Table 1.

Characteristics of Included Studies.

Characteristic	Chen (2017)24	Fabbian (2017)18	Fabbian (2018)25	Moore (2017)20	Rasmussen (2018)19	Sakhnini (2017)21	Schwartz (2018)22	Soffer (2020)23
Study objective	Development and internal validation of a novel model, and external validation of existing models	Model modification	External validation	Development and internal validation of a novel model, and external validation of existing models	Model modification and external validation of existing models	Model development and internal validation	Model development and internal validation	Model development and internal validation
Location (number of centers)	Taiwan (1)	Italy (1)	Italy (1)	Gabon, Malawi, Sierra Leone, Tanzania, Uganda, Zambia (unknown)	Denmark (2)	Israel (1)	Israel (1)	Israel (1)
Study period*	2010-2012	2000-2013	2013-2016	2009-2015	2013-2015	2013-2015	2012-2015	Development (2013-2017), Validation (2018)
Inclusion criteria	Adults in the emergency department with ICD-9 codes for infection, and with ≥2 sets of blood cultures	Admissions to the medical ward	Admissions to the medical ward with an infectious disease diagnosis	Admissions to the medical ward with mortality data and >50% of recorded vital signs	Admissions to the medical ward and suPAR analysis	Admissions to the medical ward ≥24 h	Admissions to the medical ward ≥24 h	Patients (18-100 years) admitted to the medical ward
Exclusion criteria	Patients transferred from other medical institutions, with repeat hospital visits, and/or with traumatic injuries	Patients transferred to surgical departments	Patients transferred to surgical departments or intensive care units	Not applicable	Admissions for surgical intervention or acute pediatric, obstetric, and gastroenterological conditions; patients in whom suPAR level was not ordered, suPAR result was missing, or other reasons	Admissions classified under symptoms, signs, and ill-defined conditions (ICD-9 codes 780-799), and under observation (ICD-9 codes V71, V71.2, V29.0, V29.1, V29.2, V29.8, V29.9)	Not applicable	Not applicable
Age (years), mean ± SD, median (IQR)†	65 (49-78)	73 ± 16	65 ± 25	36 (27-49)	61 (43-76)	Survived: 68 (18-105)⁎⁎ Died: 78 (23-105)⁎⁎	Survived: 64 (18-105)⁎⁎ Died: 77 (23-105)⁎⁎	Survived: 73 (62-83) Died: 82 (71-89)
Number of patients	Development: 7011 Validation: 12,110	Total: 75,586	Total: 12,173	Total: 5573 With infection: 3153	Total: 17,312	Development: 7268 Validation: 7843	Development: 10,788 Validation: 6867	Total: 118,262§§
Women, number (%)	3216 (45.9)	40,329 (53.4)	8053 (66.2)	2829 (50.8)	9194 (53.1)	Survived: 2984 (46.7)†† Died: 453 (51.4)††	Survived: 4404 (44.4) Died: 450 (51.5)	Died: 3042 (48.2)
Major admitting diagnoses (%)	Unspecified infections (71.2), respiratory infections (57.9), genitourinary infections (38.2)	% not provided	Pulmonary infection (34.3), nonspecified infection (33.8), urinary tract infection (17.5)	Not provided	Not provided	Survived: nonspecified diagnosis (20.5), heart failure (12.1), cerebrovascular disease (11.4). Died: pneumonia (23.4), sepsis and septicemia (14.5), malignant neoplasms (14.1)	Survived: nonspecified diagnosis (52.2), heart failure (6.5), pneumonia (5.7). Died: nonspecified diagnosis (22.8), pneumonia (19.3), sepsis and septicemia (13.2)	Survived: nonspecified chest pain (5.8), pneumonia (5.0), CHF exacerbation (2.7). Died: septic shock (13.7), pneumonia (13.3), respiratory failure (3.9)
Number of deaths§	Development: 479 Validation: 1145	6007	1545	Total: 966 With infection: 720	587	Development: 882 Validation: 582	Development: 874 Validation: 515	6311
Mortality prediction model	CHARM, CURB-65, MEDS, PIRO, SIRS	Modified Elixhauser Index	Modified Elixhauser Index	MEWS, qSOFA, UVA	NEWS, modified NEWS	Not applicable	Not applicable	Not applicable

^⁎All studies used single retrospective cohorts. Moore (2017)²⁰ used a pooled cohort.

^†Rounded to integers.

^§All studies included all hospital mortalities, except Soffer (2020),²³ which included only ward mortalities.

^⁎⁎Mean (range).

^††From development cohort.

^§§One dataset was used with gradient boosting.

Abbreviations listed in Supplemental Table 13 (available online).

Figure 2.

Summary of the main results of the systematic review on models for hospital mortality in general medical patients. Used with permission of Mayo Foundation for Medical Education and Research, all rights reserved.

Study Design, Cohorts, and Outcomes

The studies were based on cohorts of 5000 to more than 100,000 participants. All cohorts had a mean or median age above 60 years, except in Moore 2017,²⁰ which had a median age of 36 years and additionally provided subgroup analysis for general medical patients with infection. Two studies exclusively enrolled patients with infection (Chen, 2017²⁴ and Fabbian, 2018²⁵), while the others enrolled all patients admitted to the general medical wards. One study (Soffer 2020²³) focused on mortality in the general medical ward. The other studies focused on hospital mortality (in and outside the general medical ward) after admission to a general medical ward.

Prediction Models for All General Medical Patients

Ten of 15 models were based on general medical patients hospitalized with various medical conditions (Table 2). The Universal Vital Assessment (UVA) score²⁰ and 3 other models21, 22, 23 were developed and internally validated in general medical patients.20, 21, 22, 23 Two models were modified versions of the Elixhauser index¹⁸ and National Early Warning Score (NEWS).¹⁹ The Elixhauser Index,¹⁸ Modified Early Warning Score (MEWS),²⁰ NEWS,¹⁹ and Quick Sequential Organ Failure Assessment (qSOFA) score²⁰ were preexisting models externally validated in general medical patients. None of the studies both developed and externally validated model(s) for mortality. Studies that developed or modified models examined different predictors for hospital mortality, with the most common categories being vital signs (5 models) and patient comorbidities (4 models) (Table 2 and Supplemental Table 3 [available online]). Based on the type of predictors, the models were classified as Early Warning Score (EWS), comorbidity, and combination models (Supplemental Table 4, available online). EWS models were generally based on vital signs, with few models also integrating easily obtainable biomarkers. Comorbidity models were typically weighted scores of comorbidities from International Classification of Diseases (ICD) codes. Combination models contained variable categories of predictors including vital signs, comorbidities, and biomarkers. Most predictors were captured on hospital presentation or admission to the medical ward, with a minority from hospital progress notes and discharge summaries.^18,21, 22, 23 Predictors in all novel and modified models were selected using multivariable analysis. The most common methods to select predictors were probability values (P values) (n = 3 models)^18,19,21; UVA score used Bayesian information criterion (BIC)²⁰; and Schwartz (2018)²² used Akaike information criterion (AIC), corrected Akaike information criterion (AICC), and BIC (Table 2). Most general models used data in their native form, except the modified Elixhauser Index, which used age as a categorical variable (Supplemental Table 5, available online).

Table 2.

Prediction Models for All General Medical Patients (General Models)

Characteristic	Elixhauser Index18	Modified Elixhauser Index18	MEWS20	NEWS19	Modified NEWS19	qSOFA20	UVA20	Model by Sakhnini (2017)21	Model by Schwartz (2017)22	Model by Soffer (2020)23
Purpose of analysis	External validation/ comparison	Model modification	External validation	External validation/ comparison	Model modification	External validation	Model development/ internal validation	Model development/ Internal validation	Model development/ Internal validation	Model development/ Internal validation
Timing of predictor assessment	At diagnosis of comorbidity (ICD9 codes)	At diagnosis of comorbidity (ICD9 codes)	On admission	On admission	On admission	On admission	On admission	On admission	On admission	On admission
Number of candidate predictors	Not applicable	33	Not applicable	Not applicable	10	Not applicable	13	28	32	Not provided
Predictors in final model (no.)	Congestive heart failure, cardiac arrhythmias, valvular disease, pulmonary circulation disorders, peripheral vascular disorders, hypertension, paralysis and other neurological disorders, chronic pulmonary disease, diabetes mellitus, hypothyroidism, renal failure, liver disease, peptic ulcer disease excluding bleeding, HIV, lymphoma and cancer, rheumatoid arthritis/collagen vascular diseases, coagulopathy, obesity, weight loss, fluid and electrolyte disorders, anemia, alcohol and drug abuse, psychoses, depression (30)	Age, renal failure, male gender, other neurological disorders, lymphoma, solid tumor without metastasis, ischemic heart disease, congestive heart failure, coagulopathy, fluid and electrolyte disorders, liver disease, weight loss, metastatic cancer (13)	Temperature, HR, RR, SBP, level of consciousness (5)	RR, Temp, SBP, HR, AVPU, SpO2, supplemental oxygen (7)	NEWS predictors, age, sex, suPAR level (10)	RR, SBP, level of consciousness (3)	Temperature, HR, RR, SBP, SpO2, level of consciousness (GCS), HIV serostatus (7)	Age, BMI, mean arterial pressure on admission, previous admission within 3 prior months, heart failure, active malignancy, chronic use of statins, chronic use of antiplatelet agents, main admission diagnosis and secondary conditions (heart failure, urinary tract infection, pneumonia, sepsis and septicemia, renal failure, cancer, and acute coronary syndrome) (16)*	Age, BMI, admission within 3 prior months, statin intake, 4 laboratory variables (serum creatinine, hemoglobin, RDW, and hypoalbuminemia), and 2 background diseases (heart failure and malignancy) (10)	Chief complaint, age, home medications, number of home medications, comorbidities, number of comorbidities, RR, SpO2, fever, SBP, DBP, pulse, albumin, BUN, CRP, LDH, neutrophil count, eosinophil count, WBC count, calcium, AST, PCO2, lactate, protein, lymphocyte count, serum creatinine, phosphorus level, ALK-P, troponin-I, hemoglobin, potassium, glucose, sodium, PT, INR, GGT, platelet count, ED diagnosis, ED administered medications, arrival mode, ED wing, emergency severity index, pain score and textual words (catheter, shock, fluid, bedridden, oxygen, feeding tube, culture, the family, low, gas, deterioration, nursing, dementia, consciousness, ambulance, intubated, brought, condition, breathing)(62)†
Method for handling missing values	Not provided	Not provided	k-nearest neighbors imputation§	Missing vital signs assigned value of 0	Missing suPAR level values were excluded; missing vital signs assigned value of 0	k-nearest neighbors imputation§	k-nearest neighbors imputation§	Exclusion	Exclusion	Integrated into gradient boosting algorithm
Method for prediction model development	Not applicable	Logistic regression	Not applicable	Not applicable	Poisson regression	Not applicable	Logistic regression; decision trees and linear regression	Logistic regression	Penalized logistic regression (LASSO)	Multiple tree-based classifiers with gradient boosting
Method for selection of predictors during multivariable analysis	Not applicable	P value	Not applicable	Not applicable	P value	Not applicable	BIC	P value	AIC, AICC, BIC	Not applicable
Calibration method (result)	None	Hosmer-Lemeshow test (P < .001)	None	None	None	None	None	Hosmer-Lemeshow test (not provided)	None	None
AUC value (95% CI)⁎⁎	0.66 (0.65-0.66)	0.72 (0.71-0.73)	0.70 (0.68-0.71)	0.87 (0.85-0.88)	0.92 (0.91-0.92)	0.69 (0.67-0.72)	0.77 (0.75-0.79)	Development: 0.90 Validation: 0.81	Development: 0.89 (0.88-0.90) Validation: 0.86 (0.84-0.87)	0.92 (0.92-0.93)
Type of validation	External	None	External	External	None	External	Internal	Internal	Internal	Internal
Method of validation	Different time, area and investigators		Different time, area and investigators	Different time, area and investigators		Different time, area and investigators	10-fold cross-validation	Temporal data split	Bootstrap and temporal data split	Bootstrap and temporal data split

^⁎Obtained from Supplement.

^†List of predictors obtained from different tables reported in the article.

^§Variables with more than 50% missing values were excluded. Hospital mortality and HIV serostatus were not imputed.

^⁎⁎Rounded to 2 decimal places.

Abbreviations listed in Supplemental Table 13 (available online).

Most models were developed using logistic regression, and model performance was reported using discrimination. The modified Elixhauser Index,¹⁸ UVA score,²⁰ and model by Sakhnini (2017)²¹ were developed using logistic regression, while the model by Schwartz (2018)²² used the least absolute shrinkage and selection operator method. The modified NEWS¹⁹ was developed using Poisson regression, while the machine learning model by Soffer (2020)²³ employed tree-based classifiers with gradient boosting. Calibration was assessed using the Hosmer-Lemeshow test in the modified Elixhauser Index¹⁸ and in the model by Sakhnini (2017),²¹ whereas other models did not report calibration. In terms of model discrimination, NEWS,¹⁹ modified NEWS,¹⁹ and the models by Sakhnini (2017),²¹ Schwartz (2018)²², and Soffer (2020)²³ had high discrimination (area under curve [AUC] > 0.8), while the other models showed moderate discrimination (AUC 0.65-0.79) (Table 2). Most models19, 20, 21, 22, 23, 24 reported other measures including sensitivity and specificity (Supplemental Table 6, available online). Only the UVA score reported a subgroup analysis²⁰ (Supplemental Table 7, available online).

The studies differed in their handling of missing values (Table 2 and Supplemental Table 8 [available online]). After excluding participants with missing data regarding human immunodeficiency virus (HIV; n = 2171) and mortality outcomes (n = 8), MEWS, qSOFA, and UVA score in Moore (2017)²⁰ imputed missing values. The model by Soffer (2020)²³ used gradient boosting algorithms for missing values. Other models either excluded^19,21,22 or did not provide information¹⁸ on handling of missing values.

Prediction Models for General Medical Patients with Infection

Table 3 outlines the characteristics of infection models. The CHARM score²⁴ and UVA score²⁰ were the only novel models developed and internally validated in general medical patients with infection. Fabbian (2018)²⁵ externally validated the modified Elixhauser Index, which they reported previously (Fabbian, 2017¹⁸). Chen 2017²⁴ externally validated 4 novel models: (i) Predisposition, Infection/Insult, Response and Organ dysfunction (PIRO); (ii) Confusion, blood Urea nitrogen, Respiratory rate, Blood pressure, and age 65 (CURB-65); (iii) Mortality in Emergency Department Sepsis (MEDS) score; and (iv) Systemic Inflammatory Response Syndrome (SIRS). None of the models were developed and externally validated in an independent dataset within the same study. Predictors for the novel models (CHARM and UVA) were selected using multivariable analysis. Vital signs were used as predictors in 5 models, while comorbidities were used in 3 models (Table 3 and Supplemental Table 3 [available online]). Predictors for all models were captured on admission, except for the modified Elixhauser Index,²⁵ which captured ICD codes from discharge summaries. The CHARM score²⁴ converted continuous predictors into categorical predictors (Supplemental Table 5, available online) and used AIC to select predictors. All models in Chen (2017)²⁴ excluded missing values. Calibration was only reported for the CHARM score and had adequate calibration based on a calibration plot and Hosmer-Lemeshow test. The CHARM score had the highest AUC of 0.77 (95% CI, 0.75-0.79) in the development cohort and 0.76 (95% CI, 0.75-0.77) in the validation cohort, while SIRS had a modest AUC of 0.58 (95% CI, 0.56-0.61). Other models had an AUC ranging from 0.68 to 0.75 (Table 3). Alternative performance measures (Supplemental Table 9, available online) and subgroup analysis (Supplemental Table 7, available online) were also reported.

Table 3.

Prediction Models for General Medical Patients with Infection (Infection Models)

Characteristic	CHARM24	PIRO24	CURB-6524	MEDS24	SIRS24	Modified Elixhauser Index25	UVA20
Purpose of analysis	Model development/internal validation	External validation/model comparison	External validation/model comparison	External validation/model comparison	External validation/model comparison	External validation	Model development/internal validation
Timing of predictor assessment	On admission	On admission	On admission	On admission	On admission	At diagnosis of comorbidity (ICD-9)	On admission
Number of candidate predictors	62	Not applicable	Not applicable	Not applicable	Not applicable	Not applicable	13
Predictors in final model (no.)	Absence of chills, anemia, hypothermia, malignancy, RDW (5)	Age, bands >5%, BUN, COPD, HR, lactate, liver disease, malignancy, nursing home residence, other infection, platelet count, pneumonia, respiratory failure/hypoxemia, RR, SBP, skin/soft tissue infection (16)*	Age, BUN, confusion, RR, SBP or DBP (5)*	Age, altered mental status, bands >5%, lower respiratory tract infections, nursing home resident, platelet count, respiratory disease, septic shock, terminal disease (9)*	HR, RR or PaCO2, temperature, WBC count or bands >10% (4)*	Age, coagulopathy, congestive heart failure, fluid and electrolyte disorders, gender, ischemic heart disease, liver disease, lymphoma, metastatic cancer, other neurological disorders, renal failure, solid tumor without metastasis, weight loss (13)	HIV serostatus, HR, level of consciousness (GCS), RR, SBP, SpO2, temperature (7)
Method for handling missing data	Variables with more than 5% missing values were not considered candidate predictors	Not applicable	Not applicable	Not applicable	Not applicable	Not provided	k-nearest neighbors imputation†
Method for prediction model development	Logistic regression	Not applicable	Not applicable	Not applicable	Not applicable	Not applicable	Logistic regression; decision trees and linear regression
Method for selection of predictors during multivariable analysis	AIC	Not applicable	Not applicable	Not applicable	Not applicable	Not applicable	BIC
Calibration method (result)	Calibration plot and Hosmer-Lemeshow test (P = .42)	None	None	None	None	None	None
AUC value§	Development: 0.77 (0.75-0.79); Validation: 0.76 (0.75-0.77)	0.74 (0.72-0.77)	0.68 (0.65-0.70)	0.67 (0.64-0.69)	0.58 (0.56-0.61)	0.72 (0.71-0.74)	0.75 (0.72-0.77)
Type of validation	Internal	External	External	External	External	External	Internal
Method of validation	Temporal data split	Different time, area, and investigators	Different time, area, and investigators	Different time, area, and investigators	Different time, area, and investigators	Different time and area	10-Fold cross-validation

^⁎Predictors of PIRO, CURB-65, MEDS, SIRS were extracted from references in Chen (2017).²⁴

^†Variables with more than 50% missing values were excluded. Hospital mortality and HIV serostatus were not imputed.

^§Rounded to 2 decimal places.

Abbreviations listed in Supplemental Table 13 (available online).

Quality and Applicability Assessment

Based on PROBAST criteria, all general models had an overall high risk of bias (Supplemental Figure 1A, available online). All general models had low risk of bias for the Participant domain, indicating the appropriateness of the populations for our study. For the Predictors domain, 4 of 10 general models had a low risk of bias. Common reasons for high risk of bias were the timing of predictor measurement and availability of predictors on admission, in particular for models requiring laboratory data^19,20,22,23 or ICD codes^18,21, 22, 23 (Supplemental Table 10, available online). For the Outcomes domain, 5 of 10 general models had an unclear risk of bias, attributed to unclear duration from predictor measurement to hospital mortality. All general models had high risk of bias for the Analysis domain because none of the studies reported both model calibration and discrimination. MEWS,²⁰ qSOFA,²⁰ NEWS,¹⁹ and the model by Soffer (2020)²³ had low concern for applicability to predict hospital mortality for all general medical patients (Supplemental Figure 1C, available online).

All infection models had an overall high risk of bias and a low risk of bias for the Participant domain (Supplemental Figure 1B and Supplemental Table 10, available online). For the Predictors domain, 5 of 7 infection models had low risk of bias. For the Outcomes domain, all infection models, except for the UVA score,²⁰ had low risk of bias. All infection models had a high risk of bias for the Analysis domain. Only the CHARM score²⁴ provided sufficient information on model calibration and discrimination and appropriately used univariate models to select predictors (Table 3). CHARM,²⁴ PIRO,²⁴ CURB-65,²⁴ MEDS,²⁴ and SIRS²⁴ scores had low concern for applicability as hospital mortality predictors for patients with infection in the medical ward (Supplemental Figure 1D, available online).

Discussion

In this systematic review of 8 studies on 280,793 patients from 10 countries, we identified 15 risk prediction models for hospital mortality in general medical patients. Of these models, 5 were novel, 2 were adapted, and 8 were external validation of preexisting models. For general models, NEWS and the model by Soffer (2020) had an optimal balance of discrimination, bias, and applicability. For infection models, the CHARM and PIRO scores had an optimal balance. Overall, there was sparse data on risk prediction and the available models had a high risk of bias attributed, in part, to suboptimal calibration and handling of missing values. The novel models had higher discrimination compared to the adapted and validated models. To our knowledge, there is no comparable synthesis of studies on risk prediction models, and our findings identify the need for studies on risk prediction models for hospital mortality in general medical patients.

EWS models, frequently incorporated as track and trigger systems, are simple point-based systems ideal for predicting short-term mortality (eg, 24 or 48 hours) in ICU and general medical patients.^26,27 Compared with general medical wards, there was more information on predicting mortality in the ICU, which has informed studies in non-ICU settings.28, 29, 30, 31 Similar to ICU settings, early and prompt recognition of general medical patients at high risk for mortality is an important first step in preventing mortality.³² Furthermore, general medical patients frequently have complex acute and chronic illnesses, and early prediction of nonpreventable deaths may facilitate their transition to palliative or hospice care. Simple EWS models (eg, MEWS, NEWS) have variable performance^27,33 and are well suited for patients with infection, who typically experience frequent change in vital signs reflecting potential clinical deterioration. A retrospective study showed that EWS models outperformed other models in patients with infections but were less promising in patients without infection.³³ Given the dependence on vital signs, EWS models are less suitable for predicting long-term hospital mortality (eg, 7 days).^26,34 Machine learning models^32,35 have been explored as alternatives to EWS models and have the advantage of automated calculations that reduce human error, improve integration in electronic health record (EHR) systems, and generate fewer false alarms. Mortality prediction models have helped to identify high-risk patients in non-ICU settings. A recent randomized controlled clinical trial showed that automated detection and monitoring of clinical deterioration in hospitalized adults was associated with a 16% reduction in 30-day mortality following an alert.³⁶

In contrast to EWS models, comorbidity models including the Charlson Comorbidity Index³⁷ and the modified Elixhauser Comorbidity Index³⁸ may be better for predicting longer-term (ie, more than 48 h) hospital mortality.39, 40, 41 Comorbidity models have applications beyond predicting hospital mortality and have been used to predict healthcare expenditure, tailor treatment, and control for confounders in epidemiologic studies.^42,43 However, their benefit to healthcare providers may be limited because most comorbidity models require knowledge of comorbidities on hospital admission, which may not be readily available for many patients. Furthermore, comorbidity models do not detect rapid deterioration in clinical status and may have high concern for applicability in hospitals with limited use of EHRs.

Real-world performance and usability of prediction models may be affected by many factors. The number of predictors in a model does not necessarily correlate with its predictive performance, as summarized in a previous systematic review.⁴⁴ In the current study, 6 of 10 general models and 2 of 7 infection models had 10 or more predictors. The number of predictors in a model can affect model performance in real-world settings, particularly in hospitals with limited resources.⁴⁵ Implementing models in hospitals with understaffed wards, scarce monitoring systems, and limited technology may increase the burden of manual labor with minimal benefit on mortality.45, 46, 47 Further, some predictors may be relevant in a specific world region, improving its performance. As Moore (2017)²⁰ illustrated in sub-Saharan Africa, adding HIV serology to vital signs resulted in better discrimination of the UVA score over other EWS scores.

During screening, we excluded articles that focused on specific time horizons for hospital mortality (eg, 72-hour or 7-day mortality). While a time horizon of a few days may be appropriate for EWS models,²⁶ mortality at any point during hospitalization may be more relevant to guide overall hospital care that integrates biopsychosocial factors and patients’ preferences.³³ Some suggest that 30-day postdischarge mortality, rather than hospital mortality, is a better indicator of hospital performance.^48,49 However, hospital mortality may be more relevant, in places with limited ability to provide, influence, or monitor postdischarge care.^50,51 We excluded models such as the eCART score, which did not meet study selection criteria.^52,53 However, the eCART score, based on medical and surgical patients, integrated vital signs and laboratory data to predict outcomes in the subsequent 24 hours. eCART and other automated scores that guide short-term care may have high clinical uptake.

Fourteen of 15 models were developed or validated in single-site studies. The most common method of internal validation in the included models was temporal data split (n = 4 models); however, cross-validation and bootstrapping may be preferred over random and temporal data split, particularly in smaller datasets.⁵⁴ None of the novel or modified models in general medical patients were externally validated, and preexisting models were only externally validated in 1 country. To ensure model stability and generalizability, Riley et al.⁵⁵ recommend externally validating prediction models using big datasets from individual participant data metaanalysis or EHRs that include participants from different regions and providing subgroup analysis to study geographic heterogeneity in model performance. Validating these models in regions with different demographics, patient case-mix, staffing volumes, and technologic capability may improve generalizability and guide care in hospitals that have limited ability for research and evaluation.^55,56 Additionally, in the current study, heterogeneity in population characteristics precluded direct comparison of models across studies. For instance, the population was younger in the Moore (2017) study (median age: 36 years; interquartile range: 27-49 years) compared with other studies. Thus, the generalizability of these models to other hospitals, countries, and world regions is unknown. However, studies based on local data have the ability to calibrate existing models to local demographics and patient case-mix, leading to informative models that can guide mortality reduction interventions.^55,57,58

As highlighted, no single model will apply to all populations and healthcare settings. Thus, the resources needed to implement a prediction model would depend on the model. Machine learning models are of emerging popularity in many high-income countries. The popularity of these models stems from their high performance, abundance of EHR data, advanced technological expertise, and capability to evaluate and enhance them. Due to the simplicity of their implementation in resource-limited settings, early warning scores may be more feasible. Ultimately, these models may aid in timely escalation of care and/or transition to palliative or hospice care. Incorporating models into hospital practice will require an impact analysis on hospital mortality.⁵⁹

Limitations and Strengths

The study has potential limitations. We restricted our search to English-language articles, which may influence the generalizability of our results, but not necessarily result in systemic bias.⁶⁰ During screening, we excluded articles that did not explicitly distinguish general medical patients from others (eg, surgical patients) and inadvertently may have excluded relevant articles. To mitigate this, we used a broad search strategy for primary screening with independent review by two investigators.⁶¹ We excluded articles on COVID-19 infection because risk factors and treatment evolved during the pandemic, thereby influencing the risk of mortality. Therefore, our findings may not be generalizable to patients with COVID-19. Studies in our analysis were conducted in the prepandemic period; the COVID-19 pandemic has resulted in a global excess mortality of ∼3 million deaths in 2020⁶² and hospital mortality rates in some countries increased for non-COVID-19 patients.⁶³ Therefore, the performance of prediction models for non-COVID-19 patients may be different during the pandemic. The study has several strengths. Our review focused on mortality in contemporary cohorts and reflected current practice and advances in model prediction, machine learning, and artificial intelligence. We used CHARMS and PROBAST, which are rigorous tools for data extraction and quality assessment.¹⁷ We included studies with >500 mortality events resulting in ∼50 events per predictor variable, well above the recommendation for prediction models.^26,64

Conclusions

In this systematic review of 8 studies, 14 of 15 risk prediction models for hospital mortality were from single-site studies, which have high local relevance but unknown generalizability. All models had a high risk of bias and differed in model covariates, applicability, and discrimination. There is a need for rigorous models to predict mortality in general medical patients. Rather than disease-specific models, unified prediction models for general medical patients calibrated to the local determinants of hospital care, including patient case-mix, technologic availability, and workforce capability, may be better incorporated into clinical decision support tools and facilitate the delivery of safer hospital care.

Funding/Support

S.B.D. was supported by the National Institutes of Health/National Institute on Minority Health and Health Disparities (NIH K23 MD016230) and the Robert and Elizabeth Strickland Career Development Award, Mayo Clinic, Rochester, Minn, USA.

Declaration of Competing Interest

None of the authors has any conflict of interest.