Prediction models for hospital readmissions in patients with heart disease: a systematic review and meta-analysis

Abstract

Objective

To describe the discrimination and calibration of clinical prediction models, identify characteristics that contribute to better predictions and investigate predictors that are associated with unplanned hospital readmissions.

Design

Systematic review and meta-analysis.

Data source

Medline, EMBASE, ICTPR (for study protocols) and Web of Science (for conference proceedings) were searched up to 25 August 2020.

Eligibility criteria for selecting studies

Studies were eligible if they reported on (1) hospitalised adult patients with acute heart disease; (2) a clinical presentation of prediction models with c-statistic; (3) unplanned hospital readmission within 6 months.

Primary and secondary outcome measures

Model discrimination for unplanned hospital readmission within 6 months measured using concordance (c) statistics and model calibration. Meta-regression and subgroup analyses were performed to investigate predefined sources of heterogeneity. Outcome measures from models reported in multiple independent cohorts and similarly defined risk predictors were pooled.

Results

Sixty studies describing 81 models were included: 43 models were newly developed, and 38 were externally validated. Included populations were mainly patients with heart failure (HF) (n=29). The average age ranged between 56.5 and 84 years. The incidence of readmission ranged from 3% to 43%. Risk of bias (RoB) was high in almost all studies. The c-statistic was <0.7 in 72 models, between 0.7 and 0.8 in 16 models and >0.8 in 5 models. The study population, data source and number of predictors were significant moderators for the discrimination. Calibration was reported for 27 models. Only the GRACE (Global Registration of Acute Coronary Events) score had adequate discrimination in independent cohorts (0.78, 95% CI 0.63 to 0.86). Eighteen predictors were pooled.

Conclusion

Some promising models require updating and validation before use in clinical practice. The lack of independent validation studies, high RoB and low consistency in measured predictors limit their applicability.

PROSPERO registration number

CRD42020159839.

Strengths and limitations of this study.

• Largest investigation of unplanned hospital readmission risk to date, including 81 unique prediction models in the systematic review.

• Independent and standardised procedures for study selection, data collection and risk of bias (RoB) assessment.

• High RoB in current prediction models and unexplained heterogeneity between models limit recommendations for using prediction model in clinical practice.

Introduction

Hospital readmissions in patients with acute heart disease are associated with a high burden on patients, healthcare and costs.¹ The identification of high-risk hospitalised patients is important to provide timely interventions. Prediction models guide healthcare providers in daily practice to assess patients’ probability of readmission within a certain time frame and include candidate variables identified by clinical perspectives, literature or data-driven approaches, for example, using machine learning techniques.² Data are often collected from observational cohorts of intervention studies and subsequently analysed to examine what set of predictors best predict the risk of readmission. The clinical applicability of risk prediction models in daily practice is currently limited. Statistical models are often not presented in a clinically useful way or models based on administrative data are considered.³ These models therefore cannot be readily used in daily practice. In addition, prediction models are often developed for a very specific population, which asks from clinicians to be familiar with several models. Furthermore, patients may belong to multiple populations because of cardiac comorbidities. Numerous systematic reviews have previously investigated the prediction of unplanned hospital readmissions in several populations.^3–12 While some have included hospitalised patients in general,^{11 12} others have focused specifically on patients with heart failure (HF)^{4–8 10} or acute myocardial infarction (AMI).^{3 9} The conclusion is generally the same, the discrimination is poor to adequate, and there is little consistency in the type of predictors included in the models.

We believe that the state of the art on risk prediction can be improved if more knowledge is available on the performance of clinical risk prediction models and risk predictors across different populations of patients with heart disease. Although heterogeneity in models and predictors is often considered as a limitation, it can inform effect moderators on how predictions can be improved.¹³ For example, perhaps we can identify predictors who demonstrate a consistent association with hospital readmission regardless of the underlying disease. If this can be identified, a more general prediction model could be developed that is relevant for the heterogeneous group of patients on cardiac care units. This might contribute to the early recognition and onset of preventive interventions in patients with heart disease at risk of readmission.

We therefore performed a systematic review and meta-analysis on clinical risk prediction models for the outcome unplanned hospital readmission in patients hospitalised for acute heart disease. Our aims were to describe the discrimination and calibration of clinical prediction models, to identify characteristics that contribute to better predictions, and to investigate predictors that are consistently associated with hospital readmissions.

Methods

A protocol was registered in PROSPERO (registration number: CRD42020159839). The results are reported following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement.¹⁴

Eligibility criteria

Studies were eligible if (1) the study population included hospitalised adult patients with (symptoms of) heart disease; (2) a prediction model with c-statistic was reported; (3) a clinically useful presentation of the model with risk factors was reported; (4) the outcome was unplanned hospital readmissions within 6 months; (5) the study design was appropriate, that is, (nested) case–control study (prospective and retrospective) cohort study, database and registry study, or secondary analysis of a trial; (6) they were reported in English.

Information sources

A search strategy was designed with an information specialist (PROSPERO protocol and online supplemental text 1). We searched the Medline, EMBASE, WHO ICTPR search portal (for study protocols) and Web of Science (for conference proceedings) databases up to 25 August 2020 without any restrictions for eligible studies. We searched for full-text manuscripts of the identified protocols. After selecting the full-text manuscripts, we screened references lists and prospective citations (using Google Scholar) for additional eligible studies.

Study selection

Three reviewers were involved in the study selection process. Each reviewer independently screened two-thirds of the titles, abstracts and full-text articles of potentially relevant references identified in the literature search. Disagreements were resolved through consensus. Sixteen authors were contacted and six delivered data for readmission when a composite outcome was used. Two authors were also contacted when data were reported combining multiple patient populations. However, no additional data were provided for the population with heart disease and these studies were excluded.

Data extraction

Data extraction was performed based on the ‘Critical Appraisal and Data Extraction for Systematic Reviews’ of prediction modelling studies checklist using standardised forms in the Distiller Systematic Review Software (see online supplemental text 2 for the data items).¹⁵ The checklist includes items on 11 relevant domains, including source of data, participants, outcomes, candidate predictors, sample size, missing data, model development, model performance, model evaluation, results and interpretation. One reviewer collected the data and the second reviewer verified the extracted data. Disagreements were resolved through consensus. Eight authors were contacted and two delivered data to resolve uncertainties or missing data.

Risk of bias

The Prediction model Risk Of Bias ASsessment Tool (PROBAST) tool¹⁶ was used to assess the risk of bias (RoB) for four ‘quality’ domains, that is, the participants, predictors, outcome and analysis for each model. One author assessed the RoB as low, high or unclear, and the second author verified the extracted data and RoB conclusion. Disagreements were resolved through consensus. In addition, the applicability of the included studies based on our research question was assessed for three domains, that is, participants, predictors and outcome domains and rated as low concerns, high concerns or uncertain concerns regarding applicability.

Summary measures

The discrimination of the prediction models was described using the concordance (c)-statistic. Missing SEs were derived from the sample data.¹⁷ The calibration was described using the number of observed and expected events, the calibration slope, calibration in large or the Hosmer-Lemeshow test. A definition of the commonly used measures is described in box 1.

Box 1. Definitions of commonly used measures.

Discrimination:

Refers to the ability of a prediction model to discriminate between a patient with and without the outcome, for example, readmission.

C-statistic:

Is a measure of discrimination. For binary outcomes, the c-statistic is equivalent to the area under the curve: 1 indicates perfect discrimination, and 0.5 indicates that the models does not perform better than chance. Harrell’s c-statistic is often used in survival models.

Calibration:

Refers to the agreement between the predicted and the observed probability (or the outcome value for linear models). Calibration is expressed using different measures, for example, calibration slope, calibration in large, Hosmer-Lemeshow test.

Calibration slope:

The slope should be 1, a value <1 indicates extreme predictions, and a value of >1 indicates to moderate predictions.

Calibration in large:

The value should be 0, a negative value indicates overestimation of the prediction, and a positive value indicates underestimation of the prediction.

Hosmer-Lemeshow test:

This is a goodness-of-fit test for binary outcomes. A significant p value, usually <0.05, indicates poor goodness-of-fit.

Derivation/development cohort:

A cohort of patients that is used to estimate the predictor values that are used in a prediction model to estimate a patient’s probability for an outcome.

Validation cohort:

A cohort of patients that is used to evaluate how well the developed model performs (in terms of discrimination and calibration).

Internal validation:

Estimates how well the performance of a model will be reproduced in the target population. Several techniques can be used, for example, random-split sample, cross-validation and bootstrapping techniques.

External validation:

Evaluates how well a model performs in a new sample and can consist of temporal validation (sample contains more recently treated patients), geographical validation (sample is from a different centre) of a fully independent validation (validation by an independent team).

The association between risk predictors and hospital readmission was described using regression coefficients. Missing SEs for the coefficients were considered missing completely at random and were not imputed. A complete case analysis was performed.

Synthesis of results and analyses

Meta-analyses using random-effects models, with the Hartung-Knapp modification, were performed to describe the distribution of the between-study variance of the different prediction models and their predictors. Because we considered that there would be substantial heterogeneity, conclusions were not based on the precision of the pooled estimates.

The c-statistic from each model was pooled and a meta-regression was performed to investigate the moderation effect of age and the number of predictors on the discrimination. A subgroup analysis was performed to investigate the moderation effect of the different patient populations, design, outcome definition and endpoint. The c-statistic of the validated model was used if available; otherwise, the c-statistic from the development phase was used.

The c-statistics of specific prediction models that were evaluated in multiple studies were pooled for the endpoint 30-day follow-up.

Coefficients of predictors that were similarly defined in at least five studies were pooled for the endpoint 30-day follow-up. The patient populations were defined as subgroups to explore consistency and heterogeneity (I², tau) in the effect estimates.

Analyses were performed using the ‘metan’ package in STATA V.15 IC and the ‘metamisc package’ in Rstudio.

Public and patient involvement

Because of the design of the study and because we did not collect primary date, we did not involve patients or the public in the design, conduct or reporting of our research.

Results

A total of 8588 abstracts were reviewed and 60 studies describing 81 separate models were included (figure 1). Table 1 provides an overview of the included studies and models, which were published between 2001 and 2020. The majority of the studies (n=40) was performed in the USA. The data sources used were mostly retrospective cohort studies (n=15), hospital databases (n=13) and registries (n=13). Included populations were mainly patients with HF(n=29), surgical patients (n=14) and patients with an AMI or acute coronary syndrome (n=10). The average age was between 56.5 and 84 years. The sample size of development cohorts ranged from 182 to 193 899 patients and of the validation cohorts between 104 and 321 088 patients. The outcome of interest was mostly all-cause readmission (n=41) and measured on 30 days (n=55). The incidence of readmission per study ranged from 3% to 43%.

Figure 1.

Flowchart. In total, 8592 records were screened and 60 studies with 81 prediction models were included.

Table 1.

Study characteristics

Study	Model	Data source	Development	Validation	Sample size		Population	Average age	Outcome	Readmission (%)
					Dev	Val				Dev	Val
Moretti et al57	EuroHeart PCI score	Hospital database	NA	Ext	–	1192	ACS	71 (7)	30d		4.7
Asche et al46	NR	Retrospective cohort	Yes	Split	2446	612	AMI	65 (15)	30d	8.9
Cediel et al58	TARRACO Risk score	Retrospective cohort	Yes	No	611	401	AMI type 2, ischaemia	D: 78 (17) V: 60 (21)	30d	2.6
		Retrospective cohort	Yes	No	611	401	AMI type 2, ischaemia	D: 78 (17) V: 60 (21)	180d	7.9
Chotechuang et al36	GRACE	Retrospective cohort	NA	Ext	–	152	AMI	60.5 (6.3)	30d		5.3
	GRACE	Retrospective cohort	NA	Ext	–	152	AMI	60.5 (6.3)	180d		9.2
Hilbert et al59	AMI decision tree	Registry	Yes	Ext	10 848	10 701	AMI	NR	30d	20.6	19.7
Dodson et al18	SILVER-AMI 30-day readmission calculator	Prospective cohort	Yes	Split	2004	1002	AMI	81.5 (5.0)	30d	18.2
Kini et al60	NR	Registry	Yes	Split	60 742	26 107	AMI	76.5 (8.0)	90d	27.5
Nguyen et al19	AMI READMITS score	Retrospective cohort	Yes	Split	661	165	AMI	65.5 (12.8)	30d	13
	Full-stay AMI model	Retrospective cohort	Yes	Split	661	165	AMI	65.5 (12.8)	30d	13
	CMS AMI administrative model	Retrospective cohort	NA	Ext	–	826	AMI	65.5 (12.8)	30d		13
Krumholz et al20	CMS AMI administrative model	Registry	Yes	Split, Ext	100 465	321 088	AMI	78.7 (8.0)	30d	18.9	20.0 (Ext) NR (split)
	CMS AMI medical model	Registry	Yes	Split	130 944	130 944	AMI	76.2 (7.3)	30d	20
Rana et al33	Elixhauser index	Hospital database	NA	Ext	–	1660	AMI	67.9	30d		6.3
	HOSPITAL score	Hospital database	NA	Ext	–	1660	AMI	67.9	30d		6.3
Atzema et al47	AFTER Part 2 scoring system	Retrospective cohort	Yes	Split	2343	1167	Arrhythmia, AF	D: 68.6 (14.7) V: 68.3 (15.1)	30d	7	7.6
Lahewala et al40	CHADS2	Administrative	NA	Ext	–	116 450	Arrhythmia, AF	<75	30d		15.8
	CHADS2	Administrative	NA	Ext	–	116 450	Arrhythmia, AF	<75	90d		25.1
	CHA2DS-VASc	Administrative	NA	Ext	–	116 450	Arrhythmia, AF	65–74	30d		15.8
	CHA2DS-VASc	Administrative	NA	Ext	–	116 450	Arrhythmia, AF	65–74	90d		25.1
Benuzillo et al61	CRSS	Hospital database	Yes	Boot, Ext	2589	896 (Ext) 500 (Boot)	CABG	66.7 (9.9)	30d	9.1	8.2 (Ext) 9.1 (Boot)
Deo et al62	30-day CABG readmission calculator	Administrative	Yes	Boot	155 054	1000	CABG	65.4 (10.4)	30d	12.5
Engoren et al55	NR	Hospital database	Yes	Split	2644	2711	CABG	NR	30d	7.6	8
Lancey et al63	NR	Registry	Yes	Split	2341	2520	CABG	64.5 (10.5)	30d	8.8	9.5
Rosenblum et al41	The STS PROM score	Hospital database	NA	Ext	–	21 719	CABG	63.5 (10.7)	30d		9.3
Zitser-Gurevich et al64	NR	Prospective cohort	Yes	Split	2266.5	2266.5	CABG	65–74	30d	13.3
	NR	Prospective cohort	Yes	Split	2266.5	2266.5	CABG	65–74	100d	24.1
Zywot et al42	CABG risk scale	Administrative	Yes	Ext	126 519	94 318	CABG	D: 70–74 V: 70–74	30d	23	21
Ahmad et al21	CMS HF administrative model	Prospective cohort	NA	Ext	–	183	HF	61 (18)	30d		22.4
Amarasingham et al22	ADHERE	Hospital database	NA	Ext	–	1372	HF	56.5	30d		24.1
	CMS HF administrative model	Hospital database	NA	Ext	–	1372	HF	56.5	30d		24.1
	Tabak mortality score	Hospital database	NA	Ext	–	1372	HF	56.5	30d		24.1
Au et al23	Administrative claims model: HF 30-day mortality	Administrative	NA	Ext	59 652	59 652	HF	75.8 (12.7)	30d		15.9
	Charlson Comorbidity Score	Administrative	NA	Ext	59 652	59 652	HF	75.8 (12.7)	30d		15.9
	CMS HF administrative model	Administrative	NA	Ext	59 652	59 652	HF	75.8 (12.7)	30d		15.9
	LACE	Administrative	NA	Ext	59 652	59 652	HF	75.8 (12.7)	30d		15.9
Bardhan et al65	NR	Hospital database	Yes	No	40 983	–	HF	69.2 (15.7)	30d	7
Betihavas et al66	NR	RCT secondary analysis	Yes	Boot	280	200	HF	74 (64–81)	28d	18
Cox et al24	CMS HF administrative model	Hospital database	No	Ext	–	1454	HF	75 (12)	30d		21.5
	CMS HF medical model	Hospital database	No	Ext	–	1454	HF	75 (12)	30d		21.5
Delgado et al67	15-day CV readmission risk score	Prospective cohort	Yes	Boot	1831	500	HF	72.4 (12.1)	15d	7.1
	30-day CV readmission risk score	Prospective cohort	Yes	Boot	1831	500	HF	72.4 (12.1)	30d	13.9
Formiga et al30	CMS HF medical model	Hospital database	NA	Ext	–	719	HF	78.1 (9)	30d		7.6
	CMS HF medical model	Hospital database	NA	Ext	–	719	HF	78.1 (9)	90d		14.4
Frizzell et al25	CMS HF administrative model	Registry	NA	External	–	56 477	HF	80 (2)	30d		21.2
Hammill et al26	CMS HF administrative model	Registry	NA	Ext	–	24 163	HF	81	30d		21.9
Hilbert et al59	HF decision tree	Registry	Yes	Ext	39 682	38 409	HF	NR	30d	25.5	25.2
Hummel et al31	CMS HF medical model	Prospective cohort	NA	Ext	–	1807	HF	79.8 (7.6)	30d		27
Huynh et al48	NR	Prospective cohort	Yes	Ext	430	1046	HF	D: 75 (19) V: 67 (17)	30d	21	24
	NR	Prospective cohort	Yes	Ext	430	1046	HF	D: 75 (19) V: 67 (17)	90d	43	42
Ibrahim et al34	HOSPITAL score	Retrospective cohort	NA	Ext	–	692	HfpEF	68.3 (11.8)	30d		27.3
	LACE/LACE+ index	Retrospective cohort	NA	Ext	–	692	HfpEF	68.3 (11.8)	30d		27.3
Keenan et al27	CMS HF administrative model	Registry	Yes	Split, Ext.	28 319	845 291	HF	79.9 (7.8)	30d	23.6	23.7 (Ext) NR (Split)
	CMS HF medical model	Registry	Yes	Split, Ext.	64 329	64 329	HF	75–84	30d	23.7
Kitamura et al53	FIM	Retrospective cohort	NA	Ext	–	113	HF	80.5 (6.7)	90d		20.4
Leong et al68	30-day HF readmission risk score	Retrospective cohort	Yes	Split	888	587	HF	D: 70.0 (12.7) V: 69.1 (12.8)	30d	9.9
Li et al49	NR	Retrospective cohort	Yes	Split	51 783	25 887	HF	D: 84 (12) V: 84 (11)	30d	24.2
Lim et al69	NR	Registry	Yes	No	4566	–	HF	70.5 (12.0)	30d	6.6 (car) 13 (all)
Reed et al28	AH model	Administrative	Yes	Split	NR	NR	HF	NR	30d	NR
	CMS HF administrative model	Administrative	NA	Split	–	NR	HF	NR	30d		NR
	Hasan	Administrative	NA	Split	–	NR	HF	NR	30d		NR
	LACE	Administrative	NA	Split	–	NR	HF	NR	30d		NR
	PARR-30	Administrative	NA	Split	–	NR	HF	NR	30d		NR
Salah et al70	ELAN-HF score	Prospective cohort secondary analysis	Yes	No	1301	–	HF	74 (16)	180d	36.1
Sudhakar et al32	CMS HF medical model	Hospital database	NA	Ext	–	1046	HF	65.2 (16.6)	30d		35.3
Tan et al71	NR	Hospital database	Yes	Split	246	104	HF	D: 67.7 (12.3) V: 69.0 (12.9)	90d	24.5	11.7
Wang et al72	NR	Hospital database	Yes	No	4548	–	HF	68.5 (27.6)	30d	25.1
Wang et al38	LACE	Retrospective cohort	NA	Ext	–	253	HF	56.6 (11.5)	30d		24.5
Yazdan-Ashoori et al29	CMS HF administrative model	Prospective cohort	NA	Ext	–	378	HF	73.1 (13.1)	30d		26
	LACE	Prospective cohort	NA	Ext	–	378	HF	73.1 (13.1)	30d		26
Disdier Moulder et al73	NR	Prospective cohort	Yes	No	258		HF, ACS, NR	70.5 (23)	30d	17
	NR	Prospective cohort	Yes	No	258		HF, ACS, NR	70.5 (23)	180d	38
Raposeiras-Roubín et al37	GRACE	Retrospective cohort	NA	Ext	–	4229	HF, ACS	68.2 (18.7)	30d		2.6
Burke et al35	HOSPITAL score	Retrospective cohort	NA	Ext	–	HF: 3189 AMI: 767	HF, AMI	65.8 (16.8)	30d		HF: 18.2 AMI: 17.4
Minges et al74	NR	Registry	Yes	Split	193 899	194 179	HF, PCI	65+	30d	11.4
Pack et al75	NR	Administrative	Yes	Split	30 826	7706	HVD	64.9 (12.2)	90d	12.8
Oliver-McNeil et al76	ICD readmission-risk score	Registry	Update	Ext	182	–	ICD	69 (11)	30d		17.6
Wasfy et al52	Pre-PCI model	Registry	Yes	Split	24 052	12 008	NR	64.8 (12.5)	30d	10.4
Barnett et al77	NR	Registry	Update	Ext	19 964	19 964	Surgical	65.3 (12.4)	30d	11.4
Brown et al43	STS augmented clinical model	Prospective cohort	Update	Boot	1046	NR	Surgical	65.4 (9.8)	30d	NR
	STS 30-day readmission model	Prospective cohort	NA	Ext	–	1194	Surgical	73.3 (10.1)	30d		NR
Espinoza et al78	30-day readmission score after cardiac surgery	Retrospective cohort	Yes	Split	2529	2567	Surgical	65.1 (11.5)	30d	11.9
Ferraris et al54	READMIT	Prospective cohort	Yes		2574		Surgical	63 (11)	30d	9.8
Kilic et al79	NR	Retrospective cohort	Yes	Split	3898	1295	Surgical	D:61.9 (14.7) V: 61.6 (15.1)	30d	10	11
Stuebe et al80	NR	Hospital database	Yes	No	4800		Surgical	60–69	30d	12
Tam et al44	NR	Retrospective cohort	Yes	Boot	63 336	NR	Surgical	66.2 (10.7)	30d	11.3
Khera et al45	TAVR 30-Day readmission risk model	Administrative	Yes	Boots, Ext	39 305	40 (Boot) 885 (Ext)	TAVR	D: 81.3 V: 81.7	30d	16.2	16.2 (Boot) 18.9 (Ext)
Sanchez et al50	NR	Registry	Yes	Split	6903	3442	TAVR	D: 81.1 (7.9) V: 81.3 (7.9)	30d	9.8	10.7

Age is reported as mean (SD); median (IQR) or average age as reported in the study.

ACS, acute coronary syndrome; ADHERE, Acute Decompensated Heart Failure Registry; AF, atrial fibrillation; AH, Adventist Health Off-the-shelf model; AMI, acute myocardial infarction; Boot, bootstrapping; CABG, coronary artery bypass grafting; Car, cardiac-related; CHA2DS2-VASc, congestive heart failure, hypertension, age ≥ 75 years, diabetes mellitus, stroke or transient ischemic attack (TIA), vascular disease, age 65 to 74 years, sex category; CMS, Centers for Medicare and Medicaid Services; CRSS, CABG Readmission Risk Score; d, days; Dev, development; ELAN-HF, European Collaboration on Acute Decompensated Heart Failure; Ext, external validation; FIM, motor and cognitive Functional Independence Measure; GRACE, Global Registration of Acute Coronary Events; HF, heart failure; HFpEF, heart failure with preserved ejection fraction; HVD, heart valve disease; ICD, implantable cardioverter defibrillator; LACE, Length of stay, acuity of the Admission, Comorbidity of the patient and Emergency department use in the duration of 6 months before admission; NA, not applicable; NR, not reported; PARR-30, Patients at Risk of Re-admission within 30 days; PCI, percutaneous coronary intervention; READMITS, Renal Function, Elevated Brain Natriuretic Peptide, Age, Diabetes Mellitus, Nonmale Sex, Intervention with Timely Percutaneous Coronary Intervention, and Low Systolic Blood Pressure; SILVER-AMI, Comprehensive Evaluation of Risk Factors in Older Patients with AMI; Split, random split; STS PROM, Society of Thoracic Surgeons Predicted Risk of Mortality; TARRACO, Troponin Assessment for Risk stRatification of patients without Acute COronary atherothrombosis; TAVR, transcatheter aortic valve replacement; Val, validation.

Risk of bias

Figure 2 summarises the RoB and applicability assessment (online supplemental table 1A). The overall RoB was high in 98.9% of the models and only one study¹⁸ showed low RoB in all four domains.

Figure 2.

PROBAST (Prediction model Risk Of Bias ASsessment Tool) risk of bias and applicability. The PROBAST tool¹⁶ was used to assess the risk of bias for the participants, predictors, outcome and analysis for each model. Only one study demonstrated low risk of bias on all domains.

For the domain participants, 82.4% of studies was assessed as high RoB because most studies performed retrospective data analyses or used data from existing sources with large number of candidate predictors that were originally developed for other purposes, for example, administrative databases or registries. The domain predictors were assessed as high RoB in 27.5% of the models, 24.2% as low RoB and 48.4% as unclear RoB. For the domain outcome, 41.8%, 34.1% and 24.2% were assessed as high, low and unclear RoB, respectively.

The domain analysis was assessed as high RoB in 97.8%. Most studies did not use appropriate statistics for the development or validation of prediction models. For example, a description on how complexities in data were handled (eg, competing risk of death) was often missing and relevant performance measures were incomplete (eg, calibration).

The domain participants and predictors were assessed as low concerns regarding applicability in all studies. For the domain outcome, 70.3% of studies used all-cause readmission as the outcome of interest and were therefore assessed as low concerns regarding applicability.

Prediction models

A total of 43 new models were developed for patients with HF (n=15), undergoing surgical procedures (n=12), AMI (n=9), transcatheter aortic valve replacement (TAVR) (n=2), a mixed sample with HF and coronary syndromes (n=2), arrhythmias (n=1), valvular disease (n=1), while one study did not specify the sample (table 1). The c-statistic was lower than 0.6 in 5 models, between 0.6 and 0.7 in 24 models, between 0.7 and 0.8 in 6 models, and between 0.8 and 0.9 in 2 models. In six models, the c-statistic was only reported for a validation cohort (table 2).

Table 2.

Model discrimination and calibration

Study	Model	Setting	Predictors; n	Cohort	Discrimination	Type calibration	Calibration
Moretti et al57	EuroHeart PCI score	ACS	16	External	0.59 (0.48–0.71)	NA
Asche et al46	NR	AMI	19	Development; random split	0.74; NR	NA
Cediel et al58	TARRACO risk score	AMI type 2; ischaemia	7	Development (30d)	0.71 (0.61–0.82)	NA
		AMI type 2; ischaemia	7	Development (180d)	0.71 (0.64–0.78)	NA
Burke et al35	HOSPITAL score	AMI	7	External	0.66 (0.61–0.71)	HLT	p=0.49
Chotechuang et al36	GRACE	AMI	9	External (30d)	0.77 (0.65–0.88)	NA
	GRACE	AMI	9	External (180d)	0.63 (0.49–0.77)	NA
Hilbert et al59	AMI decision tree	AMI	44	Development; External	0.65 (0.64–0.66) 0.61 (0.61–0.62)	NA
Dodson et al18	SILVER-AMI 30-day readmission calculator	AMI	10	Development; random split	0.65; 0.63	HLT	p>0.05; p=0.05
Kini et al60	NR	AMI	12	Development; random split	NR; 0.66	Slope; in large; plot	0.973 (p=0.330); −0.038 (p=0.221)
Nguyen et al19	AMI READMITS score	AMI	7	Development; random split	0.75 (0.70–0.80) 0.73 (0.71–0.74)	Plot; plot
	Full-stay AMI model	AMI	10	Development; random split	0.78 (0.74–0.83) 0.75 (0.74–0.76)	Plot
	CMS AMI administrative model	AMI	32	External	0.74 (0.69–0.74)	Plot
Krumholz et al20	CMS AMI administrative model	AMI	32	Development; external; random split	0.63; 0.63; 0.62	In large; slope
	CMS AMI medical model	AMI	45	Development; random split	0.58; 0.59	NA	0, 1/0.015; 0.997/0.015; 0.983
Rana et al33	Elixhauser index	AMI	30	External	0.53 (0.42–0.65)	NA
	HOSPITAL core	AMI	7	External	0.60 (0.47–0.73)	NA
Atzema et al47	AFTER Part 2 scoring system	Arrhythmia; AF	12	Development	0.69; NR	NA
Lahewala et al40	CHADS2	Arrhythmia; AF	5	External (30d)	0.64	NA
	CHADS2	Arrhythmia; AF	5	External (90d)	0.63	NA
	CHA2DS-VASc	Arrhythmia; AF	9	External (30d)	0.65	NA
	CHA2DS-VASc	Arrhythmia; AF	9	External (90d)	0.63	NA
Benuzillo et al61	CRSS	CABG	5	Development; bootstrapping	0.63; 0.63	HLT	7.13 (p=0.52); 9.31 (p=0.32)
Deo et al62	30-days CABG readmission calculator	CABG	20	Development	0.65	NA
Engoren et al55	NR	CABG	6	Development; random split	0.68 (0.64–0.72) 0.68 (0.64–0.68)	NA
Lancey et al63	NR	CABG	8	Development; random split	0.64; 0.57	NA
Rosenblum et al41	The STS PROM score	CABG	40	External	0.59 (0.57–0.60)	NA
Zitser-Gurevich et al64	NR	CABG	17	Development; external (30d)	0.63; 0.66/0.63	HLT	7.91 (p=0.44)
	NR	CABG	13	Development (100d)	0.65	HLT	6.76 (p=0.56)
Zywot et al42	CABG risk scale	CABG	27	Development; external	NR; 0.70	Plot
Ahmad et al21	CMS HF administrative model	HF	37	External	0.66 (0.57–0.76)	HLT	p=0.19
Amarasingham et al22	ADHERE	HF	3	External	0.56 (0.54–0.59)	NA
	CMS HF administrative model	HF	37	External	0.66 (0.63–0.68)	NA
	Tabak mortality score	HF	18	External	0.61 (0.59–0.64)	NA
Au et al23	Administrative claims model, HF 30-day mortality	HF	17	External	0.58 (0.58–0.59)	NA
	Charlson Comorbidity Score	HF	32	External	0.55 (0.55–0 56)	NA
	CMS HF administrative model	HF	37	External	0.59 (0.59–0.60)	NA
	LACE	HF	18	External	0.58 (0.58–0.59)	NA
Bardhan et al65	NR	HF	30	Development	0.56	NA
Betihavas et al66	NR	HF	7	Development; bootstrapping	NR; 0.80	NA
Burke et al35	HOSPITAL score	HF	7	External	0.67 (0.65–0.70)	HLT	p=0.10
Cox et al24	CMS HF administrative model	HF	37	External	0.61	NA
	CMS HF medical model	HF	20	External	0.60	NA
Delgado et al67	15-day CV readmission risk score	HF	5	Development; bootstrapping	0.65; 0.63	Plot
	30-day CV readmission risk score	HF	11	Development; bootstrapping	0.66; 0.64	Plot
Formiga et al30	CMS HF medical model	HF	19	External (30d)	0.65 (0.57–0.72)	NA
	CMS HF medical model	HF	19	External (90d)	0.62 (0.56–0.68)	NA
Frizzell et al25	CMS HF administrative model	HF	37	External	0.60	NA
Hammill et al26	CMS HF administrative model	HF	37	External	0.59	Plot
Hilbert et al59	HF decision tree	HF	44	Development; External	0.59 (0.58–0.60) 0.58 (0.58–0.59)	NA
Hummel et al31	CMS HF medical model	HF	28	External	0.61	NA
Huynh et al48	NR	HF	12	Development; external (30d)	0.82 (0.76–0.87) 0.73 (0.69–0.77)	NA
	NR	HF	12	Development; external (90d)	NR; 0.65	NA
Ibrahim et al34	HOSPITAL score	HfpEF	7	External	0.60 (0.55–0.64)	NA
	LACE	HfpEF	18	External	0.55 (0.50–0.60)	NA
	LACE+ index	HfpEF	24	External	0.57 (0.52–0.62)	NA
Keenan et al27	CMS HF administrative model	HF	37	Development; external; random split	0.60; 0.60; 0.61	In large; slope	0, 1/0.02; 1.01/ 0.09; 1.05
	CMS HF medical model	HF	30	Development; random split	0.58; 0.61	In large; slope	0, 1/0, 1
Kitamura et al53	FIM	HF	13	External	0.78	NA
Leong et al68	30-day HF readmission risk score	HF	7	Development; random split	0.76; 0.76	NA
Li et al49	NR	HF	10	Development; random split	0.63 (0.62–0.63) 0.63 (0.62–0.63)	HLT; plot	0.15 (p>0.005)
Lim et al69	NR	HF	13	Development	0.68 (car); 0.62 (all)	HLT	27.5 (p=0.001) (car) 8.0 (p=0.429) (all)
Reed et al28	AH model	HF	14	Development; random split	0.86 (0.85–0.86) 0.85 (0.84–0.86)	NA
	CMS HF administrative model	HF	37	Random split	0.55 (0.54–0.56) 0.55 (0.54–0.57)	NA
	Hasan	HF	9	Random split	0.80 (0.79–0.81) 0.80 (0.80–0.82)	NA
	LACE	HF	18	Random split	0.75 (0.74–0.81) 0.74 (0.73–0.76)	NA
Reed et al (continued)28	PARR-30	HF	10	Random split	0.82 (0.81–0.83) 0.81 (0.80–0.82)	NA
Salah et al70	ELAN-HF Score	HF	10	Development	0.60 (0.56–0.64)	NA
Sudhakar et al32	CMS HF medical model	HF	20	External	0.61 (0.57–0.64) ≥65 y, 0.59 (0.53–0.64) Random patient-level, 0.58 (0.50–0.65)	NA
Tan et al71	NR	HF	3	Random split	0.73	HLT; plot	p=0.62
Wang et al72	NR	HF	12	Development	0.65	NA
Wang et al38	LACE	HF	18	External	0.56 (0.48–0.64)	NA
Yazdan-Ashoori et al29	CMS HF administrative model	HF	37	External	0.61 (0.55–0.67)	NA
	LACE	HF	18	External	0.59 (0.52–0.65)	HLT	p=0.73
Disdier Moulder et al73	NR	HF; ACS; NR	4	Development (30d)	0.68	NA
	NR	HF; ACS; NR	5	Development (180d)	0.69	NA
Raposeiras-Roubín et al37	GRACE	HF; ACS	9	External	0.74 (0.73–0.80)	HLT	p=0.14
Minges et al74	NR	HF; PCI	35	Development; random split	0.67; 0.66	NA
Pack et al75	NR	HVD	28	Development; random split	0.67 (full dev)/ 0.65 (nomogram); 0.67 (full val)	Harrell’s E; O, E; Harrell’s E; plot	0.1%; 1.9%; 1.6%
Oliver-McNeil et al76	ICD readmission-risk score	ICD	4	Update; External	0.69 (0.58–0.79)	HLT; plot	3.44 (p=0.49)
Wasfy et al52	Pre-PCI model	NR	23	Development; random split	0.68; 0.67	HLT; plot	p=0.59
Barnett et al77	NR validation	Surgical	15	External	0.59	NA
	NR update	Surgical	18	Update	0.60 (0.59–0.62)	NA
Brown et al43	STS augmented clinical model	Surgical	27	Update (bootstrap); random split; external (bootstrap)	0.66 (0.61–0.72); 0.56; 0.47 (0.42–0.53)	HLT	p=1.0
	STS 30-day readmission model	Surgical	21	Update (bootstrap); random split; external (bootstrap)	0.66 (0.62–0.71), 0.58, 0.47 (0.41–0.52)	HLT	p=0.492
Espinoza et al78	30-day readmission score after cardiac surgery	Surgical	5	Development; random split	0.66 (0.63–0.70) 0.64 (0.61–0.67)	NA
Ferraris et al54	READMIT	Surgical	9	Development	0.70	HLT	5.966 (p=0.651)
Kilic et al79	NR	Surgical	15	Development; random split	NR; 0.64	HLT; plot	p=0.45; p=0.57
Stuebe et al80	NR	Surgical	7	Development	0.63	NA
Tam et al44	NR	Surgical	29	Development; bootstrapping	0.63; 0.65	Plot
Khera et al45	TAVR 30-Day readmission risk model	TAVR	11	Development; random split; external	NR; 0.63; 0.69	HLT; RMSE; RMSE; plot	p=0.33; 0.978; 0.928
Sanchez et al50	NR	TAVR	10	Development; random split	0.61; 0.60	HLT	p=0.749; p=0.403

ACS, acute coronary syndrome; ADHERE, Acute Decompensated Heart Failure Registry; AF, atrial fibrillation; AH, Adventist Health Off-the-shelf model; AMI, acute myocardial infarction; CABG, coronary artery bypass grafting; Car, cardiac-related; CHADS2, Congestive heart failure, Hypertension, Age, Diabetes, previous Stroke/transient ischemic attack; CHADS2-VASc, congestive heart failure, hypertension, age ≥ 75 years, diabetes mellitus, stroke or transient ischemic attack (TIA), vascular disease, age 65 to 74 years, sex category; CMS, Centers for Medicare and Medicaid Services; CRSS, CABG Readmission Risk Score; d, days; dev, development; FIM, motor and cognitive Functional Independence Measure; GRACE, Global Registry of Acute Coronary Events; HF, heart failure; HFpEF, heart failure with preserved ejection fraction; HLT, Hosmer-Lemeshow test; HOSPITAL, Hemoglobin level, discharged from Oncology, Sodium level, Procedure during admission, Index admission Type, Admission, Length of stay; HVD, heart valve disease; ICD, implantable cardioverter defibrillator; LACE, Length of stay, acuity of the Admission, Comorbidity of the patient and Emergency department use in the duration of 6 months before admission; NA, not applicable; NR, not reported; O, E, observed, expected; PARR-30, Patients at Risk of Re-admission within 30 days; PCI, percutaneous coronary intervention; plot, calibration plot; READMITS, Renal Function, Elevated Brain Natriuretic Peptide, Age, Diabetes Mellitus, Nonmale Sex, Intervention with Timely Percutaneous Coronary Intervention, and Low Systolic Blood Pressure; SILVER-AMI, Comprehensive Evaluation of Risk Factors in Older Patients with AMI; STS, Society of Thoracic; STS, Society of Thoracic; STS PROM, Society of Thoracic Surgeons Predicted Risk of Mortality; TAVR, transcatheter aortic valve replacement; val, validation.

A total of 38 separate models were externally validated for patients with HF (n=26), AMI (n=4), surgical patients (n=3), acute coronary syndrome (n=2), arrhythmias (n=2), mixed sample with HF and coronary syndromes (n=1). The discrimination was lower than 0.6 in 16 models, between 0.6 and 0.7 in 15 models, between 0.7 and 0.8 in 5 models, and between 0.8 and 0.9 in 2 models (table 2).

The discrimination of six models was evaluated in multiple independent cohorts and was pooled in meta-analyses (figure 3, online supplemental figures 1–6): the CMS AMI (Centers for Medicare and Medicaid Services Acute Myocardial Infarction) administrative model^{19 20} (0.65, 95% CI 0.56 to 0.73); the CMS HF (Heart Failure) administrative model^21–29 (0.60, 95% CI 0.58 to 0.62); the CMS HF medical model24 27 30–32 (0.60, 95% CI 0.58 to 0.62); the HOSPITAL (Hemoglobin level, discharged from Oncology, Sodium level, Procedure during admission, Index admission Type, Admission, Length of stay) score^33–35 (0.64, 95% CI 0.58 to 0.70); the GRACE (Global Registration of Acute Coronary Events) score^{36 37} (0.78, 95% CI 0.63 to 0.86); and the LACE (Length of stay, acuity of the Admission, Comorbidity of the patient and Emergency department use in the duration of 6 months before admission) score23 28 29 34 38 (0.62, 95% CI 0.53 to 0.70).

Figure 3.

Meta-analysis of prediction models. Random-effect models were used to pool similar models reported in independent cohorts. For the HOSPITAL score, the discriminations for the HF and AMI samples were similar (0.65 and 0.64). For GRACE, the discriminations for the AMI and reinfarction samples were similar (0.77 and 0.74), and was higher for the HF sample (0.83). Only GRACE demonstrated adequate discrimination in external cohorts. AMI, acute myocardial infarction; CMS, Centers for Medicare and Medicaid Services; CF, heart failure. Abbreviations: CMS = Centers for Medicare and Medicaid Services; AMI = Acute Myocardial Infarction; HF = Heart Failure; HOSPITAL = Hemoglobin level, discharged from Oncology, Sodium level, Procedure during admission, Index admission Type, Admission, Length of stay; GRACE = Global Registry of Acute Coronary Events; LACE = length of stay (L), acuity of the admission (A), comorbidity of the patient (C) and emergency department use in the duration of 6 months before admission.

On average, models for patients with AMI had the best discrimination (0.67, n=16), followed by patients with TAVR (0.65, n=2), patients with HF (0.64, n=45) and surgical patients (0.63, n=17). The discrimination was highest in studies using secondary analysis (0.70, n=2) and retrospective cohort studies (0.69, n=23), and was lowest in studies using registries (0.61, n=17) and hospital databases (0.61, n=18). The discrimination decreased when the number of predictors increased (beta −0.002, n=90). There were no moderation effects based on the average age of the sample, outcome definition and endpoint of the prediction (online supplemental figures 7–8 and online supplemental table 1B).

The calibration was reported for 27 models using multiple measures and could not be pooled (table 2).

Predictors

A total of 766 predictor values were estimated in the included models. The median number of predictors per model was 15 (IQR=9–28). The predictors were mostly situated in the domains medical comorbidities (n=211), disease and hospital characteristics (n=128), demographic data (n=128), laboratory values (n=97) and medical history characteristics (n=51). Age (n=47), presence of diabetes (n=26), insurance status (n=24), length of stay (n=28) and gender (n=23) were the most prevalent predictors. There was little consistency in the definition of predictors, and most studies did not report how they were measured.

Only 18 predictors were similarly defined in multiple studies and could be pooled for the outcome readmission at 30 days (figure 4, online supplemental table 2A and online supplemental figures 9–26). The coefficients of four predictors demonstrated a consistent and significant association across the different samples: chronic obstructive pulmonary disease (COPD), HF or history of HF, and valvular disease. The coefficients of 11 predictors demonstrated an overall significant association, that is, age, female gender, arrhythmias, chronic lung disease, diabetes mellitus, cerebrovascular disease, cardiovascular accident, anaemia, peripheral vascular disease, urgent admission and infection, but this was not consistent across the samples and the prediction intervals were not significant. The effect of these predictors was mostly smaller in the HF samples.

Figure 4.

Predictors of unplanned hospital readmission. The plot provides an overview of the random-effects meta-analyses that were performed for predictors who were similarly defined for the outcome unplanned hospital readmission at 30-day follow-up. See online supplemental table 2A and online supplemental figures 9–26 for more details. CABG, coronary artery bypass grafting; COPD, chronic obstructive pulmonary disease; PCI, percutaneous coronary intervention.

The coefficients for most predictors could not be pooled because they had different definitions, cut-off values or reference categories. However, renal disease, including dialysis, a longer length of stay, creatinine, NT-proBNP (N-Terminal-PRO hormone Brain Natriuretic Peptide) and previous hospital admissions demonstrated a consistent association with readmissions.

Discussion

In this systematic review, we included 60 studies that reported the results from 81 separate clinical risk prediction models and 766 risk predictors for unplanned readmission in patients with acute heart disease. We found some promising prediction models, however, no clinical model demonstrated good discrimination (ie, c-statistic >0.8) in independently externally validated cohorts, regardless of the underlying patient populations. GRACE was the only model that demonstrated adequate discrimination in multiple cohorts in patients with acute coronary syndromes^{36 37} and HF.³⁷ There was little consistency in the measurement of risk predictors.

The results of our review are in line with previous systematic reviews which have mainly focused on samples of patients with HF, AMI or focused on generic prediction models. All reviews confirm that the discrimination is generally low. Our review confirms the importance of previous HF^{5 6} and previous hospital admissions^{6 8} as consistent predictors of the risk of readmission. In addition, two prevalent comorbidities, COPD and valve disease, were also consistent predictors across the different populations. Other reviews also identified the importance of age, gender, comorbidities and certain laboratory values. These were also significant in our review but the association was not always consistent across the different populations or heterogeneously measured making comparisons difficult. As a result, no clinical risk prediction model or set of predictors that is relevant for different populations of heart disease could be identified.

Our review focused specifically on prediction models with a clinical presentation that can be used in daily practice, for example, risk scores or nomograms. These simple models do not consider interactions between predictor values or non-linear link functions in their predictions. This may partially explain the poor discrimination.³⁹ Using web applications or electronic patient records to run more complex prediction algorithms can likely offer a solution for future models. A recent systematic review observed an average c-statistic of 0.74 for models using electronic patient records and machine learning algorithms.¹¹ Our review included 11 studies18 29 32 36 37 40–45 that developed or validated electronic tools for risk prediction and their discrimination ranged between 0.59 and 0.77. However, these electronic tools were mostly derived from score charts and nomograms.

There are also concerns about the generalisability of the prediction models. The median age of patients included in the samples was 68 years (IQR=65–75). However, older and frail patients suffer more multimorbidity and geriatric syndromes, and the distribution of predictor and outcome values will also be different than in younger samples. It is therefore unlikely that the majority of the current models will hold their value in daily clinical practice where there is a high prevalence of older patients. Only eight studies18 20 27 46–50 included one or more geriatric risk factors (eg, physical performance, dementia) as predictors for readmission. The performance of models including geriatric conditions was similar to models without these conditions. This might be explained by the relative young mean age of the samples in our review. Mahmoudi et al¹¹ reported that functional and frailty status are important predictors, but were only included in a small number of studies. Frailty was not identified in any of the models in our review. It might be valuable to examine the additive value of these predictors in prediction models for patients with heart disease.

We observed high RoB in almost all clinical risk prediction models (98.8%). This was mainly because the calibration was lacking or not fully reported (eg, only p value of Hosmer-Lemeshow test). Furthermore, most studies performed retrospective data analyses or used data from existing sources. However, our results demonstrate that studies using these data sources had the lowest c-statistic, and that the c-statistic decreased when more predictors were tested. Databases often have missing data, misclassification bias and random measurement error, which likely explains their average poor performance.⁵¹ Only the SILVER-AMI (Comprehensive Evaluation of Risk Factors in Older Patients with AMI) study¹⁸ demonstrated low RoB on all domains. However, their readmission risk calculator for older patients with AMI only discriminated modestly (c-statistic=0.65).

Our review shows the current state-of-the art of risk prediction in patients with acute heart disease. The timely identification of patients with acute heart disease at risk of readmission remains challenging with the prediction models identified in this systematic review. Therefore, further research in risk prediction remains important and some recommendations for further research can be derived from this review. First, consistency is needed in the definition and measurement of predictors. More homogeneity might improve the identification of important predictors and their effect on readmission. Based on our insights, we believe that models could be improved by incorporating some key predictors, that is, age, gender, comorbidity scores (or at least heart failure, COPD, cardiovascular disease, diabetes mellitus), admission status, readmission history and the geriatric profile (eg, functional status, cognitive status). Because there are a still a large number of potential predictors, a large sample size is needed to estimate the coefficients with sufficient precision, and to prevent against overfitting the models. Some selection of predictors may still be warranted, and penalised techniques (eg, lasso regression) should be preferred over traditional selection based on p values. Second, the results suggest that multiple predictors are associated with readmissions regardless of the underlying population. Therefore, attention might be shifted from developing new risk prediction models to updating and externally validating existing prediction models in different populations with heart disease. For example, the Adventist Health Off-the-shelf model²⁸ showed high discrimination rates in both the development (0.86) and the validation cohorts (0.85). External validation is recommended to examine the generalisability of this model in other settings. In addition, the AMI READMITS (Acute Myocardial Infarction Renal Function, Elevated Brain Natriuretic Peptide, Age, Diabetes Mellitus, Nonmale Sex, Intervention with Timely Percutaneous Coronary Intervention, and Low Systolic Blood Pressure) score,¹⁹ full-stay AMI readmission model,¹⁹ pre-PCI model,⁵² motor and cognitive Functional Independence Measure (FIM),⁵³ READMIT,⁵⁴ 30-day readmission model of Huynh et al,⁴⁸ and the model of Engoren et al⁵⁵ were examined in one study and showed reasonable c-statistics in the development (0.68–0.82) and validation cohorts (0.64–0.78). For these studies, model updating recalibration and external validation is recommended to improve the predictive performance and generalisability of these prediction models. Third, the applicability of current prediction models in daily practice is an important concern as most models had poor performance, were not replicated and had high RoB. More high-quality studies are needed that evaluate the discrimination, calibration and clinical usefulness. To limit the RoB as much as possible, future studies should adhere to the relevant reporting guidelines⁵⁶ and could use PROBAST¹⁶ as a guidance to plan their study. Fourth, more complex models integrated in electronic patient records may results in better predictions.

Limitations

Although we performed an extensive literature search, we might have missed some eligible studies, particularly those published in non-English languages. We were able to perform meta-analysis for predictors that were often (≥5 models) reported. However, it might be possible that some less frequently mentioned predictors (eg, geriatric predictors) are a valuable addition in clinical practice. The review included a large number of results and statistical tests which may result in an inflated alpha error. The meta-regression identified that models with less predictors had a better discrimination, but this could also be explained by overfitting models; this could not be tested.

Conclusion

A large number of clinical models have recently been developed. Although some models are promising as they demonstrated adequate to good discrimination, no model can currently be recommended for clinical practice. The lack of independently validated studies, high RoB and low consistency in measured predictors limit their applicability. Model updating and external validation is urgently needed.

Data availability statement

All data relevant to the study are included in the article or uploaded as supplementary information.

Ethics statements

Patient consent for publication

Not required.