A systematic review of cardiac surgery clinical prediction models that include intra-operative variables

Abstract

Background

Most cardiac surgery clinical prediction models (CPMs) are developed using pre-operative variables to predict post-operative outcomes. Some CPMs are developed with intra-operative variables, but none are widely used. The objective of this systematic review was to identify CPMs with intra-operative variables that predict short-term outcomes following adult cardiac surgery.

Methods

Ovid MEDLINE and EMBASE databases were searched from inception to December 2022, for studies developing a CPM with at least one intra-operative variable. Data were extracted using a critical appraisal framework and bias assessment tool. Model performance was analysed using discrimination and calibration measures.

Results

A total of 24 models were identified. Frequent predicted outcomes were acute kidney injury (9/24 studies) and peri-operative mortality (6/24 studies). Frequent pre-operative variables were age (18/24 studies) and creatinine/eGFR (18/24 studies). Common intra-operative variables were cardiopulmonary bypass time (16/24 studies) and transfusion (13/24 studies). Model discrimination was acceptable for all internally validated models (AUC 0.69-0.91). Calibration was poor (15/24 studies) or unreported (8/24 studies). Most CPMs were at a high or indeterminate risk of bias (23/24 models). The added value of intra-operative variables was assessed in six studies with statistically significantly improved discrimination demonstrated in two.

Conclusion

Weak reporting and methodological limitations may restrict wider applicability and adoption of existing CPMs that include intra-operative variables. There is some evidence that CPM discrimination is improved with the addition of intra-operative variables. Further work is required to understand the role of intra-operative CPMs in the management of cardiac surgery patients.

Introduction

The ability to reliably predict post-operative mortality and morbidity in patients undergoing cardiac surgery helps support shared decision making and risk stratification. ¹ This is especially important to clinicians when attempting to determine the most appropriate treatment option for each patient. The application of clinical prediction models (CPMs) in cardiac surgery has helped to improve patient selection and risk-adjusted outcome analysis and has been a key feature in both institutional benchmarking and quality improvement programmes. ² CPMs developed to predict post-operative outcomes in patients undergoing cardiac surgery have typically used pre-operative variables only, because they are primarily used as part of the pre-operative decision-making process.

The EuroSCORE II model, ³ recognised as one of the most frequently used cardiac surgery CPMs across the UK and Europe, was designed to pre-operatively predict post-operative mortality but includes a small number of variables which could technically be modified intra-operatively. ⁴ While inclusion of a variable such as the extent of surgery can largely be anticipated pre-operatively, the inclusion of intra-operative variables which cannot be predicted or calculated prior to surgery would render a model unsuitable for use as part of pre-operative work-up. However, the inclusion of intra-operative variables in CPMs could allow for the updating of a predicted risk estimate initially calculated using pre-operative variables. ⁵ Information from CPMs that incorporate intra-operative data could be used to facilitate post-operative clinical management and decision making.

The emergence of new electronic health data platforms within the clinical environment that can capture complex intra-operative data means that CPMs that utilise this information can be readily calculated. ⁵ Intra-operative data are likely to more accurately reflect surgical complexity, provide information on significant unexpected intra-operative events and capture the individual physiological response to the insult of surgical and anaesthetic intervention. ⁶ Information on these parameters could help to optimise CPMs for the prediction of specific modifiable post-operative outcomes, such as acute kidney injury.

Several CPMs for cardiac surgery that include intra-operative variables have previously been developed but none are widely used in clinical practice. The objective of this review was to systematically identify developed cardiac surgery CPMs that include intra-operative variables and assess their quality and characteristics to understand potential reasons for a lack of clinical adoption.

Methods

Search strategy

This systematic review was undertaken in conjunction with a medical librarian. It has been registered with PROSPERO (International Prospective Register of Systematic Reviews, CRD42021277013) and was based on the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines. Separate literature searches of the MEDLINE (searched using OVID, the online library of databases) and EMBASE databases were undertaken in order to identify studies published between inception of the databases and April 2022. The search topics used adapted medical subject heading (MeSH) terms, keywords and wildcards. Search terms used were “cardiac surgery”, “preoperative”, “intraoperative”, “perioperative”, “mortality”, “morbidity” and other prediction model terminology based on Geersing et al. ⁷ The search strategy used can be found in the Supplemental Material.

Selection criteria

Articles were screened by title and abstract. Inclusion criteria were studies that included adult patients with acquired heart disease, who had received cardiac surgery. Exclusion criteria included CPMs developed specifically for paediatric surgery, adult congenital surgery, cardiopulmonary transplantation or surgery for mechanical circulatory support.

Post-operative outcomes to be assessed could include short-term mortality or morbidity with CPMs developed for longer-term outcomes excluded. Abstracts were screened and the full texts of those that were considered relevant were subsequently evaluated for suitability. Only articles describing the development of a predictive CPM including at least one intra-operative variable were included. Studies performing univariable or multivariable analysis but not undertaking development of a prediction model were not included. Predictive CPMs were defined when the purpose of the multivariable analysis was to detect the optimal combination of risk factors through association, that best predict a current diagnosis or future event. Studies describing only external validation of a model were excluded.

Data extraction

All studies identified as potentially suitable following review of titles and abstracts were analysed in full by two investigators (CJ & MT). The reference lists of included studies were also reviewed in full. Any disagreements over inclusion of studies between the two reviewers were resolved by discussion by a third reviewer (SWG). The Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) framework was adhered to when documenting and evaluating these studies. ⁸ Information on data source, study date, participants, outcome of interest, predictors associated with the outcome, their availability at the time of prediction and included in the final model, cohort sample size, methods of handling missing data, model development, events per predictor parameter (EPP) and model performance were extracted. ⁸ Risk of bias and study quality were assessed in each prediction model using both, the Prediction model Risk Of Bias Assessment Tool (PROBAST) and Quality in Prognostic Factor Studies (QUIPS) instruments. ⁹

Model analysis

Model performance was evaluated through measures of discrimination and calibration. Model discrimination, referring to the ability of the model to differentiate between patients who do and do not experience the event, is generally assessed by the Area Under the Curve (AUC). An AUC of 1 represents perfect discrimination and a value of 0.5 indicates that the model is no better than chance. An AUC ≥0.7 is deemed acceptable and an AUC ≥0.8 represents excellent discriminatory ability. ¹⁰ Calibration refers to how closely predicted outcomes match observed outcomes. This can be calculated for the cohort as a whole, by calculating the observed to expected (O:E) ratio. Other measures include the Hosmer-Lemeshow (H-L) test, flexible calibration plots, calibration-in-the-large and calibration slope. ¹⁰ Finally, the number of variables included in a model must be considered in relation to the number of events in the cohort used to develop the model. Traditionally, a minimum EPP of at least 10 was recommended, although more sophisticated measures to determine sample size are now also available. ¹¹

Results

Selected studies

In total, the literature search returned 5352 articles (Medline n = 1991, Embase n = 3361). After exclusion of duplicates and non-English language articles, 4009 articles remained with 30 articles identified for full assessment. Following the identification of additional studies from the reference lists of articles reviewed in full and exclusions a total of 24 studies remained for analysis.^3,12–34 The study selection process is detailed in Figure 1. In total, 21 models were developed using retrospective data, with the other three models developed using prospectively collected data. Models were developed using datasets ranging in size from 168 to 378,572 and included patients undergoing surgery between 2000 and 2022. The median sample size for model development was 2446. Full details of CPM development and performance are detailed in Table 1.

Figure 1.

Data extraction from Embase and Medline searches.

Table 1.

Model performance for cardiac surgery prediction models with preoperative and intraoperative variables

Prediction model	Sample cohort (n)	Events (n)	Candidate/final variables	Model type	EPP	Candidate variable selection	Final variable selection	Validation type	Model performance	Preoperative model
										Model name	Performance
Models developed to predict mortality
Nashef et al, 2012 (3) * Euroscore II	16,828	779	26/18	LR	29.9	Pre-selected (EuroSCORE)Univariable (p < .05)	Forwards	Int: 10-Fold CV (D75:V25) Ext: (n = 5553, e = 233)	Disc: AUC:0.81 Cal: HL: p = .051	Euroscore	Disc: 0.76 Cal: 0.68
Lamarche et al, 2017 (12) *	18,207	492	29/18	LR	16.9	Univariable (p < .05)	Backwards	Int: RSS (D60:V40) (n = 12153, e = 298); Ext: n/a	Disc: AUC:0.856 Cal: HL = 0.958	Lamarche -Nested model	Disc:0.823 Cal: NR
Zamperoni et al, 2022 (13)*	13,211	562	Pre- 21/31 Intra 10	LR	18.1	Univariable (p < .25)	Stepwise	Int: RSS (D:85:V15); Ext: n/a	Disc:0.84 Cal: GiViT:0.17	Zamperoni -Nested model	Disc: 0.79 Cal: 0.07
Models developed to predict morbidity
Aronson et al, 2007 (14) MCPSI (AKI)	2381	231	32/8	LR	7.2	Univariable (p < .15)	Stepwise	Int: RSS (D49.5:V50.5) Ext: n/a	Dis: AUC: 0.80 Cal: HL; p = .84	Cleveland	Dis: 0.82 Cal: NR
Magee et al,2007 (15) (atrial fibrillation)	19,620	4215	33/14	LR	128	Univariable (p < .05)	Backwards	Int: BS (1000) Ext: n/a	Dis: AUC: 0.72 Cal:HL = 0.19	NR	Disc:NR Cal:NR
Rahmanian et al, 2011 (16) (renal failure - dialysis)	2511	98	36/9	LR	2.72	Univariable (p < .05)	Stepwise	Int: RSS (D80:V20) Ext: n/a	Dis: AUC: 0.851 Cal: NR	NR	Dis: NR Cal: NR
Kim et al, 2013 (17) (AKI)	417	123	25/6	LR	4.92	Univariable (p < .05)	Backwards	Int: RSS(D4:V3) (V320) Ext: n/a	Dis: AUC: 0.74 Cal: HL = 0.99	Thakar, Mehta Palomba, Wijeysundra	Disc: 0.68,0.62 0.60, 0.65; Cal:NR
Tribuddharat et al, 2014 (18) (OHIR model) (ICU-LOS)	168	66	10/6	LR	6.6	Prespecified variables (p < .20)	NR	Int: RSS Ext: n/a	Disc: AUC: 0.75 Cal: HL;p = .242	SOFA	Disc: 0.85 Cal: NR
Kilic et al, 2016 (19) (postop. Pneumonia)	4666	282	15/6	LR	18.8	Univariable (p < .20)	NR	Int: RSS (D75:V25) (V = 1556; Ext: n/a)	Dis: AUC:0.76 Cal:HL: p = .79	STS	Disc: 0.71 Cal: NR
Hessels et al, 2019 (20) (prolonged ventilation)	1496	356	14/7	LR	25.4	Univariable (p < .001)	Backwards	Int: RSS (D75:V25) (V = 498; Ext:n/a)	Disc: C:0.91 Cal:HL: p = .94	Euroscore	Disc: 0.76 Cal: 0.68
Coulson, 2020 (21) (AKI/RRT)	17,048	5829 488	AKI:26/5 RRT:22/5	LR	22,4 22.2	Univariable (p < .05)	Backwards	Int: RSS (D75:V25) (V = 5682); Ext:n/a	Disc: 0.69(AKI);0.87(RRT) Cal: O:E = (AKI)0.991; (RRT):1.103	Ng	Disc: 0.77 Cal:NR
Liu et al, 2021 (22)* (LCOS; AKI; MODS)	930	88 4,2 46	LCOS 48/2 AKI 48/4 MODS 48/6	LR	0.85	LASSO	Stepwise LASSO	Int: BS (1000) (V = 713) Ext: n/a	Disc:0.82 (LCOS),0.78 (AKI), 0.77 MODS) Cal: NR	Liu - Nested model	Disc: 0.57, 0.69, 0.66 Cal: NR
D Wang et al, 2021 (23) (postoperative pneumonia)	5323	530	32/13	LR	16.6	Univariable (p < .1)	Forwards	Int: BS(1000); Ext: n/a	Disc: 0.80; Cal: H-L: p = .443	Kilic, allou	Disc: C: 0.69;0.57 Cal: NR
YS Wang et al, 2022 (24) (AKI)	6888	1013	$\begin{array}{l}52/13\\ 78/8\\ 26/15\end{array}\}10$	LR LASSO RF-RFE	19.4 12.4 37.2	Univariable (p < .1) LASSO Random forest	Stepwise LASSO Backwards	Int: RSS(D7:V3)V = 554); BS(1000);Ext:n = 1575,e = 666) Int: 10-Fold cross validation	Disc: C Stat = 0.733 Cal:Brier score = 0.111, Cal.Slope = 0.995 (E) Disc = C stat = 0.676 (E)Cal:Brier score = 0.260,Cal.Slope = 0.52	Guan	Disc: 0.789 Cal: MSE:0.012
Zainab et al, 2022 (25) (respiratory failure)	2131	457	53/12 -No ASA 53/13 -ASA	LR	8.6	Univariable (p < .05)	LASSO	Int: RSS(1:1) (V = 2131) Ext: n/a	Disc: 0.70 Cal: NR	NR	Disc: NR Cal: NR
Ranucci et al, 2022 (26) (CSA-AKI)	830	84	28/7	LR	3.0	Prespecified variables	NR	Int: BS (1000) Ext:n/a	Disc: 0.769 Cal: Cal.Plot = NR	Ranucci - Bedside tool	Disc:0.59 Cal:NR
Jing et al, 2022 (27) (AKI)	758	288	17/7	LR	16.9	LASSO	LASSO	Int: RSS (D70:V30) (n = 324) Ext: (n = 108; e = NR)	Disc: 0.709; ext. Val: 0.973; N/N:0.749 Cal: NR	Cleveland, SRI score	Disc: 0.82; 0.81 Cal:NR; HL:0.27
Kalisnik et al, 2022 (28) (AKI)	7507	1699	58/21	LR; ML: XGB	29.2	Univariable (p < .1)	Forwards	Int: 10 fold cross Validation Ext: n/a	Disc: 0.83, 0.86, 0.88; Cal:HL: (LR) = 0.41 ML = 0.527;0.521; Slope:1.107, 1.022, 1.028; CITL:-0.066, −0.012, −0.011	Karkouti	Disc: 0.77 Cal: NR
X Wang et al, 2022 (29) (AKI)	226	39	55/8	LR	0.70	LASSO	LASSO	Int: BS (1000) (n = 104) Ext: n/a	Disc: AUC: 0.813 Cal: NR	Cleveland, SRI score	Disc: 0.791;0.786 Cal:NR;HL:0.27
D Wang et al, 2022 (30) (pneumonia – Redo C/S)	251	72	27/4	LR	2.7	Univariable (p < .1)	Forwards	Int: RSS (D66:V33) BS (1000) (n = 125) Ext: n/a	Disc: AUC: 0.78 Cal: HL:0.49	Kilic, STS	Disc: 0.76;0.71 Cal:HL:0.79;NR
Zhang et al, 2022 (31) (AKI)	1170	284	193/14	DF, RF, XGB	1.47	LASSO	LASSO	Int: RSS (D70:V30) (n = 287) (e = 69); Ext. Val = n/a	Disc: AUC:DF: 0.881, RF:0.872, XGB:0.86 Cal: Plot:DF:0.109, RF:0.117, XGB:0.116	Cleveland, SRI score	Disc: 0.791;0.786 Cal:NR;HL:0.27
Models developed to predict Mortality and Morbidity
Stoica et al, 2002 (32)*	1575	451	24/22	LR	18.8	Univariable (p < .05)	Forwards	Int: RSS(V = 1543) (D51:V49);Ext:n/a	Dis: AUC: 0.757 Cal: HL = (p = .11)	Stoica- Nested model	Disc: AUC = 0.717 Cal: H-L (p = .11)
Durant et al, 2020 (33) *	2905	465	31/10	LR	14.9	Prespecified Variables	Shrinkage LASSO	Int: RSS Ext: n/a	Dis: AUC: 0.79 Cal: NR	Durant - Nested model	Disc: C = 0.75 Cal:NR
Mori et al, 2021 (34)*	378,572	45,807	47/40	LR XGB	975	Prespecified Variables	NR	Int: RSS (D70:V30); Ext: n/a	Dis: AUC = 0.755(Mort); 0.82 (Ren.Fail) Cal: NR; Brier score:0.018(Mort);0.02 (Ren. Fail)	Mori - Nested model	Disc: AUC:0.72 Cal:NR

Int = Internal Validation, NR = Not Recorded, Disc = Discrimination, AUC = Area under the curve, EPCP = Events Per Candidate Predictors, EPP = Events per Predictor per Parameter, Ext = External Validation, D = Development, Cal: Calibration, HL = Hosmer-Lemeshow test, RSS = Random Split Sampling, Ren. Fail = Renal Failure, V = Validation, O/E: Observed to Expected Ratio, BS = Bootstrapping, C = Concordance Index, LR = Logistic Regression* Models compared in the same dataset, ML = Machine Learning, LIR = Linear Regression, XGB = XG Boost.

Variable selection and inclusion

The number of variables included in the 24 final models ranged from 4 to 40, with a median number of 9 variables per model. EPPs ranged from 0.70 to 975, with an EPP <10 identified in ten studies. Predictors were selected for multivariable modelling using methods including backwards stepwise selection algorithm (n = 5), forwards stepwise selection algorithm (n = 5), forwards and backwards stepwise selection algorithm (n = 5) and LASSO regression for predictor selection (n = 5), with four models not recording their method of predictor selection.

The most frequently used pre-operative variables were age (18 models) and creatinine/eGFR (18 models). With regards to intra-operative variables, CPB time (16 models) and red blood cell (RBC) transfusion (13 models) were the most frequently included variables. Full details of pre- and intra-operative predictors included in each model are outlined in Tables 2 and 3, while Table 4 displays the variables common to each post-operative endpoint, respectively.

Table 2.

Summary of pre-operative variables included in the development studies.

Prognostic model	Single model risk factor (a)	Age	Gender	Comorbidities	Previous cardiac surgery	Shock	NYHA	Hypertension	Previous MI	LV ejection fraction	Congestive heart failure	Cystatin C	Creatinine/eGFR	Renal disease	Extracardiac arteriopathy	Chronic lung disease	Endocarditis	Critical state	Diabetes mellitus	Angina	Peripheral vascular disease	BMI	Total	Outcome
Mortality
Stoica, 2002	7	X	X		X					X			X		X	X	X	X		X			17	Mort./Morb
Nashef, 2012	5	X	X		X		X			X				X	X		X	X	X	X			16	Mortality
Lamarche, 2017	5	X	X	X									X						X		X		11	Mortality
Durant, 2020	1	X				X					X		X										5	Mort/Morb
Mori, 2021	14	X	X	X	X	X		X		X	X		X	X	X	X			X		X		28	Mort/Morb
Zamperoni, 2022	14	X			X		X		X		X		X	X									21	Mortality
Morbidity
Aronson, 2007	1	X							X		X			X									5	Acute renal event
Magee, 2007	8	X			X						X					X							12	POAF
Rahmanian, 2011	3	X					X						X						X		X		8	RF - dialysis
Kim, 2013	0	X								X			X										3	AKI
Kilic, 2016	0	X														X					X		3	Pneumonia
Tribuddharat, 2016	1	X																					2	LOS-ICU
Hessels, 2019	0																						0	Mech.Vent
Coulson,2020 a	1	X											X									X	4	AKI
Coulson,2020 b	0												X										1	RRT
D Wang, 2021	3	X			X		X	X						X		X			X			X	11	Pneumonia
Liu, 2021 a	0												X										1	Cardiac Dysfunction
Liu, 2021 b	2												X										3	AKI
Liu, 2021 c	4												X										5	MODS
YS Wang, 2022	2	X	X					X				X	X										7	AKI
Zainab, 2022	4					X	X						X			X			X			X	10	Resp.Failure
Ranucci, 2022	0																						0	CSA-AKI
Jing, 2022	0	X	X					X															3	AKI
Kalisnik, 2022	4	X			X					X			X			X					X		10	AKI
X Wang, 2022	3											X	X										5	AKI
D Wang, 2022	0	X											X			X							3	Pneumonia
H Zhang, 2022	3	X	X										X										6	AKI
Total		18	7	2	7	3	5	4	2	5	5	2	18	5	3	8	2	2	6	2	5	3

*Variables not described in research paper. a predictors only evident in single model; b same predictors previously described in precursor model:

The following risk factors were only observed in a single model: Single model risk factor = Nashef: poor mobility; recent MI, thoracic aorta, pulmonary hypertension, CCS4; Stoica: Neurological and muscular dysfunction, Pulmonary hypertension, emergency, other than isolated CABG, postinfarct septal rupture, surgery on thoracic aorta, Recent MI; Lamarche: emergency, GI history, refusal of blood products, alcohol, pulmonary hypertension; Durant: Intra-Aortic Balloon Pump (IABP) or inotropes preoperatively; Mori: Race, Cerebrovascular disease, Body Surface Area, Urgency of surgery, Previous PCI, weight of procedure, left main disease, atrial fibrillation, recent MI, IABP or inotrope, Valve insufficiency, Aortic stenosis, Immunosuppressed, Tribuddharat: Atrial fibrillation; Zamperoni: type of surgerymvr/repair, aortic arch repl, dialysis, infection, IABP; IV vasoactive drugs; Redo operation; complexity of surgery; urgency of surgery; non-ruptured aneurysm, infection-elective surgery, other cardiac surgery; Aronson: Pulse pressure; Magee: Weight, Height, Preop. arrhythmia, ACE inhibitors, Beta blocker, anticoagulants, race, smoke; Rahmanian: Atrial fibrillation, pulmonary hypertension; MI in past 21 days, Kilic: MI in past 24 hours; Coulson: Preop. Haemoglobin; Liu: Cystic Fibrosis (CF), RBC -DW, Bun/BCr, Pulm.Disease; Total bilirubin (T-Bil), cTVR Wang(2021): Smoking history, Preop Anaemia, Hypoalbuminemia; YS Wang: BUN, RBC count, Zainab: Home oxygen, Recent pneumonia, ASA score, previous cardiac intervention; Kalisnik: History of Atrial Fibrillation, Diuretics, Infection, Haemoglobin; X Wang: Emergency, HsCRP, Leucocyte; Zhang: Neutrophil to Lymphocyte ratio, blood glucose, High Density Lipoprotein (HDL).

Table 3.

Summary of intra-operative variables included in the development studies.

Prognostic model	Single model risk factor (a)	Operating time	IABP requirement	CPB time	Mean arterial pressure	O2 delivery	Red blood cells	Blood products: ffp,pltt	Inotropes	Total	Outcome
Mortality
Stoica, 2002	2			X						3	Mort./Morb
Nashef, 2012	2									2	Mortality
Lamarche, 2017	3		X	X			X		X	7	30 day mort
Durant, 2020 a	3						X	X		5	Mort./Morb
Mori, 2021	9	X	X					X		12	Mort./Morb
Zamperoni, 2022	7					X	X	X		10	Mortality
Morbidity
Aronson, 2007	0		X	X					X	3	Acute renal event
Magee, 2007	2									2	POAF
Rahmanian, 2011	0			X						1	Renal failure
Kim, 2013	2	X								3	AKI
Kilic, 2016	0		X	X			X			3	Pneumonia
Tribuddharat, 2016	2					X			X	4	LOS-ICU
Hessels, 2019	2		X	X	X		X		X	7	Mech.Vent
$\begin{array}{l}\text{Coulson},2020\hspace{0.17em}\left(\mathrm{i}\right)\\ \text{Coulson},2020\hspace{0.17em}\left(\text{ii}\right)\end{array}\}\text{Same}\hspace{0.17em}\text{Model}$	0			X						1	AKI
	0		X	X						2	RRT
Liu, 2021 a	0			X						1	LCOS
Liu, 2021 b	0			X						1	AKI
Liu, 2021 c	0			X						1	MODS
D Wang, 2021	0			X			X			2	Pneumonia
YS Wang, 2022	0	X		X			X			3	AKI
Zainab, 2022	1			X			X			3	Resp. Failure
Ranucci, 2022	3			X	X	X	X			7	CSA-AKI
Jing, 2022	2			X			X			4	AKI
Kalisnik, 2022	2						X			3	AKI
X Wang, 2022	1						X			2	AKI
D Wang, 2022	0						X			1	Pneumonia
Zhang, 2022	8									8	AKI
Total		3	6	16	2	3	13	3	4

a predictors only evident in single model.

The following risk factors were only observed in a single model: Single model risk factor = Stoica: return to CPB, mechanical support, Nashef: weight of procedure, urgency of surgery; Lamarche: Intraoperative complications, High dose vasopressors, VAD/ECMO, Durant: cardioplegia delivery, antegrade delivery – cardioplegia, e-aminocaproic acid Mori: highest glucose, lowest body temp., blood cardioplegia, post-op ejection fraction, antibiotics, antegrade delivery of cardioplegia, tricuspid regurgitation, full sternotomy, LIMA not used. Zamperoni:.serum creatinine, pupils-ist day of ICU, cardiogenic shock, systolic arterial pressurerenal failure, sodium, number of organ failures, Kilic: emergent operation; Tribuddharat: Platelet count, potassium level; Hessels: emergency procedure, Cardiogenic shock, lactate; Magee: Use of CPB; Prolonged Ventilation, Kim: Oliguria, Furosemide (Diuretics); Zainab: Mechanical Support, Ranucci: Nadir Haematocrit risk, Time of exposure to critical delivery risk, Peak lactates risk. Jing: 5%SBS solution, Urine Output, X Wang: Aortic Cross Clamp Time, Kalisnik: Aortic Cross Clamp, Emergency Operation, Zhang: Urine output, Conventional Ultrafiltration (CUF), Central venous pressure (CVP), Perfusion flow, PaO2/FiO2, haemoglobin, serum potassium, lactate dehydrogenase.

Table 4.

Summary of postoperative endpoints from the models selected and the intraoperative variables common to each individual endpoint.

Postoperative endpoint	Intraoperative variables common to each endpoint
Mortality	CPB time; type of surgery; blood products i.e FFP, platelets; mean arterial pressure (MAP); trainee operating; return to CPB; mechanical support; mean aortic cross clamp time; weight of procedure; urgency of surgery; IABP requirement; intraoperative complications; high dose vasopressors; VAD/ECMO; oxygen delivery; pupils – 1st day of ICU; platelets; sodium; 1st day of ICU; RBCs transfused; cardioplegia delivery; antegrade delivery – cardioplegia; e-aminocaproic acid; intraoperative TOE; glucose; body temperature; blood cardioplegia; complexity of surgery; postoperative ejection fraction; antibiotics; tricuspid regurgitation; full sternotomy; planned use of combined CPB
Acute kidney injury	CPB time, RBCs transfused, inotropes, operating time, IABP requirement, oliguria, furosemide, aortic cross clamp time, revision of procedure, haemoglobin level, drainage output (0-12 h), change in eGFR, postoperative creatinine, urine output, conventional ultrafiltration, central venous pressure, perfusion flow, PaO2/FiO2 ratio, serum potassium, lactate dehydrogenase, MAP, nadir HCT risk, time of exposure to delivery risk, peak lactate risk
Renal replacement therapy (RRT)	IABP requirement, CPB time
Pneumonia	CPB time, IABP requirement, emergent operation, CABG, valve+CABG, other, RBCs transfused
Respiratory failure	CPB time, RBC transfused
Neurological effect	CPB time, mean arterial pressure (MAP)
POAF (postoperative atrial fibrillation)	Use of CPB
MODS	CPB time, combined operation with tricuspid valve replacement (TVR)
LCOS	CPB time
Mechanical ventilation	IABP requirement, CPB time, RBCs transfused, inotropes used, emergency procedure, cardiogenic shock, lactate
Renal failure	CPB time

Outcomes

Amongst the studies identified, 3 CPMs were developed solely to predict mortality endpoints.^3,12,13 Mortality endpoints included in-hospital and 30-day mortality and a composite endpoint comprising both. A further 18 models were developed to predict morbidity endpoints.^14–31 The most common morbidity endpoints were acute kidney injury (9/18 studies) and post-operative pneumonia (3/18 studies). Additional morbidity endpoints included renal dysfunction, (including both acute kidney injury [AKI] and the need for renal replacement therapy), acute renal event, neurological complications, low cardiac output syndrome (LCOS), atrial fibrillation, multi-organ dysfunction (MOD), respiratory complications, pneumonia, re-operation and post-operative length of stay. Three models included both mortality and morbidity endpoints.^32–34

CPM performance

Calibration was assessed in 16 of the 24 models and was claimed to be acceptable in 15. However, a number of different measures of calibration were used across the models identified, several of which are now known to be problematic. These include the H-L test (used in 12 studies in this review), which cannot provide any information on either the extent or direction of miscalibration. ¹ Superior measures of calibration, including flexible calibration plots, calibration-in-the-large and calibration slope were used in only one study, meaning that acceptable calibration results should be interpreted with a degree of caution.

Added value of intra-operative variables

Six of the models undertook a nested model comparison to compare model performance when only pre-operative variables were included versus model performance once intra-operative variables were added.^{12,13,22,32–34} All of the models identified demonstrated improved discriminatory ability once intra-operative variables were added to the initial model comprised solely of pre-operative variables. Only two of the studies undertook analysis to determine whether the difference in AUC between models was statistically significant. Liu et al. developed models to predict LCOS, AKI and MOD. The AUCs improved significantly with the addition of intra-operative variables (LCOS: 0.57 to 0.82, p < .01; AKI: 0.69 to 0.78, p < .01; MOD: 0.66 to 0.77, p < .01). Durant et al. used data from 2905 patients to develop a model with a composite endpoint of mortality or re-operation. The discrimination of the model improved from 0.75 to 0.79 once intra-operative variables were added. The improvement was found to be statistically significant (p < .01).

Study quality

All selected models were assessed for methodological quality based upon risk of bias and model applicability, using the PROBAST ⁹ framework. Of the 24 models evaluated, almost half were observed to show a high risk of bias with an unclear risk of bias seen in the remainder of the models. The outcome and analysis domains were found to be responsible for the high or unclear risk of bias in the selected models. A single model was found to have a low risk of bias, with results of risk of bias analysis displayed in Table 5.

Table 5.

Risk of bias for cardiac surgery prediction models (Determined using PROBAST framework).

Frequent criteria not met included the handling of missing data, which was associated with a high risk of bias in eleven studies, whilst a further nine studies did not discuss how missing data was handled. All the studies identified described internal validation of the models developed.^3,12–34 Seventeen models were assessed by random split sampling (commonly found in data preparation of prediction models), recognised as inefficient as it reduces the available sample size.^{12–14,16,17,19–21,24,25,27,30–34} Two models used cross-validation^3,28 and the other five models using bootstrapping.^{15,22,23,26,29}

With reference to the QUIPS tool, risk of bias was high for all studies (Table 6). For the initial domain, study participation, 17/24 (71%) had a low risk, 3/24 (12%) had moderate risk, with the remaining 4/24 (17%) having a high risk of bias. Study attrition was found to have a high risk of bias in almost all studies (22/24). Outcome measurement was mainly low risk, 16/24 (67%), although high risk and moderate risk made up the remainder of studies in this domain. The main reasons for high or moderate risk in outcome measurement were the method of outcome measurement being either subjective or not stated, and the use of multi-centre studies made determining matching outcome measurements difficult. The prognostic factor measurement, confounding and statistical analysis domains were found to have a low risk of bias in all models selected. This was the result of all models clearly describing valid prognostic factor measurements and evaluating data using non-selective approaches.

Table 6.

Quality in Prognostic Factor (PF) Studies (QUIPS tool).

	Prediction model	1. Study participation	2. Study attrition	3. Prognostic factor measurement	4. Outcome measurement	5. Study confounding	6. Statistical analysis and reporting
1	Nashef, 2012	Low	Low	Low	Low	Low	Low
2	Lamarche, 2017	Low	High	Low	High	Low	Low
3	Zamperoni, 2022	High	High	Low	Low	High	Low
4	Aronson, 2007	Low	High	Low	Low	Low	Low
5	Magee, 2007	Low	High	Low	High	Low	Low
6	Rahmanian, 2011	Low	High	Low	Moderate	Low	Low
7	Kim, 2013	Low	High	Low	Low	Low	Low
8	Tribuddharat, 2014	Low	High	Low	Low	Low	Low
9	Kilic, 2016	High	High	Low	Low	High	Moderate
10	Hessels, 2019	Moderate	High	Low	High	Low	Low
11	Coulson, 2020	High	High	Low	Low	Low	Low
12	Liu, 2021	Low	High	Low	Low	Low	Low
13	D Wang, 2021	Low	High	Low	Low	Low	Low
14	YS Wang, 2022	Low	High	Low	High	Low	Low
15	Zainab, 2022	Low	Low	Low	Moderate	Low	Low
16	Ranucci, 2022	Low	High	Low	Low	Low	Low
17	Jing,2022	Low	High	Low	Low	Low	Low
18	Kalisnik, 2022	Low	High	Low	Low	Low	Low
19	X Wang, 2022	Low	High	Low	Low	Low	Low
20	D Wang, 2022	Low	High	Low	Low	Low	Low
21	Zhang, 2022	Low	High	Low	Low	Low	Low
22	Stoica, 2002	Moderate	High	Low	High	Low	Low
23	Durant, 2020	High	High	Low	High	Low	Low
24	Mori, 2021	Moderate	High	Low	Low	High	Low

Discussion

This systematic review has identified 24 CPMs developed for use in adult cardiac surgery designed to predict short-term mortality and morbidity after cardiac surgery that include intra-operative variables. The most common variables were age and creatinine/eGFR, whilst the most frequent intra-operative variables were CPB time and RBC transfusion. The most common outcomes associated with the identified CPMs were acute kidney injury and mortality.

Overall model performance in terms of discrimination was broadly acceptable across the studies, with all models showing moderate to good discrimination on internal validation. Methodological issues were apparent in a number of the studies, with one-third of the models not reporting calibration. This is a common issue in CPM research and prevents a full assessment of model performance. Indeed, even when calibration is assessed, statistically unsound methods such as the H-L test are frequently utilised. Several models in this study have been externally validated,^3,24,27 however a full assessment of these external validations was beyond the scope of this review.

Some evidence was identified to suggest that model performance is improved with the addition of intra-operative variables. This is particularly apparent in the six nested models,^{12,13,22,32–34} where the models including intra-operative variables had better discrimination values compared with the baseline model, with improvements ranging between 0.025 to 0.25. Whilst only two studies strengthened this potentially beneficial effect by undertaking formal statistical analysis, in both cases it was found that addition of intra-operative variables led to a statistically significant improvement in the discriminatory ability. However, neither of these studies recorded calibration and both were single centre studies (n = 930 & n = 2905).

Most of the included models were observed to have a high or unclear risk of bias, as a result of issues with reporting and methodological quality. Only one model by Kalisnik et al. ²⁸ was assessed as low risk of bias. These issues included inadequate sample sizes for model development and validation, split sampling for internal validation, omitting missing data at study initiation or analysis, univariable selection, and categorising continuous predictors. While high risk of bias does not mean that the models are unsuitable for use in clinical practice, they should generally be used with caution and only following successful external validation.

The findings of this review mirror the results of other systematic reviews of prediction models that have included pre-operative and intra-operative variables. Although to the authors’ knowledge, no review exists within cardiac surgery. In a review conducted by Grantham et al. ³⁶ that observed combined intra-operative risk models in oesophagectomy surgery, no model could be confidently recommended for clinical use and all required further validation. A review of blood transfusion models in elective surgery by Dhiman et al. ³⁷ concluded that the poor methodological quality and study reporting of models meant that none of the models could be considered for clinical practice without further research. A review of pre-operative and intra-operative scores used in colorectal surgery for surgical decision making by Venn et al. ³⁸ found that calibration assessment was not always performed when assessing model performance, but when it was, sub-optimal calibration metrics were used.

There appears to be a limited number of externally validated models used in routine clinical practice that predict short term outcomes that use both pre-operative and intra-operative variables. The only model that incorporates intra-operative variables and is generalisable to different populations is the Euroscore II. ³ This model has undergone multiple external validations across a diverse range of time periods, geographical locations and patient populations. Despite the inclusion of intra-operative variables, the EuroSCORE II is designed to be used pre-operatively and is not intended to be used to facilitate post-operative management. In the context of this work, precisely defining an intra-operative variable is of key importance. Whilst most variables are fairly easy to define as either “pre-operative” (i.e. demographics, comorbidities and investigations performed prior to surgery) or “intra-operative” (i.e. CPB and cross-clamp time), there are a number of variables which can potentially straddle both of these definitions. Type and extent of procedure falls within this area. Whilst the intended procedure (such as coronary artery bypass grafting) is discussed and confirmed prior to surgery, unexpected intra-operative findings or events can sometimes necessitate additional or alternative procedures being undertaken. Consequently, variables such as these should be considered as primarily pre-operative variables which may be modified intra-operatively.

This review has identified existing models currently developed for use in cardiac surgery, the risk of bias associated with them in their prediction ability and their applicability to daily practice. It has been conducted using PRISMA method of study search and selection strategy. A limitation of this study is that a detailed review of external validations of the CPMs identified has not been performed. Another potential limitation is the decision to only include studies in English. This means that a number of relevant studies written in other languages may have been excluded. By design, the models identified in the review had a tendency to focus on patients receiving coronary artery bypass graft, valvular and combined valve and coronary artery bypass graft procedures which may limit the generalisability of the findings.Although not included in this review due to its study population not meeting the inclusion criteria, an important study on this topic is the analysis by Newland et al. ³⁵  of Australian and New Zealand Collaborative Perfusion Registry (ANZCPR) data, which demonstrated that CPB parameters improve the prediction of 30-day mortality. Despite this, the nature of the patient cohorts and outcome metrics across the studies remained relatively heterogeneous and limits direct comparisons between models. ³⁸ Although, heterogeneous cohorts can limit direct comparisons of model performance between studies, heterogeneous validation cohorts are vital for a comprehensive assessment of CPM performance. In the future, better sharing of multiple patient cohorts underpinning studies such as these could allow for models to be comprehensively evaluated and compared. ³⁹

Conclusion

This systematic review has identified 24 models designed to predict short-term outcomes after cardiac surgery that include intra-operative variables. Whilst a number demonstrated acceptable model performance, all except one had a high or unclear risk of bias. Thus, issues with model design may explain their lack of use in facilitating intra-operative or post-operative patient management. At least two studies were identified that demonstrate that the inclusion of clinically relevant intra-operative variables in CPMs may improve model performance. Further work is required if intra-operative data are to be incorporated into CPMs designed to facilitate patient management.

Appendix.

Abbreviations

AKI:

Acute Kidney Injury

AUC:

Area Under the Curve

CABG:

Coronary Artery Bypass Grafting

CPM:

Clinical Prediction Model

CPB:

Cardiopulmonary Bypass

EPP:

Events Per Predictor Parameter

H-L:

Hosmer-Lemeshow value

MOD:

Multi Organ Dysfunction

O:E:

Observed to Expected Ratio

LCOS:

Low Cardiac Output Syndrome

PROBAST:

Prediction model Risk of Bias Assessment Tool

RBC:

Red Blood Cell

ORCID iD

Ceri Jones https://orcid.org/0000-0002-0882-1773