Combination of European System for Cardiac Operative Risk Evaluation (EuroSCORE) and Cardiac Surgery Score (CASUS) to Improve Outcome Prediction in Cardiac Surgery

Abstract

Background

We hypothesized that the combination of a preoperative and a postoperative scoring system would improve the accuracy of mortality prediction and therefore combined the preoperative ‘additive EuroSCORE‘ (European system for cardiac operative risk evaluation) with the postoperative ‘additive CASUS’ (Cardiac Surgery Score) to form the ‘modified CASUS’.

Material/Methods

We included all consecutive adult patients after cardiac surgery during January 2007 and December 2010 in our prospective study. Our single-centre study was conducted in a German general referral university hospital. The original additive and the ‘modified CASUS’ were tested using calibration and discrimination statistics. We compared the area under the curve (AUC) of the receiver characteristic curves (ROC) by DeLong’s method and calculated overall correct classification (OCC) values.

Results

The mean age among the total of 5207 patients was 67.2±10.9 years. Whilst the ICU mortality was 5.9% we observed a mean length of ICU stay of 4.6±7.0 days. Both models demonstrated excellent discriminatory power (mean AUC of ‘modified CASUS’: ≥0.929; ‘additive CASUS’: ≥0.920), with no significant differences according to DeLong. Neither model showed a significant p-value (<0.05) in calibration. We detected the best OCC during the 2^nd day (modified: 96.5%; original: 96.6%).

Conclusions

Our ‘additive’ and ‘modified’ CASUS are reasonable overall predictors. We could not detect any improvement in the accuracy of mortality prediction in cardiac surgery by combining a preoperative and a postoperative scoring system. A separate calculation of the two individual elements is therefore recommended.

Background

A postoperative scoring system for intensive care medicine should help the physician to judge a patient’s condition objectively. These systems might support the evaluation of disease severity and support therapeutic decisions [1].

Several preoperative risk stratification models such as the ‘additive European System for Cardiac Operative Risk Evaluation’ (EuroSCORE) [2] are currently in daily use in cardiac surgery. These scores are population-based and offer a probability of occurrence in a set of patients with similar characteristics. A prediction of individual outcome might be achieved [3].

Postoperative models are not based on pre- or perioperative variables but solely consider parameters after ICU admission. Nevertheless postoperative scoring systems are highly accurate in mortality prediction after cardiac surgery [4].

We developed the postoperative ‘additive CArdiac SUrgery Score’ (CASUS) (Table 1) as a tool for risk stratification on intensive care unit (ICU) after cardiac surgery [1]. The score was evaluated and validated in large populations [5], and showed high accuracy in other independent patient subsets [4,6,7].

Table 1.

The ‘modified’ and the ’additive’ CASUS.

Additive CASUS		Organ system	Descriptor	Score points
				0	1	2	3	4
	Modified CASUS	Respiratory	PaO2/FiO2 (mmHg/%)	Extubated	>250	151–250	75–150	<75
		Renal	Creatinine (mg/dl)	<1.2	1.2–2.2	2.3–4.0	4.1–5.5	>5.5
			CVVH/dialysis	No				Yes
		Liver	Bilirubin (mg/dl)	<1.2	1.2–3.5	3.6–7.0	7.1–14.0	>14.0
		Cardiovascular	PAR=HR × CVP/MAP	<10.1	10.1–15.0	15.1–20.0	20.1–30.0	>30.0
			Lactic acid (mmol/l)	<2.1	2.1–4.0	4.1–8.0	8.1–12.0	>12.0
			Intraaortic balloon pump	No				Yes
			Ventricular assist device	No				Yes
		Coagulation	Platelets ×103/μL	>120	81–120	51–80	21–50	<21
		Central nervous	Neurologic state	Normal		Confused	Sedated	Diffuse neuropathy
		‘Additive EuroSCORE’		0–2 (low)		3–5(medium)		≥6 (high)

Increasing abnormality was graded on a scale from 0 to 4 points, a score of 0 representing normal or minimally deranged function, a score of 4 correlating with markedly deranged function. Diffuse neuropathy includes signs and symptoms of stroke or cerebral hemorrhage. The ‚additive EuroSCORE‘ has to be calculated in advance and added to the CASUS parameter as listed above. CVP – central venous pressure; CVVH – continuous venovenous hemofiltration; FiO₂ – fraction of inspired oxygen; HR – heart rate; MAP – mean arterial blood pressure; PAR – pressure-adjusted heart rate; PaO₂ – partial oxygen pressure.

In the past it has been questioned whether the combination of accurate preoperative and postoperative scoring systems could result in an even more precise mortality prediction. To address this question we created the ‘modified CASUS’ (Table 1) by combining the preoperative ‘additive EuroSCORE‘ [2] and the postoperative ‘additive CASUS’. We statistically compared the original ‘additive CASUS’ with the new ‘modified CASUS’.

Material and Methods

General information

The study is based on a prospective data collection of all adult patients who were admitted to our ICU after cardiac surgery during the period of January 2007 and December 2010. We only included ICU-admissions following cardiac surgery. Any readmitted patient was not considered in regard to exclude the heterogeneous population of patients suffering from post-sternotomy mediastinitis, since this condition has a negative impact on mortality and morbidity [8]. The study found approval (approval number: 2809-05/10) from the Institutional Review Board of our university. Please find further information of the data collection in our previous study [7].

The primary endpoint of this study was ICU mortality. In the past we defined our endpoint as ICU mortality rather than hospital mortality [9] to avoid potential inaccuracies that might occur due to variations in ICU discharge patterns and to exclude unrelated deaths after discharge. In our large tertiary referral university hospital a discharge to a normal ward of a smaller hospital nearby is a common scenario. This policy might affect hospital mortality as our primary endpoint and is therefore misleading [10].

Score calculation of the ‘modified CASUS’

We subdivided our study population according to the EuroSCORE in a low-, medium- and high-risk group and compared it with the original EuroSCORE database to detect any differences in the datasets and to avoid inaccuracies when combining the two models. The two respective populations were found to have no significant differences (Table 2).

Table 2.

Comparison of the study population and that of the original EuroSCORE.

Risk group	Scoring system	Total no.	No. of deaths	%	95%-CI	p-value
Low	CASUS	1262	7	0.6	0.20–1.00	0.2388
	EuroSCORE	4529	36	0.8	0.56–1.10
Medium	CASUS	1557	41	2.6	1.86–3.37	0.2006
	EuroSCORE	5977	182	3.0	2.62–3.51
High	CASUS	2388	257	10.8	9.57–11.97	0.3186
	EuroSCORE	4293	480	11.2	10.25–12.16
Overall	CASUS	5207	305	5.9	5.19–6.40	0.9961
	EuroSCORE	14799	698	4.7	4.37–5.06

95%-CI – 95%-confidence interval; no. – number.

We calculated the ‘modified CASUS’ by implementing the ‘additive EuroSCORE’ as an additional variable to the ‘additive CASUS’ to assess the preoperative mortality risk of an individual patient. The new score (‘modified CASUS’) contains, accordingly, 11 variables. The total number of points ranges from 0–44.

The new variable (‘additive EuroSCORE’) was divided into three categories – low (0–2), medium (3–5) and high (≥6) –, according to the risk groups that were defined during the developmental phase of the ‘additive EuroSCORE’ by thresholds to achieve similar sized groups [2].

Each category of the new parameter was affiliated with points. In the low risk category the points remain 0, whilst the medium and the high risk group were added 2 and 4 points, respectively. Table 1 demonstrates the ‘modified CASUS’.

Statistical analyses

We evaluated the discrimination according to ROC-curves, by assessing the AUC which indicate the discriminative ability of the parameters. For the comparison of the new ‘modified CASUS’ with the ‘additive CASUS’ we used the method according to DeLong. We calculated overall correct classification (OCC) values and assessed calibration by the Hosmer and Lemeshow goodness of fit test (HL).

We defined the p-value <0.05 as significant and conducted the statistical analyses during the first five days. Please find detailed information on the applied statistical methods in our previous study [7].

Results

Population characteristics

During the four years of data collection we included 5207 patients with a mean age of 67.2±10.9 years. 37.6% were female. Whilst the ICU mortality was 5.9% we observed a mean length of ICU stay of 4.6±7.0 days. The preoperative mean ‘additive EuroSCORE’ was 6.3±3.7 and mean ‘logistic EuroSCORE’ was 9.9±12.9. We could not detect any gender-based differences. There were no missing data in this study. Please find the types of surgical procedures in our previous study [9].

Results of the statistical analyses

Table 3 summarizes OCC, the discriminatory power, the comparison of the two models’ AUC curves by DeLong’s method, and the calibration of both CASUS models. Both preoperative EuroSCORE models are included in Table 3. The modified CASUS showed a mean AUC of ≥0.929 and the original ‘additive CASUS’ of ≥0.920; the second day results were the best (modified: 0.965; additive: 0.957). Comparison of the two scores according to DeLong’s method showed no significant differences between the models. We found the best OCC on day two (modified: 96.5%; additive: 96.6%). Both CASUS models showed good calibration on all days (p-value >0.05). Our analysis of the EuroSCORE revealed for both models similar results in discrimination; but we identified the ‘logistic EuroSCORE’ to be significant in calibration statistics according to HL test.

Table 3.

Statistical results.

Day (patients)	Scoring model	OCC	ROC-analysis		DeLong	Calibr. HL-test
		%	AUC	95%-CI	p-value*	Chi2	p-value
Preoperative (5207)	Add. EuroSCORE	94.1	0.734	0.699–0.769	0.8749	15.21	0.055
	Log. EuroSCORE	94.1	0.732	0.697–0.767		20.06	0.011
ICU-day 1 (5207)	Add. CASUS	95.4	0.905	0.887–0.924	0.9631	5.461	0.604
	Mod. CASUS	95.6	0.914	0.894–0.934		7.639	0.472
ICU-day 2 (5159)	Add. CASUS	96.6	0.957	0.938–0.976	0.9141	12.39	0.109
	Mod. CASUS	96.5	0.965	0.956–0.974		15.24	0.055
ICU-day 3 (2372)	Add. CASUS	93.6	0.935	0.920–0.950	0.6311	13.14	0.091
	Mod. CASUS	93.7	0.937	0.921–0.953		13.82	0.087
ICU-day 4 (1612)	Add. CASUS	91.6	0.912	0.904–0.920	0.5066	8.121	0.322
	Mod. CASUS	91.7	0.926	0.907–0.945		10.77	0.215
ICU-day 5 (1164)	Add. CASUS	89.8	0.893	0.886–0.901	0.5169	9.967	0.267
	Mod. CASUS	89.6	0.904	0.886–0.922		10.63	0.223

Results of overall correct classification, discrimination (receiver operating characteristic), AUC comparison by DeLong method and calibration (HL-test) for the two preoperative EuroSCOREs and for the ’additive CASUS‘ and the ’modified CASUS‘ from day 1 until day 5. 95%-CI – 95%-confidence interval; Add. CASUS – ‘additive CASUS’; Add. EuroSCORE – ‘additive EuroSCORE’; AUC – area under ROC-curve; Calibr. HL-test – calibration Hosmer-Lemeshow test; ICU-day – intensive care unit day; Log. EuroSCORE – ‘logistic EuroSCORE’; Mod. CASUS – ‘modified CASUS’; OCC – overall correct classification; ROC – receiver operating characteristic.

^*P-value of the DeLong method, which compares the ‘additive CASUS’ and the ‘modified CASUS’.

Discussion

How promising is the combination of a pre- and a postoperative scoring system?

The ‘additive CASUS’ showed excellent ability to predict ICU mortality in this study. Similar power in outcome prediction was demonstrated by the score in previous studies [1,4–7]. To summarize the preoperative risk of our study population we included the additive and ‘logistic EuroSCORE’. Both models demonstrated a moderate statistical performance regarding mortality prediction. This fact highlights the importance of postoperative parameters in risk stratification.

Stoica et al. recommend considering adverse intraoperative events to enhance preoperative risk prediction [11]. According to Gomes et al., the first postoperative day variables relate to intraoperative conditions and indicate organic dysfunction. The authors underline the importance of integrating postoperative variables for prognostic assessment in cardiac surgery by presenting strong odds ratios of these variables [12].

Nevertheless the consideration of several ‘additive EuroSCORE’ parameters such as patients’ age and emergency surgery among others, were shown to have significant impact on identifying high-risk patients [13]. Combining the preoperative EuroSCORE and the postoperative CASUS might therefore lead to an improved prediction of mortality in cardiac surgery.

There are two reasons not to consider the ‘logistic EuroSCORE’ as an element of the ‘modified CASUS’. At first we detected a significantly inaccurate calibration of the ‘logistic EuroSCORE’ in our statistical analysis (Table 3). Furthermore in spite of the introduction of the logistic model, the original ‘additive EuroScore’ remained the accessible-to-all “gold standard” of risk assessment in European cardiac surgery [14]. We therefore combined the ‘additive EuroSCORE’ and the ‘additive CASUS’ to develop the ‘modified CASUS’.

What is the most appropriate preoperative model for the combination of scoring systems?

EuroSCORE and the Society of Thoracic Surgeons (STS) risk stratification algorithm are the two leading preoperative risk stratification models in cardiac surgery [15]. Nilsson et al. compared these two models in cardiac surgery. They evaluated the ‘additive EuroSCORE’ and the STS in a prospective study on 4497 CABG-patients. The calibration statistics according to Hosmer-Lemeshow indicated a good accuracy of both models. Nevertheless the discriminatory power was significantly (p<0.00005) larger for EuroSCORE (0.84) compared to STS (0.71) [16].

In a comparative study of 19 preoperative risk stratification models Nilsson et al. [3] named the ‘Cleveland Clinic’ risk score and the ‘Magovern risk algorithms’ as the only two scoring systems beside the EuroSCORE to show high accuracy in predicting mortality after open-heart surgery. Since the ‘Cleveland Clinic’ risk score and the ‘Magovern risk algorithms’ are not widely in use, we avoided including them in our study. Though the ‘New York State’ risk score achieved good predictive results it was not considered an appropriate model for our study, since it is only applicable for CABG surgery [3].

Though showing similar predictive results to the EuroSCORE in a study of 1639 patients [17], we did not consider the ‘Parsonnet Score’ since other studies concluded that it is inferior to the EuroSCORE in mortality prediction after cardiac surgery [18]. Finally Nilsson concluded that the EuroSCORE is more reliable than other scoring models that use preoperative parameters [3].

Out of several potential preoperative systems we therefore chose to include the EuroSCORE in the ‘modified CASUS’, since it is widely in daily use and has shown valid statistical performance.

Is the ‘additive EuroSCORE’ in times of EuroSCORE II outdated?

The widely used ‘additive EuroSCORE’ is based on data from 132 centers from eight European countries providing a total of 19030 patients [2]. The score addresses 17 preoperative risk factors, which are divided into three groups – patient-related, cardiac-related and operation-related. It has been extensively tested around the world and showed good statistical results [19].

Nashef et al. published the EuroSCORE II in 2012. The new score is based on prospective data of 22381 consecutive patients after cardiac surgery in 154 hospitals in 43 countries. The data collection was conducted over a 12 weeks period from May to July 2010 [20]. The authors state that cardiac surgical mortality is significantly reduced in the last 15 years despite older and sicker patients. EuroSCORE II is better calibrated than the original models and preserves powerful discrimination [20], which was also proven in a large meta-analysis of 22 studies involving 145,592 patients [21].

The original EuroSCORE models and the new EuroSCORE II vary merely regarding their parameters. Besides minor changes such as different cut-off points in various parameters, the new variable renal impairment has replaced serum creatinine levels. Two new parameters have been introduced in the new EuroSCORE II, namely the mobility of the patient and insulin-dependent diabetes. Furthermore the variable neurological dysfunction has been dropped. The original parameter of unstable angina has been substituted by the two variables NYHA and CCS class 4 angina [2,14,20]. Small changes were made in the section of operation-related factors to consider the urgency of the procedure and the weight of intervention [20].

Barili et al. validated the new EuroSCORE II in comparison to the original additive and ‘logistic EuroSCORE’. In this multicentre study prospective data of 12325 cardiac surgery patients were analyzed. All three models achieved an identical discriminatory power of AUC=0.82. The EuroSCORE II showed a good calibration but overestimated mortality in high risk patients. Consequently the new EuroSCORE II does not significantly improve the performance of the older versions. Furthermore the authors postulate a simplification of the new model, since several parameters seem to be redundant [22].

Kunt et al. compared the additive and ‘logistic EuroSCORE’ with the new EuroSCORE II and the STS model regarding the scores’ ability to predict mortality in cardiac surgery. The authors report of an actual hospital mortality of 7.9%. However, the predicted mortality by the ‘additive EuroSCORE’ was 6.4% and by the ‘logistic EuroSCORE’ 7.9%. EuroSCORE II and STS severely underestimated the mortality (1.7% and 5.8%, respectively). The four models achieved moderate results in discrimination statistics with AUC values of 0.70 for the additive and logistic model, 0.72 for the EuroSCORE II, and 0.62 for STS. Hence, the EuroSCORE II does not improve mortality prediction in cardiac surgery compared to the two original EuroSCORE models and the established STS [23].

Kirmani et al. compared the recently published EuroSCORE II and the 2008 published STS model. The study is based on 15497 patients from the years 2001 to 2010. The EuroSCORE II and the STS were equivalent (p-value=0.343) with AUC of 0.818 and 0.805, respectively. The calibration of both scores according to Hosmer-Lemeshow was good for patients with low and moderate risk, but poor (p<0.0001) when the risk was higher than 15% [15].

Spiliopoulos et al. compared the performance of the EuroSCORE II and the additive and logistic models of EuroSCORE in predicting early and mid-term mortality after combined aortic valve replacement and CABG. The authors report an AUC of 0.749 and 0.75 for the additive and ‘logistic EuroSCORE’, respectively, and 0.77 for EuroSCORE II in early mortality prediction. The ROCs for mid-term mortality were 0.745, 0.739, and 0.718 for additive, ‘logistic EuroSCORE’ and EuroSCORE II, respectively. Furthermore in “high-risk” patients (EuroSCORE >13), EuroSCORE II underestimated early and mid-term outcomes [24].

Arnáiz-García et al. compared the predictive ability of the new EuroSCORE II with that of the original ‘logistic EuroSCORE’ on 1200 patients after cardiac surgery. The population’s overall mortality was 6.8%, whereas that predicted by ‘logistic EuroSCORE’ and EuroSCORE II was 9.7 and 3.7%, respectively. Hence, the ‘logistic EuroSCORE’ overestimated and the EuroSCORE II severely underestimated mortality. The authors state good overall discrimination abilities for both models with the ‘logistic EuroSCORE’ being superior to EuroSCORE II [25].

Our study is based on prospective data only. Since the EuroSCORE II is a new scoring system that was not available during our study period between 2007 and 2010 we considered the original EuroSCORE version in this study. This might only be a theoretical shortcoming. The above mentioned studies demonstrate clearly that the original EuroSCORE models are not inferior to the new EuroSCORE II in mortality prediction [22–25].

Why does the combination of pre- and postoperative variables not improve mortality prediction?

We have to address why the ‘modified CASUS’ did not show any significant improvement regarding the accuracy of mortality prediction. Previous studies discussed several EuroSCORE variables that might compromise the predictive ability of the system. In 2006 Jakobsen et al. questioned whether left ventricular dysfunction was associated with mortality as suggested in previous studies. They concluded that the definition of cardiac dysfunction and the accuracy of the left ventricular ejection fraction needed more precise clarification to increase the reliability of the EuroSCORE [26]. In 1997 Gabrielle et al. published a study of 6649 patients, in which the impact of the patients’ gender and neurological dysfunction was questioned [27]. In a study of 4918 patients Higgins et al. challenged the inclusion of chronic pulmonary disease, active endocarditis and pulmonary hypertension [28]. In 2001 Kawachi et al. postulated that some EuroSCORE variables showed no significant influence on mortality, namely the patients’ age, extracardiac arteriopathy, previous cardiac surgery and postinfarct septal rupture [18]. Furthermore we believe that the patients’ age is an inconsistent variable due to positive selection regarding the patients’ preoperative state and the surgeons’ experience. Finally, volatile and dynamic postoperative parameters dominate over preoperative variables, which cannot take into account intraoperative events.

Which conclusions could be drawn from this study?

The ‘additive CASUS’ and the ‘modified CASUS’ are reasonable overall predictors for all types of cardiac surgery patients. We could not detect any improvement in the accuracy of mortality prediction in cardiac surgery by combining a preoperative and a postoperative scoring system. A separate calculation of the two individual elements is therefore recommended.

Study limitations

This study is based on a prospective single-centre database of a large referral hospital in Germany. It would be necessary to conduct a similar study, which should be multi-centre based, to further validate our findings.