Comparison of European System for Cardiac Operative Risk Evaluation (EuroSCORE) and the Society of Thoracic Surgeons(STS) score for risk prediction in Indian patients undergoing coronary artery bypass grafting

Abstract

Background

For risk stratifying patients undergoing coronary artery bypass graft (CABG), the Society of Thoracic Surgeons (STS) risk score and the European System for Cardiac Operative Risk Evaluation (EuroSCORE) are currently used. However, the superiority of one over the other in the context of Indian patients has not been assessed. The aim of this study was to compare these 2 scoring systems in Indian patients undergoing CABG.

Methodology

This was a retrospective analysis of prospectively collected data between January 2015 and September 2020 of all patients undergoing CABG. Observed mortality in the cohort was compared with the predicted mortality using the STS and the EuroSCORE II. Sensitivity and specificity were calculated for both the scores. Receiver operating characteristic (ROC) curves were constructed for both the STS and the EuroSCORE II and area under the ROC curve (AUC) was calculated.

Results

A total of 4895 patients were included in the study. The overall observed mortality in the entire cohort was 74 (1.5%). The EuroSCORE II–predicted mortality was 1.9 ± 2.5 whereas the STS score–predicted mortality was 1.2 ± 1.8. The observed to predicted mortality ratio for EuroSCORE was 0.79 and 1.25 for the STS score. The discriminative ability for operative mortality of the STS score was 0.72 (0.71 to 0.74) and 0.713 for the EuroSCORE, suggesting satisfactory discriminatory power. There was no difference between the STS score and the EuroSCORE in terms of discriminatory power (p = 0.58) and a difference in the AUC being 0.01. The discriminatory power of the EuroSCORE and the STS score was best in the high-risk category.

Conclusions

Both the EuroSCORE and the STS scores had satisfactory and similar discriminatory power. However, in the Indian population, while the EuroSCORE II overestimated mortality, the STS score underestimated it to a similar degree of error.

Introduction

Risk prediction models are essential in order to objectively stratify the risk of a procedure which is an essential component of an informed patient consent. Apart from risk prediction, its value lies in risk-adjusted comparison of surgeons and institutions. Several risk prediction models exist in adult cardiac surgical patients [1]. The most common risk prediction models in adults worldwide are the European System for Cardiac Operative Risk Evaluation (EuroSCORE) and the Society of Thoracic Surgeons score (STS score) [2, 3].

EuroSCORE (additive and logistic), introduced in 1999, was derived from an international European database and was used preferentially to other existing risk stratification models. EuroSCORE II was an update on the additive and logistic EuroSCORE and was found to be better calibrated than the original model while preserving the discrimination [4]. The STS database was established in 1989 and is the largest cardiac surgery database in the world. Unlike the EuroSCORE, which has only undergone a single update since its inception, the STS score undergoes periodic updates incorporating evolving changes in patient characteristics, risk profiles, surgical practice, and outcomes. The last major update to the STS scoring system was carried out in 2018 [5].

In the absence of a risk stratification model developed from data in Indian patients, various surgeons and institutions rely on either the EuroSCORE or the STS score for risk stratification purposes. However, these risk stratification models were designed using data from European and American patients, and concerns remain over the applicability of risk stratifying models designed from a different population dataset to accurately predict the risk in Indian patients.

Studies have attempted to examine this concern in Indian patients undergoing cardiac surgery. However, while there is limited data on the STS scores, there have been conflicting reports regarding the applicability of the EuroSCORE in the Indian population. While some studies have claimed that the EuroSCORE II under predicts mortality in the Indian population [6], others have suggested the opposite, concluding that EuroSCORE II overpredicts mortality in the Indian population [7]. Another study has confirmed that EuroSCORE II in its present form is not validated for use in the Indian population [8].

The validity of the STS score has been carried out less often in the Indian population and very limited data exists on its validation. Limited evidence suggests satisfactory calibration but poor discriminatory power for the STS scores, with overprediction of risk in the high-risk group [7].

The conflicting evidence between studies is partly because of inadequate sample sizes and partly due to the presence of significant heterogeneity in the population studied. The sample sizes included in the various studies have ranged from 498 to 1211 patients [6–10]. Besides, most of these studies included a heterogeneous group of patients who had undergone coronary artery bypass grafts (CABGs), valve repairs, and replacements as well as combined procedures. The heterogeneity in procedures can affect the applicability of a score. The lack of validation of EuroSCORE as well as STS score in a robust homogenous dataset in Indian patients thus remains an important lacuna in the literature.

CABGs are the commonest adult cardiac surgical procedures carried out in India, and therefore, in this study, we sought to examine the applicability of both STS score and EuroSCORE II in Indian patients undergoing isolated coronary artery bypass grafting.

Material and methods

All patients undergoing isolated primary CABG between January 2015 and October 2020 were included in the study. Data was collected and entered prospectively into our institutional database and was analysed retrospectively. Ethics approval was obtained from the Institutional Ethics Committee (NHRTIICSEC/AP/2021/001). The aim of the study was to assess if the STS and the EuroSCORE II risk stratification models predicted the operative risk accurately in patients undergoing coronary artery bypass grafting. The objectives included comparison of the calibration as well as the discriminatory ability of the risk prediction models (STS and the EuroSCORE II) and the risk prediction in different risk-stratified sub-groups of EuroSCORE and STS score.

Inclusion criteria included all patients undergoing isolated CABG above the age of 18 years. To ensure homogeneity, we only included isolated CABG as our main study subjects. Thus, during the study period, all patients undergoing CABG were included in the study irrespective of whether it was an elective, urgent, emergency, or re-operative surgery EuroSCORE II as well as the STS predicted risk of operative mortality was calculated before the procedure and documented in the case notes and then entered into our electronic database. Exclusion criteria included all valve and congenital cases where a concomitant CABG was carried out. The strategy of revascularization off-pump coronary artery bypass (OPCAB) or CABG using cardiopulmonary bypass with or without cardioplegic arrest was not considered in the eligibility criteria, as the predicted risks do not differentiate between these variables.

Primary outcome was the correlation between EuroSCORE II and the STS score with respect to predicting mortality in patients undergoing isolated CABG. This was assessed by the observed to predicted mortality ratio in patients undergoing isolated CABG. Secondary outcome was the risk prediction ability of EuroSCORE II and the STS score in different risk groups as well their individual and comparative discriminatory power in predicting mortality in patients undergoing isolated CABG.

The EuroSCORE II as well as the STS scores stratify patients into 3 different categories. The EuroSCORE risk-stratified patients into low (0–2.99), medium (3–5.99), and high risk (6.0 and above) [1]. The STS score divides patients into 3 groups as well—low risk (<4), intermediate risk (4–8), and high risk (>8) [2]. Comparison of predicted and observed mortality was performed in the entire cohort as well as across the different sub-groups for both the EuroSCORE II and the STS score.

The STS score in addition to predicting mortality also predicts the short and prolonged postoperative length of stay, stroke rates, incidence of prolonged ventilation, deep sternal wound infections (DSWIs), renal failure, and re-operations. The observed and STS score–predicted morbidity indicators were also compared. DSWI was defined as the presence of at least one of the following: (1) an organism isolated from culture of mediastinal tissue or fluid; (2) evidence of mediastinitis seen during operation; or (3) presence of either chest pain, sternal instability, or fever (>38 °C), and purulent drainage from the mediastinum, or isolation of an organism present in a blood culture or a culture of the mediastinal area. All other definitions were as defined in the STS database.

Calibration was measured using the Hosmer-Lemeshow (HL) test; a variation of the chi-square statistic is a goodness of fit test, especially for risk prediction models with binary response variables [11]. It is essentially a measure of the model’s ability to predict survival for various levels of patient risk [12].

A receiver operating characteristic (ROC) curve is a graphical plot of true-positive rate (sensitivity) versus false-positive rate (1-specificity) at various threshold settings, thereby illustrating the diagnostic ability of a binary classifier system [13]. ROC curves were constructed to evaluate the discriminatory power of the EuroSCORE II and the STS scores. EuroSCORE II and the STS scores were compared using the ROC curves in the entire cohort as well as risk-stratified sub-groups.

Discrimination assesses the ability of a model to predict a survivor from a non-survivor and was measured by the area under the ROC curve (AUC) or c statistic. AUC, or c statistic, is defined as the proportion of the time that a patient who survives is assigned a higher probability of survival than a patient who dies in the hospital [14].

The correlation of EuroSCORE II with the STS score was also assessed using scatter plots and the Bland-Altman curves. A positive correlation between scoring models indicates that the 2 scores are in agreement. The higher the score, the higher is the agreement.

Statistical analysis

Apart from the tests described above, for the purpose of data reporting, categorical data was described as number and percentage. Continuous variables were reported using mean and standard deviation. Scatter plots and Bland-Altman curves were created to examine the correlation between EuroSCORE II and the STS scores.

All statistical analysis was performed using SPSS v.24.0 (IBM Corp., Armonk, NY, USA).

Results

There were 4895 (mean age 59.0 ± 11.5) patients who underwent isolated CABG during the study period. The cohort consisted of 662 (13.5%) women and the detailed demographic and patient-specific characteristics are described in Table 1. The overall observed mortality in the entire cohort was 74 (1.5%). The EuroSCORE II–predicted mortality was 1.9 ± 2.5, whereas the STS score–predicted mortality was 1.2 ± 1.8. OPCAB was the strategy of revascularization in 4538 (92.7%) of the patients, and left internal thoracic artery (LITA) was used in 4379 (90.3%) patients in total (Table 2). Respiratory complications were encountered most frequently in the cohort of 1126 (23%) and the re-exploration rate was 197 (4.02%) (Table 2).

Table 1.

Baseline characteristics

Pre-operative characteristics	Total no patients = 4895 (no/%)
Age(mean ± SD)	59.0 ± 11.5
Body mass index (mean ± SD)	24.7 ± 15.6
Females	662 (13.5%)
Hypertension	4180 (85.4%)
Chronic obstructive airway disease	277 (5.7%)
Diabetes	2504 (51.2%)
Pre-op renal dysfunction	497 (10.2%)
Thyroid dysfunction	319 (6.5%)
Peripheral vascular disease	200 (4.1%)
Carotid artery stenosis	234 (4.8%)
Pre-operative CVA	58 (1.2%)
Pre-op TIA	42 (0.9%)
EF (mean ± SD)	49.9 ± 12.34
EuroSCORE II (mean ± SD)	1.9 ± 2.5
STS mortality (mean ± SD)	1.2 ± 1.8
30-day mortality	74 (1.5%)

Table 2.

Intraoperative and postoperative outcomes

Intraoperative variables
LMS	795 (16.2%)
OPCAB	4538 (92.7%)
LIMA usage	4379 (90.3%)
Radials	878 (17.9%)
BIMA	322 (6.6%)
Unplanned conversion	37 (0.8%)
Postoperative outcomes
Arrhythmias	428 (8.7%)
Respiratory complications	1126 (23%)
Atelectasis	930 (19%)
Pleural effusion	303 (6.2%)
CPAP	125 (2.6%)
Re-intubation	214 (4.4%)
Tracheostomy	37 (0.6%)
24-h drain	470 ± 245
Deep sternal wound infection	8 (0.16%)
Adverse neurological outcomes	57 (1.2%)
GI complications	16 (0.3%)
Renal failure	32 (0.65%)
Re-explorations	197 (4.02%)

The Hosmer–Lemeshow test for STS and EuroSCORE has a p value of p = 0.07 and p = 0.01 respectively suggesting a better model fit for the STS score. The discriminative ability for operative mortality of the STS score was 0.72 (0.71 to 0.74) as measured by the AUC (Fig. 1). The discriminative ability of the EuroSCORE for operative mortality, as measured by the AUC, was 0.71 (0.70  to 0.72) suggesting satisfactory discriminatory power (Fig. 2). There was no difference between the STS score and the EuroSCORE in terms of discriminatory power (p = 0.58) and a difference in the AUC being 0.016 (Fig. 3).

Fig. 1.

ROC curve showing the relationship of EuroSCORE and mortality

Fig. 2.

ROC curve showing the relationship of STS score and mortality

Fig. 3.

ROC curve comparing AUC for the EuroSCORE and STS score

The discriminatory power of the EuroSCORE was best in the high-risk category (AUC = 0.70) compared to the low (AUC = 0.62) and medium (AUC = 0.65) categories (Fig. 4). Similarly, the STS score also had the highest discriminatory power in the high-risk group (AUC = 0.80) compared to low (AUC = 0.67)- and intermediate (AUC = 0.52)-risk groups (Fig. 5).

Fig. 4.

AUC for different EuroSCORE risk-stratified categories (low = 0–2.99, medium = 3–5.99, high ≥6)

Fig. 5.

AUC for different STS scores stratified risk categories (low <4, medium 4–8, high >8)

The sensitivity and specificity of the STS score at the criterion value of 1.198 were 67.6% and 68.6% respectively. The sensitivity and specificity of the EuroSCORE at the criterion value of 1.886 were 63.5% and 71.6% respectively.

The observed operative mortality in the entire cohort was 1.51% (74/4895). The predicted mortality with the EuroSCORE II and the STS score was 1.9% and 1.2% respectively. The observed to predicted mortality ratio for EuroSCORE was 0.79 and 1.25 for the STS score. Thus, the EuroSCORE tends to overestimate the mortality (p = 0.01), whereas the STS score tends to underestimate the mortality to a similar degree (p = 0.01).

The cumulative observed morbidity was 13.2% compared to the cumulative STS-predicted morbidity of 11.2% (Table 3). While the overall observed and predicted postoperative morbidity ratio of the STS score was 1.16 and appears accurate, there were significant differences in the individual morbidity components. The STS score–predicted morbidity indicators were similar to the observed rates for DSWIs, but significant variability existed with respect to both long and short postoperative length of stay.

Table 3.

STS-PROM risk prediction

STS morbidity	Observed risks	Expected risks
Cumulative morbidity	13.20%	11.32%
Prolonged PLOS	2.45%	4.86%
Short PLOS	16.38%	53.27%
Stroke	0.14%	1.22%
Prolonged ventilation	11.11%	7.68%
DSWI	0.16%	0.32%
Renal failure	0.45%	2.15%
Re-operations	4.07%	4.82%

PLOS, postoperative length of stay; DSWI, deep sternal wound infection

Both the STS score and the EuroSCORE II risk-stratify patients into low-, medium (or intermediate)-, and high-risk categories. The observed to predicted ratios of mortality in the three risk categories for the EuroSCORE II were 0.79, 0.71,and 0.94 respectively. The observed to predicted ratios of mortality in the three risk categories for the STS score were 1.21, 1.41, and 1.33 respectively. There was no significant difference between the observed and predicted mortality in any of the risk categories for either of the scoring models (Table 4). However, both the EuroSCORE and the STS scores were least accurate in the medium (intermediate)-risk category. EuroSCORE in the high-risk category had the most accurate observed to predicted mortality ratio. McNemar’s test was used to compare the predictive accuracy of the two models. The p value of the 2 times 2 contingency table of the two model predictions was 0.46. Thus, the null hypothesis could not be rejected, suggesting that none of the 2 scoring systems performed better than the other, and there was an agreement between both the risk scores.

Table 4.

Comparison of observed and predicted mortality for EuroSCORE and STS score risk-stratified sub-groups

	Observed	Expected	O/E ratio	p value
EuroSCORE sub-groups
Low risk (n = 4373)	47 (1.07%)	59 (1.35%)	0.79	0.24
Medium risk (n = 361)	10 (2.77%)	14 (3.88%)	0.71	0.53
High risk (n = 161)	17 (10.56%)	18 (11.18%)	0.94	0.99
STS score sub-groups
Low risk (n = 4766)	59 (1.23%)	49 (1.02%)	1.20	0.33
Intermediate risk (n = 91)	7 (7.79%)	5 (5.49%)	1.41	0.76
High risk (n = 39)	8 (20.5%)	6 (15.3%)	1.33	0.76

The scatter plot (Fig. 6) shows the EuroSCORE II plotted along the y-axis and the STS score along the x-axis. The trend line or the line of the best fit shows a positive linear relation. The correlation coefficient is 0.58, suggesting a moderate degree of correlation between the two scoring systems. The Bland-Altman plot was drawn (Fig. 7) and the difference between the EuroSCORE and the STS scores was plotted (y-axis) against the mean of the EuroSCORE and STS scores (x-axis). The mean was 0.67 (0.61–0.73) with an agreement range of − 3.4 and 4.7.

Fig. 6.

Scatter plot showing the correlation between STS score and EuroSCORE. EuroSCORE II plotted along the y-axis and the STS score along the x-axis

Fig. 7.

Bland-Altman plot showing the correlation between STS score and EuroSCORE

Discussion

The main finding of our study was that the risk prediction of both EuroSCORE II and the STS score was good. However, while the STS score underestimated the mortality, the EuroSCORE overestimated it to the same degree. Both the scores performed best in the high-risk category of patients. EuroSCORE II and the STS scores both had satisfactory and similar power of discrimination for mortality after isolated coronary artery bypass grafting with a positive correlation between them. The goodness of fit of the STS score model is better than that of EuroSCORE II. The goodness of fit test is used to assess if the observed mortality matched the predicted mortality across different sub-groups stratified by risk scores and thus is a statistical measure of the model’s ability to predict survival at various levels of patient risk.

The clinical implications of these findings are that both STS score and EuroSCORE can be used successfully for risk stratification in the Indian population undergoing isolated coronary artery bypass grafting. Even though none of these scores is perfect, they provide a reasonably good idea of the risk of mortality in these cases. One approach could also be to calculate the mean of EuroSCORE and the STS score which would provide an almost identical observed mortality.

The goodness of fit test with the STS scores was better, but it should be borne in mind that goodness of fit tests, such as the HL test, are subject to discrepancies in larger samples as ours. It has been documented that as power of traditional goodness of fit tests increases with the sample size, practically irrelevant discrepancies between estimated and true probabilities are increasingly likely to cause the rejection of the hypothesis of perfect fit in larger samples [15]. The reported discriminatory ability for both the STS score and the EuroSCORE is higher in the North American population. While the AUC in our study for the STS and the EuroSCORE was 0.72 and 0.71 respectively, it was found to be 0.84 for both the scores in the patients undergoing cardiac surgery in the USA. The correlation between the 2 scores was also found to be higher in this population (r 0.83, p < 0.001) compared to the correlation seen in our study (r 0.58, p < 0.001) [16]. The correlation coefficient is a measure of the strength of the linear relationship between the STS Score and the EuroSCORE. A value above 0.7 indicates a strong positive linear relationship while values between 0.3 and 0.7 represent a moderate positive linear relationship. Despite that, from a purely clinical perspective, the observed to predicted mortality ratio for the STS score and EuroSCORE II in our study was more accurate, compared to that seen in the US population. While the observed to predicted mortality ratio was 0.68 for the STS score and 0.56 for the EuroSCORE in the US population, the risk prediction was more accurate for the EuroSCORE (0.79), as well as the STS score (1.25). Even though the STS and the EuroSCORE underestimated and overpredicted the risk respectively, the deviation was much less than that seen in the US population.

Another study that examined the EuroSCORE alone for validity in the Bangladeshi population found that the observed to predicted mortality ratio was 1.1 for EuroSCORE II [17]. Unlike ours, in this study, the EuroSCORE underestimated the risk of mortality. Previous studies carried out in the Indian population have more commonly looked at the EuroSCORE and rarely examined the STS score [6–10]. This could be perhaps due to the fact that the STS score, even though it provides more information on risk prediction, is more cumbersome to calculate. The AUC from these studies for the EuroSCORE II has ranged from 0.69 to 0.82 [6–10]. However, studies which reported the AUC for only coronary artery bypass grafting reported similar discriminatory power in the Indian population and support our findings [6, 7]. The mild variation in AUC in other cases can be explained by the heterogeneity of the population and the small sample sizes.

The STS score has been uncommonly studied in the Indian population. The reported AUC for the STS score is 0.65. The AUC in our study was far superior with an AUC of 0.72.

While AUC and discriminatory power are statistical concepts, which are important to analyse risk scores, the more intuitive and clinically relevant concept is the observed to predicted ratio of different scores. Some of the previous studies have reported this ratio to be extremely high. In one of the more recent studies, the observed to predicted mortality ratio in CABG was 2.12, suggesting that observed mortality was more than twice that predicted by EuroSCORE. However, this finding is contradicted by our study, where this ratio was 0.79. The inference from our study was that EuroSCORE overpredicts the mortality to some extent in patients undergoing CABG, but it is far more accurate than previously reported. A closer look at the study reporting observed to predicted mortality ratio of 2.12 shows that the sample size for patients undergoing CABG in this study was only 435, which is much smaller than our study that included 4895 patients. The second important observation is that the mortality in patients undergoing CABG in this study was 6.6%, which is significantly higher to commonly accepted mortality in contemporary practice. The mortality after CABG has continued to fall in the last 2 decades and, according to most recent estimates, is around 1.6% [18]. This is very similar to the mortality reported in our series (1.5%).

A more recent study carried out in the Indian population had an observed to predicted mortality of 0.79, which is exactly the same as our study [7]. The ratio was variable at 1.04, 0.68, and 0.58 in the low-, medium-, and high-risk categories. However, in our study, these values were more consistent at 0.79, 0.71, and 0.94 respectively. The authors concluded that EuroSCORE II was unable to accurately predict the mortality in the high-risk group. In contrast, the finding in our study suggests that EuroSCORE was most accurate in the high-risk group, with an observed to predicted mortality ratio of 0.94. The likely explanation for this variation is again a heterogenous group that included coronary as well as various valve procedures and a very small sample size. While in this study, the high-risk group consisted of only 31 patients with a heterogeneous mix of various cardiac procedures; in our study, the high-risk group had 161 patients undergoing only CABG.

The same study compared the observed to predicted mortality based on STS score and reported it to be 1.02, 1.27, and 0.57 in the low-, intermediate-, and high-risk groups respectively. The conclusion drawn was that STS score, just like the EuroSCORE, is unable to accurately predict the mortality risk in high-risk groups. Again, examination of the high-risk group renders this assertion questionable. The high-risk group, based on the STS score, in this study included only 16 patients who had undergone various cardiac procedures. In contrast, the reported ratios were 1.20, 1.41, and 1.33 in the low-, intermediate-, and high-risk groups in our study. The high-risk group in our study, even though small (n = 39), was still more than twice as big and consisted only of patients undergoing CABG. Thus, the sample size and homogeneity of data are keys to accurate estimation of risks and thus assessment of the performance of the EuroSCORE and the STS score.

While the overall observed and predicted postoperative morbidity ratio of the STS score was 1.16 and appears accurate, there were significant differences in the individual morbidity components. The most important being the predicted “short-length of stay,” which is defined as a stay less than 6 days after CABG [19]. The observed short length of stay was 16.38%, while the predicted rate was 53.27%, making the observed to predicted ratio as 0.30. This extremely poor prediction should not be looked at as a failure of the STS model, but was an outcome of different social and cultural expectations following a major surgery in India. Patients are often reluctant to leave the hospital, even though medically fit, due to fear of lack of support once discharged. Moreover, small-sized coronaries with complex lesions are not accounted for, either in the EuroSCORE or the STS score, nor are frailty scores, all of which can potentially influence the accuracy of risk prediction. These observations makes a case for adjusting some of the definitions for the Indian population for better accuracy.

Limitations

The important limitation of our study was it being a single-center study. Despite the large sample size, it may not be representative of real-life practices in the wider surgical fraternity. For accurate validation of risk stratification models, it is important to involve several centers to account for geographical and healthcare infrastructure variations as well as the effect of social and economic differences on outcomes.

Conclusions

Both the EuroSCORE and the STS score predict the mortality outcome fairly accurately in the Indian population and should be used. Based on our study, the EuroSCORE overpredicts and the STS score underestimates mortality risk to some degree after isolated CABG. The STS score has the added advantage of predicting postoperative morbidity outcomes as well. Creating a database of Indian patients may allow making population-specific adjustments and should be aimed for. A multi-center study to validate the risk models in CABG as well as other cardiac procedures is recommended.

Funding

None.

Declarations

Informed consent

This being a retrospective study, the need for informed consent was waived off by the ethics committee.

Ethical approval

Ethical approval was obtained from the institutional ethics committee (NHRTIICSEC/AP/2021/001) on the 4th January 2021.

Conflict of interest

The authors declare no competing interests.