Validation of the European System for Cardiac Operative Risk Evaluation-II model in an urban Indian population and comparison with three other risk scoring systems

Abstract

Aims and Objectives:

The aims were to compare the European System for Cardiac Operative Risk Evaluation (EuroSCORE)-II system against three established risk scoring systems for predictive accuracy in an urban Indian population and suggest improvements or amendments in the existing scoring system for adaptation in Indian population.

Materials and Methods:

EuroSCORE-II, Parsonnet score, System-97 score, and Cleveland score were obtained preoperatively for 1098 consecutive patients. EuroSCORE-II system was analyzed in comparison to each of the above three scoring systems in an urban Indian population. Calibrations of scoring systems were assessed using Hosmer–Lemeshow test. Areas under receiver operating characteristics (ROC) curves were compared according to the statistical approach suggested by Hanley and McNeil.

Results:

All EuroSCORE-II subgroups had highly significant P values stating good predictive mortality, except high-risk group (P = 0.175). The analysis of ROC curves of different scoring systems showed that the highest predictive value for mortality was calculated for the System-97 score followed by the Cleveland score. System-97 revealed extremely high predictive accuracies across all subgroups (curve area >80%). This difference in predictive accuracy was found to be statistically significant (P < 0.001).

Conclusions:

The present study suggests that the EuroSCORE-II model in its present form is not validated for use in the Indian population. An interesting observation was significantly accurate predictive abilities of the System-97 score.

INTRODUCTION

Preoperative risk scores are an essential tool for risk assessment, and the study of therapy trends. Various scoring systems have been developed to predict mortality after adult heart surgery.[1,2,3,4,5,6,7,8,9] Although all of these score systems are based on patient derived data such as age, gender, comorbidity, and so forth, there are considerable differences between scores with regard to their design and validity. As quality control and cost-benefit analysis have gained new relevance with recent developments in the health care system, selection of appropriate score systems for the evaluation of hospital performance has become an important issue. It was the purpose of our study to test validation of the European System for Cardiac Operative Risk Evaluation-II (EuroSCORE-II) model in the Indian population and compare it with three commonly used preoperative risk scores for heart surgery with regard to their predictive values and clinical applicability for our patient population.

The EuroSCORE-II method has been shown to be a valuable measure for prediction of immediate postoperative death after adult cardiac surgery. The EuroSCORE-II has been used to calculate the risk of perioperative mortality, since 2011. The EuroSCORE has gained wide acceptance in Europe and has also been validated in North America and Japan. This has been largely demonstrated in patients undergoing on-pump and off-pump coronary artery bypass surgery (coronary artery bypass grafting [CABG]) and in valve surgeries in its recent version. The predictive value of these models is useful in deciding indications for surgery, estimating the need for resources, obtaining proper informed consent, and monitoring the quality of surgeons and institutions.[10,11,12,13,14,15,16] Operative mortality is widely used as an indicator of the quality of cardiac surgery. To make an accurate comparison between different institutions or surgeons’ mortality data must be adjusted to the risk profiles of the patients.[13]

The Society of Thoracic Surgeons (STS) National Adult Cardiac Database, which is the largest of its kind, has been used to develop an algorithm to predict operative mortality.[3] This risk stratification model is widely used in North America. The more complicated STS database algorithms remain proprietary and confidential, which could explain why only a few studies comparing the STS database algorithm with other risk algorithms have been published. The Parsonnet additive model is a useful tool for quality monitoring in surgical practice. The modified Bernstein-Parsonnet algorithm showed a good correlation between predicted and observed mortality in patients undergoing CABG (on and off-pump) or valve surgery. Its predictive ability was low for thoracic aortic surgery, surgery involving mechanical complications and combined carotid-CABG procedures.

The bulk of our Indian patients has coronary and rheumatic pathologies, as compared to western population; there are vast differences in the anatomical distribution of coronary artery disease and in their caliber. Similarly, the majority of valve disease in the western population is degenerative as compared to rheumatics in our population with significant cardiac dysfunction. With these the sociocultural and dietary habits of the North Indian population, the existing risk scoring systems may need modifications to meet the requirements of this subset.

Hence, the aim of our study was to risk prospectively stratify 1098 consecutive adult cardiac patients using the EuroSCORE-II system, and independently assess predictive accuracy of each of them, to compare the EuroSCORE-II system against three established risk scoring systems for predictive accuracy in an urban Indian population and suggest improvements or amendments in the existing scoring system for adaptation in Indian population.

MATERIALS AND METHODS

This was a prospective study conducted at Max Heart and Vascular Institute, New Delhi, from June 2012 to June 2013. One thousand and ninety-eight consecutive adult patients undergoing off-pump or on-pump CABG, valve surgery, surgeries for mechanical complications of myocardial infarction (MI), and surgeries on the thoracic aorta with the exclusion of congenital cardiac surgery were considered for prospective study using the EuroSCORE-II risk stratification system.

All patients undergoing on-pump procedures had standard myocardial preservation techniques employing intermittent antegrade and retrograde cold blood cardioplegia. All relevant data of these patients were prospectively entered into a max adult cardiac surgical datasheet and reviewed in order to analyze the various risk factor variables.

Simultaneously all these patients were risk stratified using three other risk scoring systems, namely, Parsonnet score, the System-97 score, Cleveland score. At the termination of the study, the EuroSCORE-II system was analyzed in comparison to each of the above three scoring systems with respect to its predictive value in an urban Indian population subscribing to the geographical area of North India. It was an attempt by us through this study to find out as to which of these four scoring systems is able to most accurately predict the observed mortality risk in the study population, as well as the risk factors associated, with an increased risk of in hospital mortality.

Patients included in the study were adult patients undergoing, on-pump or off-pump CABG, surgery for mechanical complications of MI, isolated, and combined valve surgeries and surgeries of the thoracic aorta. Those undergoing surgical repair of congenital cardiac lesions and patients admitted for surgeries of the infra-diaphragmatic aorta were excluded.

Statistical analysis

Data were presented as absolute numbers, mean, and standard deviation, or percentages. Data acquisition of the more than 40,000 data entries was performed using Microsoft Access and Excel (Microsoft Corporation, USA), version 2007. Data analysis was performed using the SPSS software IBM Corporation, New York, United States. Nominal data were analyzed using Chi-square or, where appropriate, Fisher's exact test. Receiver operating characteristics (ROCs) curves were plotted for the different score systems and the area under the ROC curve was calculated as an index for the predictive value of the model. Areas under ROC curves were compared, according to the statistical approach suggested by Hanley and McNeil. To analyze the predictive value of specific risk factors or score items we calculated it according odds ratios. A P < 0.05 was considered significant.

We assessed discrimination by the c-index (area under the ROC curve) and calibrated by the Hosmer–Lemeshow statistic.

Assessment of outcome

The observed hospital mortality shall be statistically analyzed in comparison to the predicted mortality based on the EuroSCORE-II, Parsonnet score, the Cleveland score, and the System-97 score. For the purposes of this study, mortality is defined as 30-day mortality.

RESULTS

There were 1098 adult cardiac surgical patients presenting to a single tertiary care hospital in North India who participated in the study. All patients at the point of entry were risk stratified using the online EuroSCORE-II calculator. All the relevant data were then entered along with the risk score as per all four scoring systems under study in the Max Adult Cardiac Surgery Database. Based on the calculated EuroSCORE-II all patients were divided into the five subgroups namely low, fair, poor, medium, high, and very high-risk groups. The patient population that was risk stratified is depicted in Figure 1.

Figure 1.

Patient population risk stratified

Patient characteristics

The mean age of patients was 57.67 (standard deviation SD - 13). Age range was 18–88 years, 6.7% were age, 75 or over and 27% of the patients were females. The mean weight of the patients was 64.4 (SD - 18.3 kg.). There were 45% of the patients detected to be hypertensive, and 23% patients were diabetics.

Vascular disease

The evidence of extracardiac arteriopathy was present in 13.50% of the patients. Three percentage of them had already undergone vascular surgery, and 25% were scheduled for the future surgeries. Seven percentage of the patients had intermittent claudication with 1.2% having rest pain. The carotid disease was present in 5% of the patients with bilateral involvement in 25% of these. The abdominal aortic disease was present in 1.5% patients.

Renal and respiratory function

Diagnosis of chronic renal insufficiency was established in 3% of the patients, but only 0.7% of them were on long-term dialysis support. Chronic pulmonary disease as defined by long-term use of bronchodilators and steroids was present in 130 patients (12.7%).

General cardiac status

Previous cardiac surgery had been carried in 4.4% of the patients with all of them having one previous cardiac surgery. Chronic cardiac insufficiency was present in 12% of the patients, with 53.9% of patients being in New York Heart Association (NYHA) class III and IV. Atrial fibrillation was present in 11% of patients. Left ventricular function was normal 59.4% with an ejection fraction (EF) of 50% or more, moderate in 32.5% (EF 30–50%), and poor in 8.1% (EF <30%).

Cardiac status of coronary patients

In patients undergoing isolated coronary artery surgery, previous MI (MI prior to 90 days before admission) was present in 12.5% patients. About 14.5% of the patients had MI within the previous 30-day, with 3.2% of them having it within 8 h of admission, 1.6% within 8–48 h of admission, 3.7% within 48 h to 7 days of admission, and 5.8% with 7 days – 1-month of admission.

Unstable angina was present in 21.9% of the patients requiring intravenous nitrate therapy and systemic heparinization. Triple vessel disease was present in 88% of the patients with only 4% having single vessel disease. Left main stem stenosis (>50%) was present in 15.9% of patients. Emergency surgery because of complications in cardiac catheterization laboratory accounted for 1% of all procedures.

Cardiac status of valve patients

In mitral valve (MV) surgery, the predominant valve lesion was a mitral stenosis, which was present 60% of the patients with 28% having MV regurgitation and 12% having both lesions. Pulmonary artery hypertension was present 11.5% of the patients. In aortic valve surgery, the predominant lesion was AS, which was present in 55% of the patients with aortic regurgitation in 25% and 20% having mixed lesions. Double valve replacement (DVR) was undertaken in 2% of the patients. None of the patients operated had active infective endocarditis.

Operations

Elective surgery accounted for 80% of the procedures with 20% being defined as urgent or emergent procedures. Isolated coronary artery bypass surgery (CABG) was performed in 79.2% patients, CABG + MV replacement (MVR) in 1.5% patients and CABG + aortic valve replacement (AVR) in 0.9% patients. MVR alone was done in 6.3% and AVR in 7.2% patients and DVR in 2% patients. About 1% had operations of the thoracic aorta with all of them the involvement of aortic root and ascending aorta. Tricuspid valve repair was performed concomitantly with MVR in 1.6% patients.

Mortality

Overall cardiac surgical mortality was 3%. A detailed breakup of procedure wise mortality is given in Table 1. Based on the EuroSCORE-II calculation as described in earlier patients were risk stratified into five subgroups. A plot of observed and predicted mortality in all cases has been given [Figure 2].

Table 1.

Breakup of procedure wise mortality

Types of procedures	Number of deaths
Off-pump CABG	8
On-pump CABG	10
Bentall	0
AVR	2
DVR	0
MVR	2
CABG+MVR	2
CABG+AVR	3
CABG+CEA	1
MVR+TV repair	0
Post-MI VSD repair	4

CABG: Coronary artery bypass grafting, AVR: Aortic valve replacement, DVR: Double valve replacement, MVR: MV replacement, MV: Mitral valve, CEA: Carotid endarterectomy, VSD: Ventricular septal defect, MI: Myocardial infarction, TV: Tricuspid valve

Figure 2.

Observed and predicted mortality in all cases

Risk factor analysis

Multivariate analysis identified increased creatinine values, peripheral vascular disease, pulmonary artery hypertension, EF <30%, low cardiac output states, perioperative intra-aortic balloon pump insertion, and type of operation to be statistically significant risk factors in causing mortality in the studied population [Table 2].

Table 2.

Risk factors causing mortality

Variables	P
Creatinine	0.017
Peripheral vascular disease	0.017
Pulmonary artery hypertension	<0.0001
Ejection fraction<30%	0.001
Low cardiac output	<0.0001
Preoperative IABP	<0.0001
Types of operation	0.001

IABP: Intra-aortic balloon pump

Validation of the scoring system under trial

All relevant data were prospectively entered into max adult cardiac data sheet and statistical analysis was performed using Microsoft Excel/Access 2007 and SPSS statistical software. After applying to the EuroSCORE-II algorithm data, logistic regression of operative mortality was performed this enabled calibration and discrimination studies to be applied.

The two critical aspects to be validated in any kind of risk stratification model are calibration and discrimination. Calibration was measured by comparing observed mortality to existed mortality. Discrimination was measured by reporting the c-index of the logistic regression model compared with the other three models.

Calibration

Each of the subgroups, as defined by the EuroSCORE-II model, was separately calibrated using the Hosmer–Lemeshow test. All subgroups had highly significant P values except high-risk group (P = 0.175).

Discrimination

Discrimination was measured by reporting the c-index of the logistic regression model. The c-index can be considered as a generalization of the area under the ROC curve and is calculated by analyzing all possible pairs of patients that can be formed such as one patient died and the other did not. Values of the c-index range from 0.5 (no ability to discriminate) to 1.0 (full ability to discriminate). The ROC curve thus plotted comparing the EuroSCORE-II, Cleveland score, Parsonett, and System-97 for all causes mortality is detailed below [Figure 3]. ROC curve was separately plotted for isolated coronary artery bypass surgery and isolated valve surgeries, as well as each of the subgroups, of EuroSCORE-II model.

Figure 3.

Receiver operating characteristics curve for all four scoring systems plotting all causes mortality

The analysis of ROC curves from different scoring systems showed that the highest predictive value for mortality was calculated for the System-97 score followed by the Cleveland score [Figure 3]. Differences between the scores for the areas under the ROC curve were not statistically significant between the System-97 score, the Cleveland score and EuroSCORE-II score (P = 0.23). ROC curve potting for all case mortality revealed that Parsonnet score had a value < 0.7 (70%) and, therefore, were disqualified as applicable models [Figure 3].

On observing ROC curve for valve-related mortality, one found that EuroSCORE-II score in addition to Parsonnet score lost their predictive accuracy (curve area <70%) [Figure 4]. However, System-97 and Cleveland score still had areas of in excess of 70%.

Figure 4.

Receiver operating characteristics curves comparing various scoring systems in valve surgery patients

ROC curve areas for each of the curve subgroups of EuroSCORE-II models revealed a lack of predictive accuracy in the high (curve are 66.4%-EuroSCORE-II, 66.2%-Parsonnet) and very high (curve area 55.0%-EuroSCORE-II and 59.2%-Parsonnet). However, System-97 revealed extremely high predictive accuracies across all subgroups (curve area >80%). This difference in predictive accuracy was found to be statistically significant (P < 0.001).

DISCUSSION

Analysis of patient outcome has gained an ever-increasing importance as institutions, healthcare providers, insurance companies, and patient demand statistically sound data on risk and prognosis for specific procedures and therapies. In particular, cost intensive surgical procedures like CABG have been at the receiving end of careful scrutiny of cost-benefit analysis and comparison of mortality rates between various institutions.[1,2] The patient populations defer significantly between institutions and countries, thus, the comparison of absolute numbers such as the mortality rates is not feasible. A large variety of risk scores have been developed to correct for differences in patients population and compare observed mortality to expected mortality.[3,4,5]

However, the crux of the issue is the significant difference between scores with regard to the initial population on which the score design was based. The clinical database used for the score development may be derived from a single institution or multiple institutions, one region, one country, or a number of countries occupying the same geopolitical space. Another major reason for the difference is retrospective versus prospective data collection and the inclusion of prospective validation study following completion of the score design.

There have been many studies of risk factors in cardiac surgery most were derived from the North American population and may not necessary apply to the Indian context. Another milestone was the emergence of the EuroSCORE-II system,[6] which is undergoing extensive validation for European and North American population.

There are two studies for the Indian population, one validating original EuroSCORE[7] and the other validating EuroSCORE-II.[8] For the original EuroSCORE (n = 1000), it accurately predicted mortality in low and moderate risk Indian patients but under-estimated in high-risk Indian patients. In contrast, the other study was for EuroSCORE-II, which stated that the risk was overestimated in high-risk patients. Apparently, the application of these scoring systems remains uncertain in high-risk patients where risk assessment is very crucial. This study was undertaken to enhance further our knowledge in this regard, and to produce a reliable and contemporary risk profile for adult cardiac surgery in India. Excellent compliance and high degree of data accuracy was obtained because the time and effort in the time and preparation of a comprehensive, user-friendly and unambiguous data collection sheet – the Max Adult Cardiac Surgery Database.

Many risk factors have been associated with cardiac surgery mortality.[1,17] The elegant three-part compartmentalization namely preoperative characteristics, type, and severity of cardiac disease, type, and extent of surgical procedure of the EuroSCORE-II system was chosen as the prime candidate, and prospective risk stratification was done.

A controversial topic among investigator is the most appropriate statistical model for score development.[18,19,20,21] Among the applied statistical tools are the calculation of simple odds ratio, logistic regression analysis, and Bayesian models. Logistic regression models have applied frequently, and results of various investigators show that a score of good predictive value can be developed using this statistical model.

The first step in our analysis was confirming whether the EuroSCORE-II model was calibrated to the Indian population at all. A test of predictive accuracy revealed that all though EuroSCORE-II predicted mortality was virtually identical to that of observed mortality in the low and medium risk groups, the model lost its predictive accuracy in the high-risk subgroup. The EuroSCORE-II model however, performed extremely well in calibration test except in the higher risk subgroup, which might be explained by the fact that there was no mortality in this subgroup and since the study design included only 1098 patients, no significant conclusion can be drawn from this observation.

Analysis of ROC curves yields results from areas under the curve, which are in fairly good agreement with those reported in the literature. With regard to mortality, the highest predictive value was calculated for the System-97 score followed by the Cleveland score [Figure 3 and Table 3]. This is in keeping with the logical observation that the System-97 score is the most comprehensive score among the five scores compared. Differences between the scores for the areas under the ROC curve were not statistically significant between the System-97 score, the Cleveland score and Euro logistic score (P = 0.23). However, it is important to note that the selected scoring systems in this study gives no information on the minimum required sample size for accurate predictions, therefore, statistical comparisons based on large patient numbers might yield different results. ROC curve plotting for all case mortality revealed that Parsonnet score had a value <0.7 (70%) and, therefore, were disqualified as applicable models [Figure 3 and Table 3]. This may be explained by the fact that Parsonnet score database is now older than 21 years and hence might not be applicable because of the loss of predictive value caused by advances in surgical and medical therapy in the intervening period. It is important to note that, this process would apply to any score system over time, hence, revalidation of score items at regular intervals is warranted.

Table 3.

Area under the curve of mortality of all scoring systems

Area under the curve
Test result variable	Area
Euro II	0.736
Cleveland	0.767
Parsonnet	0.689
System-97	0.775

An interesting observation was revealed when ROC curve was plotted for valve-related mortality. Both EuroSCORE-II in addition to Parsonnet score lost their predictive accuracy (curve area <70%) [Figure 4 and Table 4]. However, System-97 and Cleveland score still had areas of in excess of 70%. This may be explained by the fact that the majority of valvular heart disease in the subcontinent is of rheumatic origin as opposed to degenerative diseases in the western hemisphere. This is also in keeping with the worldwide observation that the EuroSCORE-II model over predicts valve-related mortality.

Table 4.

Area under the curves comparing various scoring systems in valve surgery patients

Area under the curve
Test result variable	Area
Euro II	0.630
Cleveland	0.717
Parsonnet	0.639
System-97	0.717

ROC curve areas for each of the curve subgroups of EuroSCORE-II models revealed a lack of predictive accuracy in the high (curve are 66.4%-EuroSCORE-II, 66.2%-Parsonnet) and very high (curve area 55.0% EuroSCORE-II, 59.2%-Parsonnet). However, System-97 revealed extremely high predictive accuracies across all subgroups (curve area >80%). This difference in predictive accuracy was found to be statistically significant (P < 0.001).

Odds ratio was calculated for all the variables used in the Euro score predictive model and it was shown that well-accepted risk factors such as diabetes mellitus (DM), obesity, unstable angina, and female gender were not significant. However, the predictive values of peripheral vascular disease left ventricular EF elevated serum creatinine, pulmonary arterial hypertension, critical preoperative status, and type of surgery had a significant effect on mortality outcome [Table 2].

Mortality has been referred to as the most important performance indicator in heart surgery and is undoubtedly the benchmark parameter in reporting outcomes. This is due to the fact that mortality being an unambiguous entity is not subject to observer bias. As this investigation was principally directed to address the feasibility of the EuroSCORE-II system in the Indian population, it was demonstrated that in the present study, the system failed both the critical tests of validation namely calibration and discrimination. Analysis of four different score system for our patient population yielded System-97 as having the best predictive value for mortality, this commendable predictive accuracy was maintained in the various subgroups, as well as various disease entities, (CABG, valve surgery, etc.).

LIMITATIONS

The principle limitation of the study is the primary sample size (n = 1098), which is small to draw broad generalized conclusions. Another limitation is the fact that the majority of the patient studied underwent coronary artery bypass surgery with a poor representation of patients in the valvular heart disease and diseases of the thoracic aorta category. This is in part due to the fact that, this is a single institution study conducted in a tertiary care private hospital, which is inaccessible to a majority of valvular heart disease patients coming from poor socioeconomic strata. A way of circumventing this bias would include major government cardiac care hospitals in the ambit of future studies as they cater to the majority of such patients.

The Indian patient is quite different from his western counterpart with respect to his body habitus (majority having body mass index <25) extreme prevalence of DM (>80%), diffuse coronary artery disease, small coronary artery size, more than 50% having functional disability NYHA class III and IV, and rheumatic heart disease being primary pathology in almost all valvular heart disease patients. Hence, the application of a European scoring system may be self-defeating in itself.

CONCLUSIONS

A great deal of efforts has gone into the development of risk models to predict outcomes of cardiac surgical interventions. For a population of patients undergoing operations, the models are fairly effective at predicting outcomes. The biggest drawback of all predictive risk models is that they exhibit less than satisfactory performance at predicting outcomes for individual patients. Risk adjustment to predict individual outcomes is extremely difficult to apply at bedside and patient's specific risk stratification and outcome assessment is still in its infancy. India lags far behind its western counterpart in that it has no national cardiac surgical database and no risk stratification model specific for the Indian population. The present study suggests that the EuroSCORE-II model in its present form is not validated for use in the Indian population. An interesting observation was significantly accurate predictive abilities of the System-97 score.

Research in this field is just beginning, and this study can at best be considered a pilot program to draw up broad guidelines that would propel larger studies into this emerging field. What is needed are specific patient predictors for clinical decision making that is geared to the needs of the Indian cardiac surgical patients, which would accurately predict risk-adjusted outcomes and also measure “quality-of-care.” The logical next step would be development of an Indian risk stratification model that should be weighted with risk factors pertinent to the Indian population and it should be applied across the Indian subcontinent in multiple large government and private hospitals in a fashion similar to the Euro score model development and validated centrally.

Another major advancement in the mortality prediction models would be the integration of intraoperative variables into the prediction tree. The future undoubtedly will see refinements in the risk adjustment methods and increasing use of this technique at all levels of health care delivery including the distribution of health care funds. Cardiothoracic surgeons have an obligation to understand the techniques of risk stratification so as to achieve the ultimate goal of improving patient outcomes and maintaining high professional quality.

Financial support and sponsorship

Nil.

Conflict of interest

There are no conflict of interest.