Skip to main content
PMC home page
European Journal of Cardio-Thoracic Surgery logoLink to European Journal of Cardio-Thoracic Surgery
. 2012 Nov 14;43(6):1146–1152. doi: 10.1093/ejcts/ezs584

Dynamic trends in cardiac surgery: why the logistic EuroSCORE is no longer suitable for contemporary cardiac surgery and implications for future risk models

Graeme L Hickey a, Stuart W Grant b, Gavin J Murphy c, Moninder Bhabra d, Domenico Pagano e, Katherine McAllister a, Iain Buchan a, Ben Bridgewater a,b,*
PMCID: PMC3655624  PMID: 23152436

Abstract

OBJECTIVES

Progressive loss of calibration of the original EuroSCORE models has necessitated the introduction of the EuroSCORE II model. Poor model calibration has important implications for clinical decision-making and risk adjustment of governance analyses. The objective of this study was to explore the reasons for the calibration drift of the logistic EuroSCORE.

METHODS

Data from the Society for Cardiothoracic Surgery in Great Britain and Ireland database were analysed for procedures performed at all National Health Service and some private hospitals in England and Wales between April 2001 and March 2011. The primary outcome was in-hospital mortality. EuroSCORE risk factors, overall model calibration and discrimination were assessed over time.

RESULTS

A total of 317 292 procedures were included. Over the study period, mean age at surgery increased from 64.6 to 67.2 years. The proportion of procedures that were isolated coronary artery bypass grafts decreased from 67.5 to 51.2%. In-hospital mortality fell from 4.1 to 2.8%, but the mean logistic EuroSCORE increased from 5.6 to 7.6%. The logistic EuroSCORE remained a good discriminant throughout the study period (area under the receiver-operating characteristic curve between 0.79 and 0.85), but calibration (observed-to-expected mortality ratio) fell from 0.76 to 0.37. Inadequate adjustment for decreasing baseline risk affected calibration considerably.

DISCUSSIONS

Patient risk factors and case-mix in adult cardiac surgery change dynamically over time. Models like the EuroSCORE that are developed using a ‘snapshot’ of data in time do not account for this and can subsequently lose calibration. It is therefore important to regularly revalidate clinical prediction models.

Keywords: Patient trends, Risk model, Cardiac surgery, Model expiry, EuroSCORE

INTRODUCTION

The EuroSCORE models have been the predominant risk-prediction models used in European cardiac surgery since the original additive version was published in 1999 [1]. They were developed using a pan-European database that collected data for a 3-month period in 1995 [2]. The additive model was initially assessed in a number of different countries and found to be valid [3–5]. Subsequently, the performance of the additive model, particularly for high-risk patients, was questioned. To address this issue, the full logistic version of the EuroSCORE was published in 2003 [6]. The logistic model had the advantage of improved calibration, particularly for high-risk groups [7].

Currently, both the additive and logistic EuroSCORE models significantly over-predict the risk of in-hospital mortality for cardiac surgery [8, 9]. A recent systematic review demonstrated that both models were poorly calibrated, with an aggregated predicted mortality double that of observed mortality [9]. This ‘calibration drift’ has important implications for both clinical decision-making and clinical governance analyses that utilize the original EuroSCORE models [10]. To address this issue, EuroSCORE II has been developed using data from 22 381 patients who underwent surgery during 2010 [11]. However, the systematic calibration drift of the original EuroSCORE models has not been fully explored. The objective of this study was to investigate the calibration drift of the original EuroSCORE models as this may have important implications for the EuroSCORE II model (and other similarly produced risk models).

MATERIALS AND METHODS

Data collection, validation and cleaning

Prospectively collected data for all cardiac surgery procedures performed in England and Wales between 1st April 2001 and 31st March 2011 were extracted from the Society for Cardiothoracic Surgery (SCTS) in Great Britain and Ireland database. Any records with the procedure date missing were excluded. Duplicate records were removed, transcriptional discrepancies re-coded and clinical and temporal conflicts resolved. At this stage, and prior to analysis, data for the last 3 years were returned to each contributing hospital for local validation. Data, prior to 1st April 2008, were previously reviewed by contributing hospitals for the preparation of the National Adult Cardiac Surgical Database Report [12].

In-hospital mortality was defined as death due to any cause during the base hospital admission for cardiac surgery. Missing and conflicting data for in-hospital mortality status were back-filled and validated via record linkage to the Office for National Statistics (ONS) census database, which records details of all deaths in England and Wales. Any remaining records with missing in-hospital mortality status were excluded. Within-admission re-do procedure records were also removed. Definitions of database variables used for the study are available at http://www.ucl.ac.uk/nicor/audits/Adultcardiacsurgery/datasets.

Analysis

For the purpose of analysing trends, we assumed that a missing record of a binary or categorical risk factor indicated that the risk factor was not present, i.e. equivalent to the reference level. For any record of missing patient age at the time of surgery, the median patient age for the corresponding financial year was imputed. The percentage of missingness year-on-year for each risk factor in the logistic EuroSCORE was calculated as a general indicator of data quality. In the case of some risk factor data, for example, pulmonary hypertension, which is included in the EuroSCORE models as a binary risk factor, continuous measurements were not recorded. Instead, a nominal value is included where it is considered to be normal.

Following this imputation-based approach, a time series of mean observed and expected mortality was plotted for each time quarter. Q1 denotes 1st January–31st March, Q2 denotes 1st April–30th June, Q3 denotes 1st July–30th September and Q4 denotes 1st October–31st December. Expected mortality is calculated according to the logistic EuroSCORE [6]. The Agresti-Coull approximation was used to calculate a 95% confidence interval (CI) for the observed mortality during a single period [13]. The linear trajectory of the observed and expected mortality proportions against time is indicated by means of overlaid fitted linear regression models (with time [annual quarter] replaced by an integer sequence 0, 1, 2, …, 39). A non-parametric locally weighted smoothing (LOESS) fitted curve was used to examine the trend in observed-to-expected (O:E) ratios over annual quarters [14]. Back-extrapolation of the linear regression fits was used to approximate when the O:E ratio was 1.

Discriminatory ability of the logistic EuroSCORE was measured using the area under the receiver-operating characteristic (ROC) curve (AUC). For each annual quarter, the AUC and its 95% CI was calculated using the method of DeLong et al. [15]. An AUC between 0.7 and 0.8 is generally considered to indicate good discrimination, with AUCs >0.8 indicating excellent discrimination.

Time series plots of logistic EuroSCORE risk factor means aggregated over annual quarters were produced with the exception of age, which was treated as a continuous variable and not transformed as in the logistic EuroSCORE. Similarly, time series were developed for four clinical classes of procedures collected by the SCTS that affect the operative risk factors included in the logistic EuroSCORE: coronary artery bypass graft (CABG) surgery, valve surgery, major aortic surgery and other cardiothoracic surgery. Here, other cardiothoracic surgery includes left ventricular aneurysmectomy, atrial myxoma surgery, pulmonary embolectomy, epicardial pacemaker placement, pericardectomy, atrial septal defect closure, procedure for congenital conditions, acquired ventricular septal defect closure, pulmonary endarterectomy, atrial fibrillation ablation, myomectomy, carotid endarterectomy, peripheral vascular procedures and any other cardiothoracic surgery not listed above. Due to missing data in procedure-specific fields, a refined trends analysis of actual procedures (e.g. mitral valve replacement) was inappropriate. Again, LOESS curves were fitted to characteristic time series models to extract and summarize general trends [14].

To gauge which of the risk factors contributed the most to the change in logistic EuroSCORE prediction of in-hospital mortality year-on-year, all other covariates were held at their Q2-2001 level to allow the variable of interest to change as observed over time. This was measured by multiplying the static log odds ratio (specified by the logistic EuroSCORE model) for the risk factor by the arithmetic difference in means of the risk factor for each time point with reference to Q2-2001. To gauge the loss in calibration, for each annual quarter, a logistic regression between the observed outcome and log-odds transformed logistic EuroSCORE for each record is performed [16]. This yields an intercept and slope coefficient for each time period. In the case of perfect calibration, the intercept and slope will equal 0 and 1, respectively. The intercepts, slopes and logistic calibration curves (which are functions of the intercept and slope coefficients) are plotted to identify any systematic trends.

All data cleaning and analyses were performed using the R statistical computing software (http://www.R-project.org/) Version 2.14.2 [17] and the ggplot2 package for exploratory data analysis Version 0.9.2.1 [18].

RESULTS

In total, there were 318 679 records from 37 hospitals matching the geographical and temporal inclusion criteria. Of these records, 207 were excluded due to missing data on whether the patient was alive at discharge, and a further 1180 records excluded on the basis that they corresponded to within-admission re-do procedures. In total, there remained 317 292 records available for this study. The overall degree of missingness in the EuroSCORE risk factors was 1.7%. The greatest degree of missingness during any annual quarter was for pulmonary hypertension during Q2-2001 (10.6%). As shown in Fig. 1, the degree of missingness for all EuroSCORE risk factors has remained consistent or decreased overall with the notable exception of recent myocardial infarction. There were only nine missing records of patient age during the entire study period (<0.003%).

Figure 1:

Figure 1:

Open in a new tab

Percentage of missing data year-on-year for each EuroSCORE risk factor. LV: left ventricular; MI: myocardial infarction; CABG: coronary artery bypass graft; VSD: ventricular septal defect.

During Q2-2001, the observed mortality was 4.1% (95% CI 3.6–4.6%) and the expected mortality was 5.6%, giving an O:E ratio of 0.73. At the time of the publication of the logistic EuroSCORE (Q2-2003), the observed mortality proportion was 3.5% and the expected mortality was 5.9% (O:E ratio 0.60). For the final time period included in this study (Q1-2011), the observed mortality was 2.8% (95% CI 2.4–3.2%) and the expected mortality was 7.6% (O:E ratio 0.37). As shown in Fig. 2 (top panel), the observed and expected mortality have been diverging at an approximately linear rate (observed mortality slope = −2.08 × 10−4 [P < 0.001]; expected mortality slope = 6.01 × 10−4; [P < 0.001]). Furthermore, the magnitude of the trajectory is greater for the expected mortality than for the observed mortality (absolute ratio of slopes for O:E mortality 2.89). Back-extrapolation estimated Q2-1996 as the period where the logistic EuroSCORE was on average perfectly calibrated, coinciding approximately with the EuroSCORE data collection period. The overall O:E for each annual quarter, as shown in Fig. 2 (bottom panel), demonstrates that the O:E curve has been linearly decreasing to approximately 2007, at which point the rate of O:E decline slows. Model discrimination has consistently been ‘excellent’ over time, as shown in Fig. 3; the minimum AUC was 0.79 (Q2-2002) and the maximum was 0.85 (Q4-2001).

Figure 2:

Figure 2:

Open in a new tab

(Top) The observed mortality proportion and expected mortality proportion according to the logistic EuroSCORE. (Bottom) O:E mortality ratio (as determined by the logistic EuroSCORE model) ratios calculated for each annual quarter.

Figure 3:

Figure 3:

Open in a new tab

AUC for the logistic EuroSCORE over time (95% CI, vertical black lines).

The trends for the EuroSCORE risk factors by annual quarter demonstrate a notable degree of noise; however, the overlaid LOESS smoothing curves demonstrate that there have been changes to most of the risk factors over time (Figs 4 and 5). Average patient age at the time of the procedure has increased linearly over the study period by a total of 2.6 years (mean of 64.6 years in Q2-2001 to mean of 67.2 years in Q1-2011). There have also been approximately increasing linear trends in the proportion of female patients, patients with chronic pulmonary disease, active endocarditis, critical preoperative state, neurological dysfunction and recent myocardial infarction. There has been a decreasing linear trend in the proportion of emergency procedures. Non-linear and non-monotonic trends were observed for previous cardiac surgery, unstable angina requiring intravenous nitrates, serum creatinine >200 μmol/l and poor left ventricular function. It should be noted that not all changes are of clinical significance, regardless of whether they are statistically significant, and care should be taken in reading the scale of the ordinate axes. As shown in Fig. 6, over the study period, there has been an increase in the proportion of valve procedures that involve major aortic surgery and surgery that involves other cardiothoracic procedures. In contrast, the proportion of isolated CABG procedures has decreased (from 67.5 in Q2-2001 to 51.2% in Q1-2011).

Figure 4:

Figure 4:

Open in a new tab

Time series (red lines) and fitted trend curves (blue curves) to patient characteristic risk factors included in the logistic EuroSCORE.

Figure 5:

Figure 5:

Open in a new tab

Time series (red lines) and fitted trend curves (blue curves) to patient characteristic risk factors included in the logistic EuroSCORE. LV: left ventricular; MI: myocardial infarction; CABG: coronary artery bypass graft; VSD: ventricular septal defect.

Figure 6:

Figure 6:

Open in a new tab

Time series (red lines) and fitted trend curves (blue curves) to procedure classes that affect the operative risk factors included in the logistic EuroSCORE.

The measure of contribution of each EuroSCORE risk factor to the overall trend of increasing expected mortality is shown in Fig. 7 and demonstrates that two of the risk factors have a stronger association with the increase in risk score than the others, namely age and ‘procedures other than isolated CABG’. However, even the largest change (for patient age) only yields an increase of approximately 0.15 to the risk score, which corresponds to an increase of 0.16 and 0.76% for patients with an initial risk of 1 and 5%, respectively. This would imply drift is either independent of risk factor dynamics or stems from a compounding effect. Decreasing trends in both the calibration intercept and slope coefficients are shown in Fig. 8 (left panels). It is useful to note that the change in intercept coefficients is measured on the same units, as shown in Fig. 7. Over the study period, the intercept fell from −0.41 to −1.36 (a decrease of 0.95). At the start of the study period, the slope was 0.99, dropping to 0.91 (a decrease of 0.08) at the end of the study period. The calibration curves (Fig. 8; right panel) confirm that model calibration has progressively departed from optimum calibration.

Figure 7:

Figure 7:

Open in a new tab

Change in EuroSCORE logistic regression linear determinants demonstrating that patient age and surgery other than isolated CABG have a stronger association with the increase in risk score than the others. LV: left ventricular; MI: myocardial infarction; CABG: coronary artery bypass graft; VSD: ventricular septal defect. Age is recorded as per the EuroSCORE transformation.

Figure 8:

Figure 8:

Open in a new tab

(Left) Time series plots (red lines) of calibration intercept and slope parameters and fitted trend curves (blue curves). (Right) Calibration curves for each annual quarter. Black dashed line shows perfect calibration.

DISCUSSION

This study demonstrates that there has been a systematic deterioration in the calibration of the logistic EuroSCORE model for cardiac surgery in England and Wales. While the ability of the model to discriminate between patients of low and high risk has remained very good, the calibration of the model has drifted such that, for all cardiac surgery in 2011, the logistic EuroSCORE predicts more than double the observed in-hospital mortality. This finding is consistent with a recent systematic review of EuroSCORE performance [9]. The systematic drift in calibration is a result of a steady fall in the observed mortality despite significant changes in the incidence of several important risk factors contributing to the original logistic EuroSCORE calculation. These findings have important implications for the use of cardiac surgery risk models for patient consent, clinical decision-making and governance processes.

Over the past 10 years, there have been important changes to both cardiac surgery patient risk factors and the case-mix facing cardiac surgeons. Some of these trends are approximately linear, while others are non-linear. There has been a steady rise in the mean age of patients undergoing cardiac surgery, which is now 67.2 years. Increases in the frequency of certain risk factors may represent an increased willingness of cardiac surgeons to take on higher-risk patients because of improvements in anaesthetic, surgical and postoperative critical care. With regard to the case-mix, the proportion of isolated CABG procedures has fallen by an absolute value of 16% and now represents just over half of all procedures performed. The year-on-year decrease in the proportion of CABG surgery is likely to be a direct result of the increase in the use of percutaneous coronary interventions [19]. Alongside the fall in the proportion of CABG surgery, there has been an associated rise in the proportion of valve surgery, major aortic surgery and other cardiothoracic procedures.

Despite the changes in case-mix and patient risk factors indicating that, relative to the original EuroSCORE data, there has been an increase in the number of higher-risk patients and procedures, the observed mortality has significantly fallen. This has contributed to the calibration drift because the logistic EuroSCORE model's intercept parameter has remained static. There are multiple potential reasons for this improvement in observed mortality, including the measurement and publication of UK cardiac surgery outcomes that has occurred during the study period. The SCTS has published named mortality rates for cardiac surgery by hospital since 2001 and by surgeon since 2005 [20]. There is evidence that such publications drive improvements in healthcare [21, 22]. Improvements in training, anaesthesia, critical care and surgical techniques will also have played a vital role in improving the safety of cardiac surgery and contributing to the fall in observed mortality.

Differences between the observed and expected mortality based on the EuroSCORE models, described here as calibration drift, have been extensively noted in the literature [9]. This study demonstrates that, by the time, the logistic EuroSCORE had been published in 2003, a significant calibration drift had already occurred in England and Wales and, therefore, the model may have inappropriately over-predicted risk since its publication. This is perhaps not surprising, given that the development of the additive and full logistic EuroSCORE model was based on a ‘snapshot’ of patient and operative characteristics in Europe during 1995 [2]. There is already some evidence that the EuroSCORE II may already over-predict mortality for isolated CABG surgery [23].

Calibration drift has important implications for clinical practice even if model discrimination remains acceptable. A risk-prediction model that over-predicts risk may result in falsely reassuring conclusions about performance [10]. A model that over-predicts risk may also result in an inappropriate (and excessive) risk for conventional surgery being quoted to patients, leading to a choice of novel treatments such as transcatheter aortic valve implantation. It is possible that patients may even be denied surgery because of an overstated perception of high surgical risk. As logistic EuroSCORE and EuroSCORE II are both general cardiac surgery models, trends were studied in the general cardiac surgery population. However, it may be the case that trends are being driven by dynamics within the specific population subgroups, for example, procedure types or patient characteristics.

This study has been performed using a national clinical dataset that includes all National Health Service hospitals and some private hospitals, performing cardiac surgery in England and Wales. The data included are validated on a regular basis by each contributing hospital. As with the majority of clinical studies, some data were missing, which is a limitation of this study. At the start of the study period, database missingness was more common and, therefore, the intra-risk factor variation for some variables might be heterogeneous.

A further limitation of this study was that observed noise in the database was not studied. The decision to aggregate time into annual quarters as opposed to months was made to reduce variation from low-incidence risk factors. The remaining variation might be explained by a multitude of factors including missing data, errors in the data inputting or upload and seasonal or natural variation within a homogeneous population. No obvious seasonality pattern was apparent from this analysis; however, it has recently been shown that there may be a seasonal effect in the EuroSCORE II model [24]. The intra-quarter, intra- and inter-hospital variation were not quantified, but this was beyond the scope of the present study. The contribution of risk factors conditional upon patient subgroups was also not studied. This is because the EuroSCORE models do not include any interaction terms, meaning that the contributions of risk factors are equal across all patient subgroups.

To the authors' knowledge, this is the first in-depth analysis of the changes in the logistic EuroSCORE risk factors and model performance over time. Although the logistic EuroSCORE has now been replaced by the EuroSCORE II and is no longer recommended for contemporary cardiac surgery, this study has important implications for EuroSCORE II and other risk models derived from ‘snapshots’ of data. If the dynamic trends demonstrated in this study for cardiac surgery continue then it is inevitable that EuroSCORE II will also become miscalibrated for all cardiac surgery over the coming years. It is therefore essential that static models for clinical risk prediction should undergo regular revalidation to ensure acceptable contemporary calibration, and when this calibration crosses predefined thresholds, model update methodology be applied. Alternative statistical approaches for clinical risk prediction that might be considered in the future. Such as machine learning, artificial neural networks and data querying methods over arbitrary periods of time will not be immune to a dynamical data-generating mechanism. A Bayesian dynamical modelling approach is one approach that would be adaptive to the issue of calibration drift and could replace the default method of periodic recalibration; however more research is required before any recommendations can be made.

Funding

This research was partly funded by Heart Research UK (Heart Research UK Grant RG2583) and the National Institute of Health Research through the Greater Manchester CLAHRC project.

Conflict of interest: none declared.

ACKNOWLEDGEMENTS

The authors acknowledge all members of the SCTS in Great Britain and Ireland who contribute data to the SCTS database. We also thank Norman Stein (Northwest Institute for Bio-Health Informatics) for his assistance in cleaning the SCTS database. The National Institute for Clinical Outcomes Research, UCL London, provided the data for this study.

REFERENCES

  • 1.Nashef SAM, Rogues F, Michel P, Gauducheau E, Lemeshow S, Salamon R. European system for cardiac operative risk evaluation (EuroSCORE) Eur J Cardiothorac Surg. 1999;16:9–13. doi: 10.1016/s1010-7940(99)00134-7. doi:10.1016/S1010-7940(99)00134-7. [DOI] [PubMed] [Google Scholar]
  • 2.Roques F, Nashef SAM, Michel P, Gauducheau E, de Vincentiis C, et al. Risk factors and outcome in European cardiac surgery: analysis of the EuroSCORE multinational database of 19030 patients. Eur J Cardiothorac Surg. 1999;15:816–22. doi: 10.1016/s1010-7940(99)00106-2. doi:10.1016/S1010-7940(99)00106-2. [DOI] [PubMed] [Google Scholar]
  • 3.Nashef SAM, Roques F, Hammill BG, Peterson ED, Michel P, Grover FL, et al. Validation of European system for cardiac operative risk evaluation (EuroSCORE) in North American cardiac surgery. Eur J Cardiothorac Surg. 2002;22:101–5. doi: 10.1016/s1010-7940(02)00208-7. doi:10.1016/S1010-7940(02)00208-7. [DOI] [PubMed] [Google Scholar]
  • 4.Gogbashian A, Sedrakyan A, Treasure T. EuroSCORE: a systematic review of international performance. Eur J Cardiothorac Surg. 2004;25:695–700. doi: 10.1016/j.ejcts.2004.02.022. doi:10.1016/j.ejcts.2004.02.022. [DOI] [PubMed] [Google Scholar]
  • 5.Geissler HJ, Holzl P, Marohl S, Kuhn-Regnier F, Mehlhorn U, Sudkamp M, et al. Risk stratification in heart surgery: comparison of six score systems. Eur J Cardiothorac Surg. 2000;17:400–5. doi: 10.1016/s1010-7940(00)00385-7. doi:10.1016/S1010-7940(00)00385-7. [DOI] [PubMed] [Google Scholar]
  • 6.Roques F, Michel P, Goldstone AR, Nashef SAM. The logistic EuroSCORE. Eur Heart J. 2003;24:881–2. doi: 10.1016/s0195-668x(02)00799-6. doi:10.1016/S0195-668X(02)00801-1. [DOI] [PubMed] [Google Scholar]
  • 7.Michel P, Roques F, Nashef SAM EuroSCORE Project Group. Logistic or additive EuroSCORE for high-risk patients? Eur J Cardiothorac Surg. 2003;23:684–7. doi: 10.1016/s1010-7940(03)00074-5. doi:10.1016/S1010-7940(03)00074-5. [DOI] [PubMed] [Google Scholar]
  • 8.Choong CK, Sergeant P, Nashef SAM, Smith JA, Bridgewater B. The EuroSCORE risk stratification system in the current era: how accurate is it and what should be done if it is inaccurate? Eur J Cardiothorac Surg. 2009;35:59–61. doi: 10.1016/j.ejcts.2008.10.009. doi:10.1016/j.ejcts.2008.10.009. [DOI] [PubMed] [Google Scholar]
  • 9.Siregar S, Groenwold RHH, de Heer F, Bots ML, van der Graaf Y, et al. Performance of the original EuroSCORE. Eur J Cardiothorac Surg. 2012;41:746–54. doi: 10.1093/ejcts/ezr285. doi:10.1093/ejcts/ezr285. [DOI] [PubMed] [Google Scholar]
  • 10.Grant SW, Grayson AD, Jackson M, Au J, Fabri BM, Grotte G, et al. Does the choice of risk-adjustment model influence the outcome of surgeon-specific mortality analysis? A retrospective analysis of 14 637 patients under 31 surgeons. Heart. 2008;94:1044–9. doi: 10.1136/hrt.2006.110478. [DOI] [PubMed] [Google Scholar]
  • 11.Nashef SAM, Roques F, Sharples LD, Nilsson J, Smith C, Goldstone AR, et al. EuroSCORE II. Eur J Cardiothorac Surg. 2012;41:734–45. doi: 10.1093/ejcts/ezs043. doi:10.1093/ejcts/ezs043. [DOI] [PubMed] [Google Scholar]
  • 12.Bridgewater B, Keogh B, Kinsman R, Walton P. Henley-on-Thames: Dendrite Clinical Systems Ltd.; 2009. Sixth National Adult Cardiac Surgical Database Report. [Google Scholar]
  • 13.Agresti A, Coull BA. Approximate is better than ‘exact’ for interval estimation of binomial proportions. American Statistician. 1998;52:119–26. [Google Scholar]
  • 14.Cleveland WS, Devlin SJ. Locally weighted regression—an approach to regression-analysis by local fitting. J Am Stat Assoc. 1988;83:596–610. doi:10.1080/01621459.1988.10478639. [Google Scholar]
  • 15.DeLong ER, DeLong DM, Clarkepearson DI. Comparing the areas under 2 or more correlated receiver operating characteristic curves—a nonparametric approach. Biometrics. 1988;44:837–45. doi:10.2307/2531595. [PubMed] [Google Scholar]
  • 16.Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21:128–38. doi: 10.1097/EDE.0b013e3181c30fb2. doi:10.1097/EDE.0b013e3181c30fb2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.R Development Core Team. Vienna, Austria: R Foundation for Statistical Computing; 2011. R: A Language and Environment for Statistical Computing. [Google Scholar]
  • 18.Wickham H. ggplot2: Elegant Graphics for Data Analysis. New York: Springer; 2009. [Google Scholar]
  • 19.Ludman PF. Leeds, UK: The NHS Information Centre; 2011. National Audit of Angioplasty Procedures 2010-Procedures Performed between 1st January and 31st December 2009. Report No. IC03020111. [Google Scholar]
  • 20.Bridgewater B. Cardiac registers: the adult cardiac surgery register. Heart. 2010;96:1441–3. doi: 10.1136/hrt.2010.194019. doi:10.1136/hrt.2010.194019. [DOI] [PubMed] [Google Scholar]
  • 21.Khuri SF, Daley J, Henderson WG. The comparative assessment and improvement of quality of surgical care in the Department of Veterans Affairs. Arch Surg. 2002;137:20–7. doi: 10.1001/archsurg.137.1.20. doi:10.1001/archsurg.137.1.20. [DOI] [PubMed] [Google Scholar]
  • 22.Fung CH, Lim YW, Mattke S, Damberg C, Shekelle PG. Systematic review: the evidence that publishing patient care performance data improves quality of care. Ann Intern Med. 2008;148:111–23. doi: 10.7326/0003-4819-148-2-200801150-00006. [DOI] [PubMed] [Google Scholar]
  • 23.Grant SW, Hickey GL, Dimarakis I, Trivedi U, Bryan A, Treasure T, et al. How does EuroSCORE II perform in UK cardiac surgery; an analysis of 23 740 patients from the Society for Cardiothoracic Surgery of GB and Ireland National database. Heart. 2012;98:1568–72. doi: 10.1136/heartjnl-2012-302483. [DOI] [PubMed] [Google Scholar]
  • 24.Poullis M, Fabri B, Pullan M, Chalmers J. Sampling time error in EuroSCORE II. Interact CardioVasc Thorac Surg. 2012;14:640–1. doi: 10.1093/icvts/ivs034. doi:10.1093/icvts/ivs034. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from European Journal of Cardio-Thoracic Surgery : Official Journal of the European Association for Cardio-thoracic Surgery are provided here courtesy of Oxford University Press

RESOURCES