Accuracy of two scoring systems for risk stratification in thoracic surgery

Abstract

We investigate the suitability of the two existing risk stratification systems available for predicting mortality in a cohort of patients undergoing lung resection under a single surgeon. Data from the 290 consecutive patients who underwent pulmonary resection between January 2008 and January 2011 were extracted from a prospective clinical data base. In-hospital mortality risk scores are calculated for every patient by using Thoracoscore and ESOS.01 and were compared with actual in-hospital mortality. The receiver operating characteristic (ROC) curve was used to establish how well the systems rank for predicting patient mortality. Actual in-hospital mortality was 3.1% (n = 9). Thoracoscore and ESOS values (mean ± SEM) were 4.93 ± 0.32 and 4.08 ± 0.41, respectively. The area under the ROC curve values for ESOS and Thoracoscore were 0.8 and 0.6, respectively. ESOS was reasonably accurate at predicting the overall mortality (sensitivity 88% and specificity 67%), whereas Thoracoscore was a weaker predictor of mortality (sensitivity 67% and specificity 53%). The ESOS score had better predictive values in our patient population and might be easier to calculate. Because of their low specificity, the use of these scores should be limited to the assessment of outcomes of surgical cohorts, but they are not designed to predict risks for individual patients.

BACKGROUND

Risk stratification is the way to find a measure of risk for a specific patient who is having a specific operation. It can be defined as the ability to predict the outcomes from a given intervention based on the pre-existing risks, i.e. sicker patients are expected to predict worse outcomes for a given intervention than lower-risk patients.

Multiple systems have been developed over the last three decades. None of them will ever perfectly describe patients' risk of death, complications or increased use of resources for cardiac surgery and thoracic surgery. While in different specialties there has been an extensive clinical use of risk stratification (Parsonnet and EuroSCORE in cardiac surgery) [1, 2], the application of scores in thoracic surgery is recent. The commonly used systems in thoracic surgery are Thoracoscore and ESOS [3, 4]. These allow surgeons to work out the risk of dying from particular operations, as long as the patients' medical conditions have been accurately identified.

Risk stratification does not necessarily make the decision whether to perform surgery on an individual patient, but might help to guide decisions towards the management of patients (lung preservation techniques, minimally invasive approaches, prophylactic interventions) in order to reduce risks. Decisions should only be made if the surgeons and patients know the relative risks and benefits of the operations. Large-scale and validated risk-stratification systems may be a precursor to individualized preoperative risk analysis, but must also be fundamental to quality control, the revalidation of both clinicians and institutions and benchmarking between different centres [5, 6].

Both ‘Thoracoscore’ and the ‘ESOS.01’ are slowly becoming part of better thoracic surgical practices, the British Thoracic Society has recently recommended the use of risk scores [3] and ESOS is routinely calculated for patients entered on the European Society of Thoracic Surgeons database. Although both serve the same purpose, the variables and methods used to evaluate risk-adjusted patient outcomes are different. We evaluate the predicted and observed mortality in a high-risk thoracic surgical practice using both Thoracoscore and ESOS.01 systems. The objective of the study was to assess the suitability of two of the existing risk stratification systems available for predicting mortality in a lung resection patient cohort to determine if both of them were really necessary.

PATIENTS AND METHODS

Demographics

A review of a prospective continuous data collection system identified 290 consecutive patients undergoing pulmonary resection cases by a newly qualified thoracic surgeon since appointment from January 2008 until January 2011. The patients [168 males and 122 females, median age 68 (range 37–91) years] underwent pulmonary resection for malignant (n = 259) or benign disease (n = 31). Exploratory thoracotomies (n = 4), lung biopsies for diagnosis of interstitial lung disease or resections for bullous disease are not included in this study. The demographic characteristics of each patient group are shown in Table 1.

Table 1:

Patients' demographics according to final diagnosis

	Primary malignancy (n = 236)	Metastasis (n = 23)	Benign (n = 31)
Age (median, range)	69 (37–91)	67 (45–80)	66 (41–80)
PpoFEV1% (median, range)	51 (19–109)	66 (26–99)	58 (19–103)
Male:female	135:101	17:6	16:15
Thoracoscore (mean)	5.24	5.44	1.16
ESOS (mean)	3.96	2.74	4.34

Preoperative preparation and surgical procedures

All patients with suspected malignant diseases were discussed at a local multidisciplinary team meeting. All patients routinely underwent clinical evaluation, positron emission tomography-computed tomography scan for staging and pulmonary function testing for physiological evaluation. A predicted postoperative FEV₁% (ppoFEV₁) was calculated using the working segment count method and recorded as well as performance status, ASA grade and dyspnoea score. Further staging modalities (brain magnetic resonance imaging, invasive staging procedures) and tests for fitness assessment (echocardiogram, measurement of transfer factor, shuttle walk testing, etc.) were used freely as indicated on a case-to-case basis. All data required to calculate risk scores was available on all patients.

The intention of surgery is to try to obtain complete tumour excision by an anatomical resection, but we do attempt parenchymal-sparing procedures (broncho-angioplastic resections and anatomical segmentectomies) when indicated [7, 8]. The diagnosis was benign (10.7%) and malignant 89.3% (primary lung cancer 81.4% and metastasis in 7.9%). The operative procedures were completion pneumonectomy (1.4%), pneumonectomy (11%), sleeve resection (8.6%), lobectomy/bilobectomy (46.1%), segmentectomy (12.4%) and wedge excision (20.5%) (Table 2).

Table 2:

Operations performed according to final diagnosis

	Primary malignancy (n = 236)	Metastasis (n = 23)	Benign (n = 31)
Completion pneumonectomy	4 (1.7%)	–	–
Pneumonectomy	29 (12.3%)	1 (4.3%)	–
Sleeve resection	23 (9.7%)	1 (4.3%)	1 (3.2%)
Lobectomy	120 (50.8%)	2 (8.7%)	8 (25.8%)
Segmentectomy	32 (13.1%)	-	2 (6.5%)
Wedge (including lung volume reduction)	28 (11.8%)	19 (82.6%)	20 (51.6%)

Statistical analysis

Data were prospectively collected as part of the logbook of the operating surgeon and the unit's database. These data were updated in clinic sessions, preoperative assessment clinics and during the patient's admission, and had been custom-designed for rapid data entry, and to minimize data errors by using built-in calculations and drop-down menus. The receiver operating characteristic (ROC) curve was used for the discrimination analysis. The area under the ROC curve (from 0 to 1.0) correlates with the discriminatory capability of the model. In general, the larger the area under the ROC curve, the better the discriminatory power of that particular model. However, the ROC curve value is valid and meaningful only after the model has been shown to calibrate well. We then compared our observed postoperative mortality with the predicted risk of mortality according to both Thoracoscore and ESOS.01. ROC curve and 95% confidence interval were analysed by Analyse-it (Analyse-it Software Ltd, Leeds, UK).

RESULTS

The overall median hospital stay was 7 (range 3–65) days. The actual in-hospital mortality was 3.1% (n = 9), with four additional patients requiring admission to intensive care unit requiring step-up care (2.1%). Four patients not included in the study underwent exploratory thoracotomy (1.4%) during the study period. Thoracoscore and ESOS.01 predicted a mean mortality (±SEM) of 4.93 ± 0.32 and 4.08 ± 0.41, respectively (Table 3). The 95% confidence interval for Thoracoscore and ESOS was 0.00–0.53 (P = 0.942) and 0.00–0.30 (P = 1.000), respectively.

Table 3:

Observed and predicted mortality according to ESOS and Thoracoscore in the overall group (expressed as mean ± SEM)

Observed mortality	3.1% (n = 9)
ESOS, mean ± SEM	4.08 ± 0.41
Thoracoscore, mean ± SEM	4.93 ± 0.32

Areas under ROC curve values for ESOS and Thoracoscore were 0.8 and 0.6, respectively. ESOS.01 was reasonably accurate at predicting the overall mortality (sensitivity 88% and specificity 67%), whereas Thoracoscore was a weaker predictor of mortality (sensitivity 67% and specificity 53%). Discrimination analysis of the two models using ROC curves is shown in Fig. 1, together with areas under the ROC curves for comparison.

Figure 1:

Comparison of ROC curves of the ESOS (sensitivity 88% and specificity 67%) and Thoracoscore (sensitivity 67% and specificity 53%) models in lung resection patients.

DISCUSSION

Since the first cardiac risk index was published in 1977 [9], cardiac surgeons have been aware of the importance of identifying risk factors that influence postoperative outcomes. The most important factors can be used to provide an estimate of postoperative risk of morbidity and mortality for a patient undergoing a particular procedure, and to evaluate or benchmark practices taking into accounts their risk profiles and difficulties. It can also help patients to give informed consent, help organizations with allocation of resources and aid with the assessment of the overall quality of care [10].

Again, to this end, it appears that consideration of risk profiles in cardiac surgery pre-date their thoracic equivalent by some 10 years. EuroSCORE, a cardiac surgery risk calculator first presented in 1998 and later modified to increase accuracy, claims to predict early mortality in cardiac surgical patients in Europe on the basis of objective risk factors [11]. Information collected on 97 risk factors was collected from nearly 20 000 consecutive patients from 128 hospitals. The outcome (survival or death) was compared with the preoperative risk factors. The most important, reliable and objective risk factors were then used to prepare a scoring system using a part of the dataset and then validated in the remaining patients. This system has been instrumental in advances regarding quality control in cardiac surgery and setting standards for outcomes [5].

We have recently seen the design of a similar risk stratification system for thoracic surgery. The Thoracoscore, created by the French Society of Thoracic and Cardiovascular Surgery, initially claimed predictive power regarding in-hospital mortality following thoracic surgery [12]. It was developed using prospective data from 15 183 patients undergoing thoracic surgery collected between 2002 and 2005 (although only 7480 were pulmonary resections, and most of them wedge excisions for benign disease). Following logistic regression analysis of the data of two-thirds of patients, nine factors were found to be independent predictors of in-hospital death: age, sex, dyspnoea score, ASA score, performance status classification, priority of surgery, malignant histology, type of operation and co-morbid disease. A further report has indicated that it may well predict ‘mid-term’ and ‘in-hospital’ mortality following thoracic surgery [13].

In 2005, the European Society of Thoracic Surgeons (Audit and Guidelines Committee) developed their own risk-stratification tool using their own online database, limiting it to pulmonary resection procedures: European Society Objective Score (ESOS.01) [14]. Derived from the data of 3426 patients, it predicts the risk of in-hospital mortality using only age and FEV₁%. The scoring system has been subsequently applied to three European thoracic surgery units and revealed that the performances of all units were in line with the predicted ones during each period under analysis and did not differ between each other [4]. Both systems, Thoracoscore and ESOS, could face similar criticism in the fact that prediction of mortality has been substracted from data with very few events (deaths).

We intended to assess the outcomes of our practice using these two-purpose specific tools. In our study, ‘in-hospital’ mortality was the chosen end-point. Mortality has been the most commonly used outcome in the majority of the risk-stratification models. In our patients, the results of the calibration and discrimination statistical analyses revealed that the ESOS.01 model performed better than the Thoracoscore model in predicting postoperative mortality. As mentioned already, ROC curve analysis is valid and meaningful only after the model has been shown to calibrate well in the Hosmer–Lemeshow goodness-of-fit test. In practice, the most commonly used risk stratification models have areas under the ROC curve in the range of 0.70–0.85. Nevertheless, 95% CI for thoracoscore is 0.00–0.53 and ESOS is 0.00–0.30, resulting from a limited number of patients.  We demonstrated that the ESOS.01 model performed better than the Thoracoscore model in terms of postoperative mortality prediction in the cancer population undergoing thoracic surgery. The difference between the observed and expected mortality rates provides an outcome-based measure of quality of care and provides a sound baseline for financial and human resource allocation by healthcare administrators.

There are limitations in our study and methodology that warrant consideration. Our series is small (hence with a very low incidence of events) and include some particularly high-risk patients over a limited time period. Our cohort of patients belongs to a practice that achieves resection rates of 20–30%, which is high by United Kingdom standards [15]. While comparing data with other series or datasets, there is always the limiting bias of using different definitions (for example in the definition of co-morbidities).

We have not intended to apply complex statistical analysis to validate the two stratification tools, because the purpose of this study was not to criticize the tools that represent great advances for our practices and our patients' care. We intended to show how they apply to day-to-day practice and, in doing so, highlight the uncertainty of using these tools to predict mortality risk in individual patients.

CONCLUSION

In summary, although both Thoracoscore and ESOS.01 appear to be reasonable prototypes and represent seminal ways to identify the risk for patients undergoing pulmonary resection, ESOS.01 appears to be more efficient in assessing risks of a high-risk population—the elderly, the breathless and the cardiovascularly morbid. Its use should therefore be prioritized over Thoracoscore, which is more complex to calculate (as it requires more variables), and which obviates respiratory reserve as a factor. We believe that these systems should not be used to deny access to surgery to certain patients as they are not intended to individually predict patient risk. They should solely be used to assess outcomes of entire practices, and the results in a high-risk cohort of patients or series with a complex profile should be taken cautiously. Another use would be to characterize a cohort of patients in order to facilitate benchmarking and comparisons.

Conflict of interest: none declared.