Abstract
Background.
The recently developed American College of Cardiology Foundation-Society of Thoracic Surgeons (STS) Collaboration on the Comparative Effectiveness of Revascularization Strategy (ASCERT) Long-Term Survival Probability Calculator is a valuable addition to existing short-term risk-prediction tools for cardiac surgical procedures but has yet to be externally validated.
Methods.
Institutional data of 654 patients aged 65 years or older undergoing isolated coronary artery bypass grafting between 2005 and 2010 were reviewed. Predicted survival probabilities were calculated using the ASCERT model. Survival data were collected using the Social Security Death Index and institutional medical records. Model calibration and discrimination were assessed for the overall sample and for risk-stratified subgroups based on (1) ASCERT 7-year survival probability and (2) the predicted risk of mortality (PROM) from the STS Short-Term Risk Calculator. Logistic regression analysis was performed to evaluate additional perioperative variables contributing to death.
Results.
Overall survival was 92.1% (569 of 597) at 1 year and 50.5% (164 of 325) at 7 years. Calibration assessment found no significant differences between predicted and actual survival curves for the overall sample or for the risk-stratified subgroups, whether stratified by predicted 7-year survival or by PROM. Discriminative performance was comparable between the ASCERT and PROM models for 7-year survival prediction (p < 0.001 for both; C-statistic = 0.815 for ASCERT and 0.781 for PROM). Prolonged ventilation, stroke, and hospital length of stay were also predictive of long-term death.
Conclusions.
The ASCERT survival probability calculator was externally validated for prediction of long-term survival after coronary artery bypass grafting in all risk groups. The widely used STS PROM performed comparably as a predictor of long-term survival. Both tools provide important information for preoperative decision making and patient counseling about potential outcomes after coronary artery bypass grafting.
The Society of Thoracic Surgeons (STS) Risk Calculator is commonly used to facilitate clinical decision making and patient-specific counseling on the expected risk of planned cardiac surgical procedures [1,2]. Because risk modeling has become widely accepted for the prediction of short-term outcomes, efforts are now being made to develop predictive models of long-term postoperative survival. Indeed, the usual desired goal of cardiac surgical procedures is the maximization of long-term survival and symptom improvement while minimizing the risk of short-term morbidity and death.
The first long-term predictive model to be adopted by the STS was recently developed through the American College of Cardiology Foundation-Society of Thoracic Surgeons Collaboration on the Comparative Effectiveness of Revascularization Strategies (ASCERT) trial [3]. The ASCERT Long-Term Survival Probability Calculator provides annual survival estimates for the first 7 years after isolated coronary artery bypass grafting (CABG) in patients aged 65 years or older [2]. The algorithm performed well in the validation arm of model development and has been made available for public use [4] but has yet to be externally validated by independent investigators.
External validation is an important step in the introduction of predictive models to clinical use, because real-world performance of a model may differ significantly from its developmental performance, as was recently shown for transcatheter aortic valve replacement [5, 6] and reoperative aortic valve replacement [7]. Several methodologic limitations of the ASCERT model development highlight the need for independent assessment, including imperfect matching of the STS Adult Cardiac Surgery Database with survival data from the Center for Medicare and Medicaid Services (CMS) database [8].
We therefore sought to independently assess the performance of the ASCERT Long-Term Survival Probability Calculator for isolated CABG in our institutional cardiac surgical population. We also sought to compare the ASCERT model’s performance for long-term survival prediction with that of the predicted risk of mortality (PROM) model from the STS Short-Term Risk Calculator [9]. Although designed for the assessment of operative risk, the STS PROM has also been shown to predict long-term survival after cardiac operations [10] and is already widely used by most cardiac surgical programs.
Patients and Methods
All adult patients aged 65 years or older undergoing isolated CABG at our institution during a 7-year period (January 1, 2005, to December 31, 2010) were retrospectively reviewed. Patients younger than 65 years were excluded because they were not included in the original development of the ASCERT survival model. The Washington University School of Medicine in St. Louis Institutional Review Board approved the study, with a waiver of patient consent.
Data Collection and Survival Curves
Patient data were collected from our institutional STS Adult Cardiac Surgery Database, versions 2.52 and 2.61 [11]. The online ASCERT survival probability calculator was used to calculate predicted survival probabilities at 90 days, 180 days, and annually for 7 years after the operation for all patients [7]. Mean survival probabilities were calculated for patients who achieved actual follow-up at each time point. Predicted survival curves were constructed based on the mean probability of survival over time.
Actual survival data were collected by review of the Social Security Death Index and institutional medical records. The study end point for each patient was the date of death or the last known date of living follow-up. Mortality status at 30 days was known for all patients. Patients without a known date of death were considered censored for all time points beyond their last living follow-up and were removed from analysis for time points beyond that date (Table 1). Actual survival curves were then constructed based on the proportion of patients remaining alive relative to all those with known follow-up at each time point.
Table 1.
Details of Obtained Patient Follow-Up and Patient Censoring at Each Postoperative Time Pointa
| Days | Years | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Variable | 90 | 180 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| Expected follow up, No. | 654 | 654 | 654 | 654 | 654 | 654 | 623 | 550 | 454 |
| Actual follow-up obtained, No. | 597 | 591 | 581 | 566 | 545 | 512 | 467 | 402 | 325 |
| Censored, No. | 57 | 63 | 73 | 88 | 109 | 142 | 156 | 148 | 129 |
| Percentage censored, % | 9 | 10 | 11 | 13 | 17 | 22 | 25 | 27 | 28 |
Expected follow-up represents the number of patients who had reached the given time point by the date of data analysis. Patients without a known date of death were considered censored for all time points beyond their last living follow-up.
No. = number.
Overall and Risk-stratified Survival Analysis
Performance evaluation of the ASCERT prediction model was approached through analysis of model calibration and model discrimination [12] (see Comment for more details). Model calibration was assessed by comparison of actual survival curves to ASCERT-predicted survival curves, first for the overall sample and then for risk-stratified subgroups as described below.
Two separate risk-stratified subgroup analyses were performed to better define the performance of the ASCERT survival model across variable levels of patient-related risk. In the first, the overall cohort was stratified into quartiles based on risk of long-term mortality as measured by the ASCERT 7-year survival probability. In the second, the overall cohort was stratified into quartiles based on risk of short-term mortality as measured by the PROM from the STS Short-Term Risk Calculator. STS PROM values were obtained directly from calculated fields in our institutional STS database. In each subgroup analysis, model calibration was assessed by comparing the actual survival curve to the ASCERT-predicted survival curve within each quartile.
Logistic Regression Analysis
Discrimination of the ASCERT model was assessed using logistic regression analysis with calculation of the area under the receiver operating characteristic curves (C statistics). Specifically, predictive performance of the ASCERT 1-year and 7-year predicted survival probability outputs were compared with actual 1-year and actual 7-year survival data, respectively, by logistic regression analysis. Separate logistic regression analyses were performed to assess the predictive performance of the STS PROM compared with actual 1-year and actual 7-year survival, and C statistics were compared with the ASCERT model results.
Statistical Analysis
Continuous variables are expressed as mean ± SD or as median with the interquartile range. Categoric variables are expressed as frequencies and percentages. As described above, because the data output from the ASCERT model was in the format of predicted probabilities of survival to each time point, the actual survival data were converted to proportional survival at each corresponding time point after manual censoring of those patients lost to follow-up. The depicted survival curves therefore represent a comparison of the actual proportion of followed-up patients who were alive at each time point compared with the mean predicted probability of survival for followed-up patients at each time point. Because of the proportional nature of the survival data, survival curves were compared using χ2 analysis.
Logistic regression analyses were performed as described above. A separate stepwise multivariate logistic regression analysis was performed to identify additional perioperative variables influencing long-term survival. Data analyses were performed using Systat 13 software (Systat Software Inc, San Jose, CA).
Results
Patient Characteristics
The study cohort consisted of 654 patients who met inclusion criteria. Details of obtained patient follow-up and censoring of patients lost to follow-up are provided in Table 1. Clinical characteristics of the overall cohort, including all input variables for the ASCERT prediction model, are reported in Table 2. Overall operative mortality was 2.0% (13 of 654), which was similar to the median predicted risk of mortality (PROM, 2.0%; interquartile range, 1.2% to 4.2%; p = 1.0).
Table 2.
Clinical Characteristics of the Overall Patient Cohort and of Risk-Stratified Quartilesa
| Quartile | |||||
|---|---|---|---|---|---|
| Variableb | Overall (N = 654) | 1 (n = 164) | 2 (n = 164) | 3 (n = 163) | 4 (n = 163) |
| Age, y | 73.1 ± 5.9 | 68.7 ± 3.3 | 72.2 ± 4.6 | 74.5 ± 5.6 | 77.0 ± 6.3 |
| Weight, kg | 84.6 ± 17.6 | 86.8 ± 16.8 | 84.3 ± 16.2 | 83.5 ± 17.6 | 83.6 ± 19.6 |
| Height, cm | 171.9 ± 10.6 | 173.0 ± 9.8 | 171.5 ± 10.2 | 171.2 ± 11.8 | 171.8 ± 10.6 |
| Creatinine (most recent), mg/dL | 1.22 ± 0.96 | 0.95 ± 0.21 | 0.99 ± 0.27 | 1.21 ± 0.98 | 1.76 ± 1.49 |
| Ejection fraction (most recent) | 0.509 ± 0.140 | 0.575 ± 0.08 | 0.531 ± 0.119 | 0.494 ± 0.149 | 0.434 ± 0.160 |
| Mean aortic gradient, mm Hg | 14.9 ± 0.9 | 14.9 ± 0.9 | 15.0 ± 0.8 | 15.0 ± 1.0 | 15.0 ± 1.0 |
| Sex | |||||
| Male | 467 (71.4) | 129 (78.7) | 113 (68.9) | 114 (69.9) | 111 (68.1) |
| Female | 187 (28.6) | 35 (21.4) | 51 (31.1) | 49 (30.1) | 52 (31.9) |
| Ethnicity | |||||
| White | 583 (89.1) | 149 (90.9) | 150 (91.5) | 144 (88.3) | 140 (85.9) |
| Black | 60 (9.2) | 7 (4.3) | 14 (8.5) | 17 (10.4) | 22 (13.5) |
| Hispanic | 2 (0.3) | 2 (1.2) | 0 (0) | 0 (0) | 0 (0) |
| Asian | 6 (0.9) | 4 (2.4) | 0 (0) | 1 (0.6) | 1 (0.6) |
| Other | 3 (0.5) | 2 (1.2) | 0 (0) | 1 (0.6) | 0 (0) |
| Dialysis (current) | 15 (2.3) | 0 (0) | 0 (0) | 1 (0.6) | 14 (8.6) |
| Diabetes | |||||
| Insulin dependent | 101 (15.4) | 4 (2.4) | 18 (11.0) | 32 (19.6) | 47 (28.8) |
| Noninsulin dependent | 172 (26.3) | 36 (22.0) | 50 (30.5) | 44 (27.0) | 42 (25.8) |
| Chronic lung disease (moderate-severe) | 36 (5.5) | 0 (0) | 8 (4.9) | 6 (3.7) | 22 (13.5) |
| Cerebrovascular disease | |||||
| Carotid occlusion >75% | 20 (3.1) | 2 (1.2) | 5 (3.0) | 7 (4.3) | 6 (3.7) |
| Previous carotid operation | 32 (4.9) | 2 (1.2) | 4 (2.4) | 14 (8.6) | 12 (7.4) |
| CVA or TIA | 103 (15.7) | 2 (1.2) | 18 (11.0) | 34 (20.9) | 49 (30.1) |
| Immunosuppressive therapy | 23 (3.5) | 2 (1.2) | 2 (1.2) | 6 (3.7) | 13 (8.0) |
| Peripheral vascular disease | 181 (27.7) | 16 (9.8) | 32 (19.5) | 54 (33.1) | 79 (48.5) |
| Cigarette smoking | |||||
| Current (<1 month ago) | 108 (16.5) | 8 (4.9) | 30 (18.3) | 29 (17.8) | 41 (25.2) |
| Past (>1 month ago) | 318 (48.6) | 75 (45.7) | 77 (47.0) | 80 (49.1) | 86 (52.8) |
| Atrial fibrillation/flutter (≤2 weeks) | 83 (12.7) | 4 (2.4) | 13 (7.9) | 20 (12.3) | 46 (28.2) |
| CHF/NYHA class (≤2 weeks) | |||||
| NYHA class IV | 86 (13.2) | 2 (1.2) | 13 (7.9) | 28 (17.2) | 43 (26.4) |
| NYHA class I-III | 67 (10.2) | 1 (0.6) | 8 (4.9) | 15 (9.2) | 43 (26.4) |
| Prior cardiac operation | |||||
| 1 prior operation | 19 (2.9) | 4 (2.4) | 4 (2.4) | 9 (5.5) | 2 (1.2) |
| ≥2 prior operations | 4 (0.6) | 0 (0) | 1 (0.6) | 1 (0.6) | 2 (1.2) |
| Status | |||||
| Salvage | 1 (0.2) | 0 (0) | 0 (0) | 0 (0) | 1 (0.6) |
| Emergent | 30 (4.6) | 4 (2.4) | 8 (4.9) | 4 (2.5) | 14 (8.6) |
| Urgent | 63 (9.6) | 10 (6.1) | 19 (11.6) | 22 (13.5) | 12 (7.4) |
| Elective | 560 (85.6) | 150 (91.5) | 137 (83.5) | 137 (84.0) | 136 (83.4) |
| Cardiopulmonary resuscitation (≤1 hour) | 1 (0.2) | 0 (0) | 0 (0) | 1 (0.6) | 0 (0) |
| Cardiogenic shock (at time of operation) | 34 (5.2) | 2 (1.2) | 9 (5.5) | 9 (5.5) | 14 (8.6) |
| Diseased coronary vessels, No. | |||||
| One | 29 (4.4) | 19 (11.6) | 5 (3.0) | 1 (0.6) | 4 (2.5) |
| Two | 154 (23.6) | 52 (31.7) | 37 (22.6) | 33 (20.2) | 32 (19.6) |
| Three | 471 (72.0) | 93 (56.7) | 122 (74.4) | 129 (79.1) | 127 (77.9) |
| Left main coronary artery disease | 252 (38.5) | 44 (26.8) | 63 (38.4) | 67 (41.1) | 78 (47.9) |
| Myocardial infarction | |||||
| <6 hours ago | 1 (0.2) | 0 (0) | 0 (0) | 1 (0.6) | 0 (0) |
| 6–24 hours ago | 13 (2.0) | 2 (1.2) | 1 (0.6) | 4 (2.5) | 6 (3.7) |
| 1–7 days ago | 103 (15.7) | 18 (11.0) | 23 (14.0) | 31 (19.0) | 31 (19.0) |
| 8–21 days ago | 84 (12.8) | 11 (6.7) | 13 (7.9) | 23 (14.1) | 37 (22.7) |
| None or >21 days ago | 453 (69.3) | 133 (81.1) | 127 (77.4) | 104 (63.8) | 89 (54.6) |
| Unstable angina | 182 (27.8) | 51 (31.1) | 52 (31.7) | 43 (26.4) | 36 (22.1) |
| Preoperative IABP/inotropes (≤48 hours) | 49 (7.5) | 6 (3.7) | 10 (6.1) | 15 (9.2) | 18 (11.0) |
| PCI (≤6 hours) | 2 (0.3) | 2 (1.2) | 0 (0) | 0 (0) | 0 (0) |
| Hypertension | 561 (85.8) | 134 (81.7) | 146 (89.0) | 136 (83.4) | 145 (89.0) |
| Moderate-severe valve insufficiency | |||||
| Tricuspid valve | 10 (1.5) | 0 (0) | 0 (0) | 1 (0.6) | 9 (5.5) |
| Aortic valve | 8 (1.2) | 1 (0.6) | 0 (0) | 2 (1.2) | 5 (3.1) |
| Mitral valve | 48 (7.3) | 5 (3.5) | 4 (2.4) | 7 (4.3) | 32 (19.6) |
Data are based on American College of Cardiology Foundation-Society of Thoracic Surgeons (STS) Collaboration on the Comparative Effectiveness of Revascularization Strategy (ASCERT) 7-year survival probability (quartile 1 is lowest risk). Table includes all input variables for the ASCERT survival probability calculation.
Data are shown as mean ± SD or number (%).
CHF = congestive heart failure; CVA = cerebrovascular accident; IABP = intraaortic balloon pump; NYHA = New York Heart Association; PCI = percutaneous coronary intervention; TIA = transient ischemic attack.
Overall Survival Analysis
Overall actual survival was 92.1% (535 of 581) at 1 year and 50.5% (164 of 325) at 7 years after the operation (Fig 1), neither of which were significantly different from the mean predicted probability of survival calculated from the ASCERT model at 1 year (90.4% ± 10.1%, p = 0.532) and 7 years (59.2% ± 22.7%, p = 0.240). For the overall sample, there was no difference between the actual long-term survival curve and the ASCERT-predicted survival curve (p = 0.751, Fig 1), illustrating accurate predictive performance of the ASCERT model for overall long-term survival.
Fig 1.
No difference was found between the actual survival curve and the American College of Cardiology Foundation–Society of Thoracic Surgeons Collaboration on the Comparative Effectiveness of Revascularization Strategy (ASCERT) predicted survival curve for the overall sample. Actual survival values represent the percentage of followed-up patients remaining alive at each time point. Predicted survival values represent mean predicted probability of survival to each time point, as calculated by the ASCERT survival model, for patients with follow-up data available at that time point. The error bars represent ±1 SD, and N indicates the number of patients with obtained survival data at each time point.
Risk-Stratified Survival Analysis
In two separate analyses, the overall cohort was stratified into quartiles based on risk of death as measured by the ASCERT 7-year survival probability and the STS PROM. Clinical characteristics of the patients in the ASCERT-stratified quartiles are reported in Table 2, and postoperative outcomes by quartile are reported in Table 3.
Table 3.
Postoperative Outcomes of the Overall Patient Cohort and of Risk-Stratified Quartilesa
| Quartile | |||||
|---|---|---|---|---|---|
| Variableb | Overall (N = 654) | 1 (n = 164) | 2 (n = 164) | 3 (n = 163) | 4 (n = 163) |
| Operative mortality | 13 (2.0) | 0 (0) | 1 (0.6) | 1 (0.6) | 11 (6.7) |
| STS PROM, % | 2.0 (1.2–4.2) | 0.9 (0.7–1.2) | 1.6 (1.2–2.2) | 2.7 (1.9–4.0) | 6.4 (3.8–9.6) |
| Mean ± SD | 3.8 ± 5.4 | 1.0 ± 0.8 | 2.1 ± 1.9 | 3.4 ± 2.5 | 8.4 ± 8.5 |
| 1-year survival | 535/581 (92.1) | 143/145 (98.6) | 146/148 (98.6) | 131/145 (90.3) | 115/143 (80.4) |
| 7-year survival | 164/325 (50.5) | 63/68 (92.6) | 47/78 (60.3) | 37/82 (45.1) | 17/97 (17.5) |
| Hospital length of stay, d | 7.0 (5.0–9.0) | 6.0 (5.0–7.0) | 6.0 (5.0–7.0) | 7.0 (6.0–9.0) | 9.0 (6.0–14.0) |
| Postoperative complications | |||||
| Deep sternal wound infection | 6 (0.9) | 2 (1.2) | 2 (1.2) | 1 (0.6) | 1 (0.6) |
| Renal failure | 28 (4.3) | 3 (1.8) | 7 (4.3) | 5 (3.1) | 13 (8.0) |
| Stroke | 16 (2.4) | 0 (0) | 2 (1.2) | 5 (3.1) | 9 (5.5) |
| Any reoperation | 66 (10.1) | 11 (6.7) | 13 (7.9) | 13 (8.0) | 29 (17.8) |
| Prolonged ventilation | 106 (16.2) | 8 (4.9) | 18 (11.0) | 27 (16.6) | 53 (32.5) |
Data are based on American College of Cardiology Foundation-Society of Thoracic Surgeons (STS) Collaboration on the Comparative Effectiveness of Revascularization Strategy 7-year survival probability (quartile 1 is lowest risk).
Continuous data are shown as median (interquartile range), or as indicated, and categoric data as number (%).
PROM N predicted risk of mortality.
There were no differences between the actual long-term survival curves and the ASCERT-predicted survival curves for any of the quartiles examined, whether stratification was performed by ASCERT or PROM, illustrating the accurate predictive performance of the ASCERT model over the entire range of patient risk (Fig 2). In addition, no differences were found in the actual long-term survival curves when each ASCERT-stratified quartile was compared with its equivalent PROM-stratified quartile, illustrating similar ability of the ASCERT and PROM models to stratify patient risk and long-term survival (Fig 3).
Fig 2.
Survival curves quartiles (Q) for cohort risk-stratified into quartiles by (A) the American College of Cardiology Foundation-Society of Thoracic Surgeons (STS) Collaboration on the Comparative Effectiveness of Revascularization Strategy (ASCERT) 7-year predicted survival probability, or (B) the STS predicted risk of mortality (PROM). Quartile 1 represents the lowest mortality risk group for each model. No differences were found between actual and predicted survival curves for any of the quartiles examined. The N indicates the number of patients with obtained survival data at each time point.
Fig 3.
Actual survival curves are plotted for the two sets of calculated quartiles (Q), one stratified by American College of Cardiology Foundation–Society of Thoracic Surgeons (STS) Collaboration on the Comparative Effectiveness of Revascularization Strategy (ASCERT) 7-year survival probability and one stratified by the STS predicted risk of mortality (PROM). No differences were found between actual survival curves when each ASCERT-stratified quartile was compared with its equivalent PROM-stratified quartile. The N indicates the number of patients with obtained survival data at each time point for PROM/ASCERT quartiles.
Logistic Regression Analysis
Logistic regression analysis was performed to determine whether the ASCERT or PROM models, or both, were predictive of actual survival at 1 year and 7 years after the operation. The ASCERT 1-year survival probability and PROM were both significant predictors of actual 1-year survival (p < 0.001 for both; C statistic = 0.797 for ASCERT and 0.787 PROM). Similarly, the ASCERT 7-year survival probability and PROM were both significant predictors of actual 7-year survival (p < 0.001 for both; C statistic = 0.815 for ASCERT and 0.781 for PROM).
A separate stepwise multivariate logistic regression analysis was performed to assess the additional effect of common postoperative complications (Table 3) as predictors of long-term survival when combined with the ASCERT model predictions. In addition to the ASCERT model output at each time point, independent predictors of death at 1 year after the operation included prolonged ventilation, stroke, and hospital length of stay. Additional independent predictors of death at 7 years after the operation included only prolonged ventilation (Table 4). The addition of postoperative complications and length of stay data to the ASCERT logistic regression analysis increased the C statistic from 0.797 to 0.845 for the 1-year survival model (overall p < 0.001) and from 0.815 to 0.83 for the 7-year survival model (overall p < 0.001).
Table 4.
Additional Independent Predictors of Long-Term Death Determined by Stepwise Multivariate Logistic Regression Analysis
| Variable | Odds Ratio | 95% CI | p Value |
|---|---|---|---|
| Additional 1-year death predictors | |||
| Prolonged ventilation | 3.22 | 1.37–7.6 | 0.007 |
| Stroke | 5.64 | 1.58–20.2 | 0.008 |
| Hospital length of stay (per day) | 1.04 | 1.01–1.07 | 0.017 |
| Additional 7-year death predictors | |||
| Prolonged ventilation | 3.49 | 1.63–7.45 | 0.001 |
CI = confidence interval.
Comment
Clinical prediction models are increasingly prevalent tools used to estimate future outcomes given existing patient characteristics and can aid objective clinical decision making and patient counseling [1, 13]. External validation refers to the evaluation of a prediction model in data sets distinct from those used to develop the model and is an important step in the introduction of such tools to widespread clinical use [12, 14]. Preferably performed by investigators independent from the original development team [14, 15], external validation helps to address potential methodologic limitations that might reduce the generalizability of a clinical model. Although methodologically sound overall, the development of the ASCERT model was subject to some specific limitations, including imperfect matching of patients to the CMS database and a limitation to patients aged 65 years or older [8, 16]. In this study, we performed an external performance evaluation of the ASCERT prediction model using a temporally unique patient population and multiple sources of deliberate vital status follow-up to ensure reliable patient matching.
The performance evaluation of predictive models should assess model calibration (how well the observed outcomes agree with predicted outcomes) and also discrimination (how well model predictions differentiate between patients with and without the outcome) [12]. In this study, overall operative mortality matched predicted operative death, and observed long-term survival was comparable to other reports of long-term survival after CABG [17, 18]. As an overall measure of model calibration, the observed long-term survival curve for the entire patient sample matched the expected survival curve generated from the ASCERT prediction model (Fig 1). Risk stratification was performed to further evaluate calibration of the ASCERT model over variable levels of patient-related risk, both based on the ASCERT 7-year survival probability and on the STS PROM. Regardless of the type of stratification performed, observed and expected survival curves matched for all levels of patient-related risk (Fig 2).
The ASCERT model also showed good discriminative performance in this study sample, with C statistics of 0.797 for predicting 1-year survival and 0.815 for 7-year survival. In this regard, the model actually modestly outperformed expectations compared with the original development study, in which C statistics of 0.764 for 1-year survival and 0.748 for 3-year survival were obtained [2].
Because it is already widely available and widely used by cardiac surgery programs, we also evaluated the performance of the STS PROM as a predictor of long-term survival. Interestingly, PROM performed comparably to the ASCERT model as a predictor of long-term survival, with C statistics of 0.787 for 1-year survival and 0.781 for 7-year survival. These findings are consistent with previous reports showing PROM to accurately predict long-term survival after cardiac surgical procedures [10].
Although the STS PROM performed similarly to the ASCERT model as a statistical predictor of long-term survival, predicted mortality figures from the PROM model do not intuitively extrapolate to quantifiable long-term survival probability figures for use by practitioners. This highlights a benefit of the ASCERT model, which provides specific survival probability figures for each year after the operation that are especially valuable in preoperative patient counseling. Our analysis did, however, provide some basis for extrapolation of STS PROM figures to long-term survival probability, because PROM clearly correlated with 1-year and 7-year survival outcomes in the risk-stratified quartile analyses. For example, patients in quartile 1 had a mean PROM of 1% associated with actual 1-year survival of 99% and 7-year survival of 93%. Patients in quartile 4, in contrast, had a mean PROM of 8.4% associated with actual 1-year survival of 80% and 7-year survival of 18%. These correlated figures (further detailed in Table 3) may provide a framework for quick estimation of long-term risk of death by practitioners who already routinely use the STS risk score (PROM) for preoperative risk assessment.
Postoperative complications of prolonged ventilation, stroke, and prolonged hospital length of stay were additional predictors of long-term death when combined with the ASCERT model. The addition of these variables modestly increased the C statistics for the combined models relative to the ASCERT models alone. As expected, postoperative complications appear to explain some additional risk of long-term death, although unfortunately, these factors cannot be taken into account in a preoperative risk-assessment tool. Still, this information maybe useful to inform expectations about long-term survival in patients who experience a complicated postoperative course.
This study has some limitations. To increase the sample size of the current study, some temporal overlap was allowed between our patient population (2005 to 2010) and the original ASCERT development study (2002 to 2007). Follow-up vital status data, however, were obtained from unique data sources, thereby creating an entirely unique data set for the current external validation study.
In conclusion, external validation is an important and often underused step in the introduction of new risk-prediction models to clinical use. The performance of the ASCERT Long-Term Survival Probability Calculator was confirmed, and the STS PROM was shown to provide comparable performance as a predictor of long-term survival. Both tools provide valuable information for preoperative clinical decision making and patient counseling, allowing clinicians to provide evidence-based estimates of operative risk and potential long-term outcomes after cardiac operations.
Acknowledgments
This work was supported by National Institutes of Health T32-HL-007776-19 (T.S.L., M.R.S.) and the Barnes-Jewish Hospital Foundation (S.J.M.).
Footnotes
Presented at the Poster Session of the Fifty-third Annual Meeting of The Society of Thoracic Surgeons, Houston, TX, Jan 21–25, 2017.
References
- 1.Shahian DM, Edwards FH. The Society of Thoracic Surgeons 2008 cardiac surgery risk models: introduction. Ann Thorac Surg 2009;88(1 Suppl):S1. [DOI] [PubMed] [Google Scholar]
- 2.Shahian DM, O’Brien SM, Sheng S, et al. Predictors of long-term survival after coronary artery bypass grafting surgery: results from The Society of Thoracic Surgeons Adult Cardiac Surgery Database (the ASCERT study). Circulation 2012;125: 1491–500. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Klein LW, Edwards FH, DeLong ER, Ritzenthaler L, Dangas GD, Weintraub WS. ASCERT: the American College of Cardiology Foundation—The Society of Thoracic Surgeons collaboration on the comparative effectiveness of revascularization strategies. JACC Cardiovasc Interv 2010;3:124–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.ASCERT long-term survival probability calculator for isolated CABG. Available at http://ascertcalc.sts.org/home. Accessed June 15, 2016.
- 5.Holmes DR Jr, Brennan JM, Rumsfeld JS, et al. Clinical outcomes at 1 year following transcatheter aortic valve replacement. JAMA 2015;313:1019–28. [DOI] [PubMed] [Google Scholar]
- 6.Halkin A, Steinvil A, Witberg G, et al. Mortality prediction following transcatheter aortic valve replacement: a quantitative comparison of risk scores derived from populations treated with either surgical or percutaneous aortic valve replacement. The Israeli TAVR registry risk model accuracy assessment (IRRMA) study. Int J Cardiol 2016;215:227–31. [DOI] [PubMed] [Google Scholar]
- 7.Vassileva CM, Aranki S, Brennan JM, et al. Evaluation of The Society of Thoracic Surgeons online risk calculator for assessment of risk in patients presenting for aortic valve replacement after prior coronary artery bypass graft: an analysis using the STS Adult Cardiac Surgery Database. Ann Thorac Surg 2015;100:2109–15; discussion 2115–6. [DOI] [PubMed] [Google Scholar]
- 8.Hannan EL. Predicting long-term survival for coronary artery bypass graft surgery and percutaneous coronary intervention: another important step forward. Circulation 2012;125:1475–6. [DOI] [PubMed] [Google Scholar]
- 9.Shahian DM, O’Brien SM, Filardo G, et al. The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 1—coronary artery bypass grafting surgery. Ann Thorac Surg 2009;88(1 Suppl):S2–22. [DOI] [PubMed] [Google Scholar]
- 10.Puskas JD, Kilgo PD, Thourani VH, et al. The Society of Thoracic Surgeons 30-day predicted risk of mortality score also predicts long-term survival. Ann Thorac Surg 2012;93: 26–33; discussion 33–5. [DOI] [PubMed] [Google Scholar]
- 11.The Society of Thoracic Surgeons. Adult Cardiac Surgery Data Collection. Available at http://www.sts.org/sts-national-database/database-managers/adult-cardiac-surgery-database/data-collection. Accessed June 15, 2016.
- 12.Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010;21:128–38. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Laurent M, Fournet M, Feit B, et al. Simple bedside clinical evaluation versus established scores in the estimation of operative risk in valve replacement for severe aortic stenosis. Arch Cardiovasc Dis 2013;106:651–60. [DOI] [PubMed] [Google Scholar]
- 14.Collins GS, de Groot JA, Dutton S, et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol 2014;14:40. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Bouwmeester W, Zuithoff NP, Mallett S, et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med 2012;9:1–12. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Poullis M. Letter by Poullis regarding article, “Predictors of long-term survival after coronary artery bypass grafting surgery: results from the Society of Thoracic Surgeons adult cardiac surgery database (the ASCERT study).” Circulation 2012;126:e258. [DOI] [PubMed] [Google Scholar]
- 17.Gao G, Wu Y, Grunkemeier GL, Furnary AP, Starr A. Long-term survival of patients after coronary artery bypass graft surgery: comparison of the pre-stent and post-stent eras. Ann Thorac Surg 2006;82:806–10. [DOI] [PubMed] [Google Scholar]
- 18.O’Boyle F, Mediratta N, Fabri B, et al. Long-term survival after coronary artery bypass surgery stratified by Euro-SCORE. Eur J Cardiothorac Surg 2012;42:101–6; discussion 106–7. [DOI] [PubMed] [Google Scholar]