Abstract
Background
The goal of this study was to create a comprehensive, integer‐weighted predictive scale of adverse events after carotid endarterectomy (CEA), which may augment risk stratification and patient counseling.
Methods and Results
The targeted carotid files from the prospective NSQIP (National Surgical Quality Improvement Program) registry (2011–2013) comprised the derivation population. Multivariable logistic regression evaluated predictors of a 30‐day adverse event (stroke, myocardial infarction, or death), the effect estimates of which were used to build a weighted predictive scale that was validated using the 2014 to 2015 NSQIP registry release. A total of 10 766 and 8002 patients were included in the derivation and the validation populations, in whom 4.0% and 3.7% developed an adverse event, respectively. The NSQIP registry CEA scale included 14 variables; the highest points were allocated for insulin‐dependent diabetes mellitus, high‐risk cardiac physiological characteristics, admission source other than home, an emergent operation, American Society of Anesthesiologists’ classification IV to V, modified Rankin Scale score ≥2, and presentation with a stroke. NSQIP registry CEA score was predictive of an adverse event (concordance=0.67), stroke or death (concordance=0.69), mortality (concordance=0.76), an extended hospitalization (concordance=0.73), and a nonroutine discharge (concordance=0.83) in the validation population, as well as among symptomatic and asymptomatic subgroups (P<0.001). In the validation population, patients with an NSQIP registry CEA scale score >8 and 17 had 30‐day stroke or death rates >3% and 6%, the recommended thresholds for asymptomatic and symptomatic patients, respectively.
Conclusions
The NSQIP registry CEA scale predicts adverse outcomes after CEA and can risk stratify patients with both symptomatic and asymptomatic carotid stenosis using different thresholds for each population.
Keywords: adverse events complication, carotid endarterectomy, complication, National Surgical Quality Improvement Program, stroke
Subject Categories: Cerebrovascular Procedures, Cerebrovascular Disease/Stroke, Treatment
Clinical Perspective
What Is New?
This study uses a large patient population from the NSQIP (National Surgical Quality Improvement Program) registry to help quantify the odds of a perioperative death, infarction, or myocardial infarction on the basis of preoperative characteristics.
The scale weighs several different predictors proportional to their effect estimate in a multivariable model and was validated using patients from the more recent NSQIP registry release.
What Are the Clinical Implications?
The NSQIP registry carotid endarterectomy scale can be used to risk stratify patients being considered for carotid endarterectomy and aid in patient counseling.
Introduction
The decision analysis of pursuing carotid endarterectomy (CEA) weighs the benefit of surgery on reduced risk of cerebral infarction against the risk of developing a major perioperative adverse event, particularly stroke, death, or myocardial infarction. Preoperative characteristics impact the risk of perioperative complications; although some authors have proposed predictive models,1 including constructing predictive scales,2, 3, 4, 5, 6, 7, 8, 9 these scales have limitations that reduce their utility. An easily quantifiable, yet comprehensive, prediction score may augment risk stratification and patient counseling. The NSQIP (National Surgical Quality Improvement Program) registry is a national, multi‐institution registry that is maintained by the American College of Surgeons and enrolls patients prospectively at academic and community hospitals across the United States. The goal of the present analysis was to use the NSQIP registry targeted carotid files, which collect data on 22 variables that are pertinent to CEA,10 to construct and validate a predictive scale based on standard preoperative factors that can risk stratify both symptomatic and asymptomatic patients using the same scale, but different thresholds for each population.
Methods
Data Source and Inclusion Criteria
Data were extracted from the NSQIP registry targeted carotid files (2011–2015). All data are publicly available from the NSQIP registry.11 The NSQIP is a multi‐institutional program including federal hospitals and voluntary participation from nonfederal hospitals from varied settings across the United States. Surgical reviewers prospectively collect data with a standard protocol, and the American College of Surgeons regularly audits the data for accuracy.12, 13 The NSQIP registry follows up patients longitudinally for 30 days, and all adverse events during the hospitalization and after discharge are recorded. Studies using the deidentified NSQIP registry have been exempted as not human subject research by our institutional review board. Patients aged at least 18 years were included; these patients underwent CEA, as determined by the Current Procedural Terminology code 35301.
Predictors
Pertinent preoperative predictor variables were extracted, including patient demographics, American Society of Anesthesiologists’ physical classification designation, and functional status. Age was evaluated categorically, with the decades that approximated the median and upper quartile in the derivation population used as cutoffs. Comorbidities evaluated were smoking, hypertension requiring medication, recent congestive heart failure exacerbation (within 30 days), chronic obstructive pulmonary disease, dyspnea, diabetes mellitus, a bleeding disorder (any hypothrombotic condition other than aspirin use), and weight loss. Body habitus was classified by World Health Organization criteria as normal weight (body mass index <25 kg/m2), overweight (25 kg/m2 ≤ body mass index <30 kg/m2), class I obese (30 kg/m2 ≤ body mass index <35 kg/m2), or class II or III obese (body mass index ≥35 kg/m2). Relevant preoperative laboratory values were extracted and categorized by pertinent values: sodium (by 135 mEq/L), white blood cell count (by 4000/μL and 12 000/μL), hematocrit (by 36%), platelet count (by 150 000/μL), partial thromboplastin time (by 50 and 80 seconds), and international normalized ratio (by 1.4). Renal insufficiency was evaluated by calculating the glomerular filtration rate using the Modification of Diet in Renal Disease equation, which incorporates patient serum creatinine, age, sex, and race (stratified by 40 mL/min per 1.73 m2).14 Additional variables evaluated were revision endarterectomy (using concurrent Current Procedural Terminology code 35390) and anesthesia type.
Carotid predictors evaluated were presentation (asymptomatic, amaurosis fugax, transient ischemic attack, or stroke), preoperative modified Rankin Scale score (which is only recorded for patients who presented with a stroke), and preoperative medications. The degree of ipsilateral and contralateral carotid artery stenosis, as defined by Doppler ultrasound or angiography, is recorded by the NSQIP registry as <50%, 50% to 79%, 80% to 99%, occlusion, or not obtained. High‐risk anatomical characteristics were defined by the NSQIP registry as prior ipsilateral CEA or carotid artery stent placement, prior ipsilateral neck dissection, contralateral carotid artery occlusion, prior radiation to the neck, or contralateral laryngeal nerve injury or palsy. High‐risk physiological characteristics were defined as New York Heart Association class III/IV congestive heart failure, left ventricular ejection fraction <30%, unstable angina, or myocardial infarction within 30 days. An emergent operation includes those that are performed within 12 hours of admission. Admission type is from home versus transfer (from another short‐term care facility, the emergency department, or a nursing home).
Adverse Events
Thirty‐day complications are recorded by the NSQIP registry. A composite end point of 30‐day stroke, death, or myocardial infarction was used to build the predictive scale.15 Postoperative cerebrovascular accidents were defined by the NSQIP registry as any embolic, thrombotic, or hemorrhagic vascular accident with acute neurologic injury that persists for at least 24 hours. A myocardial infarction is indicated by new ECG findings (new left bundle branch block or ST‐segment elevation or new q waves in ≥2 leads), new elevation in troponin >3 times the upper limit of reference, or physician diagnosis. Other complications extracted were cardiac (cardiac arrest or arrhythmia requiring treatment); extubation failure; mechanical ventilation for >48 hours; symptomatic venous thromboembolism; infections (surgical site, urinary tract, pneumonia, and sepsis); packed red blood cell transfusions; and any reoperation. Carotid‐specific end points include postoperative transient ischemic attack, acute carotid artery thrombosis, and cranial nerve injury. An extended hospitalization was a hospital stay longer than the upper quartile of the derivation population. A nonroutine hospital discharge was any disposition other than to home, and readmission was any return to a short‐term care facility.
Missing Data
Data were missing on specific predictor variables, in some cases because that variable was not evaluated in individual patients (such as laboratory values that may not be obtained in all cases) and in other situations because the data were missing from the registry. For variables with missing data, the predictor was evaluated as a categorical variable, and patients with missing values were categorized as a distinct subgroup for that variable, so that known data on the predictor (patients with and without the variable) could be compared. No data were missing on adverse events.
Statistical Analysis
Statistical analyses were performed in STATA 13 (StataCorp, College Station, TX), and a probability value <0.05 defined significance. The patient population was divided into derivation (2011–2013) and validation (2014–2015) populations: the derivation population was used for the analyses that generated the predictive scale, and the validation population was used as a separate sample to evaluate the scale. Univariable logistic regression screened predictors of adverse events. Predictors with a P<0.20 in any strata were considered for inclusion in the multivariable model, which was generated with stepwise, backward selection. To optimize model parsimony, a maximum of one predictor for every 10 events was included (a standard ratio for models based on logistic regression).16 Subgroup analyses were also performed, evaluating the composite outcome stratified by symptomatic status.
Generation of the Predictive Scale
The multivariable model of adverse events was used to create the predictive scale. Point allocation was proportional to the effect size of statistically significant predictors: 3 points for an odds ratio (OR) of 1.30 to 1.49 (corresponding to a regression coefficient of 0.20–0.39), 4 points for an OR of 1.50 to 1.99 (corresponding to a regression coefficient of 0.40–0.69), and 5 points for an OR of >2.0 (corresponding to a regression coefficient of >0.70).17 In addition to the variables that were significant in the multivariable model, 2 other entry criteria into the scale were used. Two points were assigned to variables that were significant in the subgroup analyses, but not significant in the overall population. In addition, a single point was designated for variables that were available in the NSQIP registry and previously used in at least 2 other CEA risk stratification scales, but not significant in this model. Similar methods for integer‐weighted predictive scale generation have been previously used.18, 19, 20, 21, 22
Validation of the Predictive Scale
The predictive capacity of the score was evaluated in the validation population (2014–2015), as the independent variable by logistic regression with the outcome as the dependent variable. First, the composite end point was evaluated and, thereafter, other outcomes available in the NSQIP registry. Concordance statistics assessed the discrimination of all logistic regression models (ie, the ability of the model to differentiate between those who do and do not develop the outcome). Concordance statistics range between 0.50, where the model has no better prediction than chance, to 1.0, where the model has perfect discrimination.23 The calibration of models was assessed using the Hosmer‐Lemeshow test, which evaluates the null hypothesis that the observed and expected counts in each decile are equal, and a model is accepted as well calibrated if it fails to reject the null hypothesis (has a value >0.05).24
Results
Study and Validation Populations
A total of 10 766 patients were included in the derivation population (NSQIP registry 2011–2013), of whom 56.9% (n=6120) were asymptomatic, 41.2% (n=4441) were symptomatic, and 1.9% (n=205) had missing symptomatic status. The preoperative characteristics of the derivation population are shown in Table 1 and compared by the development of an adverse event in Table S1. Within the validation population (NSQIP registry 2014–2015), 8002 patients were included, in whom 54.5% were asymptomatic (n=3504), 43.8% were symptomatic (n=1695), and 1.8% (n=140) had missing symptomatic status.
Table 1.
Preoperative Characteristics of the Derivation Population (NSQIP Registry 2011–2013) and Validation Population (NSQIP Registry 2014–2015)
| Variable | Definition | Derivation Population, % | Validation Population, % |
|---|---|---|---|
| Age, y | 18–70 | 44.7 | 46.6 |
| 71–80 | 37.3 | 37.5 | |
| >80 | 18.1 | 16.0 | |
| Sex | Men | 60.9 | 61.8 |
| Women | 39.1 | 38.2 | |
| Race or ethnicity | White | 86.6 | 79.8 |
| Black | 4.4 | 4.3 | |
| Hispanic | 2.4 | 3.2 | |
| Asian | 1.9 | 1.8 | |
| Unknown | 4.7 | 10.9 | |
| Preoperative functional status | Independent | 96.5 | 97.3 |
| Dependent | 3.3 | 2.5 | |
| Missing | 0.2 | 0.2 | |
| Smoking | … | 26.6 | 27.0 |
| Hypertension | … | 85.2 | 82.0 |
| COPD | … | 10.0 | 10.1 |
| CHF | … | 1.4 | 1.5 |
| Dyspnea | … | 13.9 | 12.0 |
| Diabetes mellitus | None | 71.1 | 69.0 |
| Noninsulin | 18.2 | 19.0 | |
| Insulin | 10.8 | 12.1 | |
| Bleeding disorder | … | 20.8 | 20.0 |
| Body habitus | Normal weight | 28.3 | 25.7 |
| Overweight | 38.2 | 38.0 | |
| Class I obesity | 21.6 | 22.2 | |
| Class II/III obesity | 11.0 | 11.8 | |
| Missing | 0.9 | 2.3 | |
| Weight loss | … | 0.5 | 0.4 |
| Preoperative sodium, mEq/L | >135 | 86.1 | 85.5 |
| ≤135 | 10.1 | 10.2 | |
| Missing | 3.8 | 4.4 | |
| Preoperative GFR, mL/min per 1.73 m2 | ≥40 | 86.2 | 87.3 |
| <40 | 10.6 | 9.1 | |
| Missing | 3.2 | 3.6 | |
| Preoperative white blood cell count, cells/μL | 4000–12 000 | 90.1 | 89.0 |
| >12 000 | 3.7 | 4.1 | |
| <4000 | 1.8 | 2.1 | |
| Missing | 4.4 | 4.9 | |
| Preoperative hematocrit, % | >36 | 73.8 | 74.2 |
| ≤36 | 23.0 | 21.6 | |
| Missing | 3.2 | 4.2 | |
| Preoperative platelet count, platelets/μL | ≥150 000 | 85.2 | 85.3 |
| <150 000 | 10.3 | 9.9 | |
| Missing | 4.5 | 4.9 | |
| Preoperative PTT, s | <50 | 56.7 | 52.9 |
| 50–80 | 3.7 | 3.5 | |
| >80 | 1.3 | 1.0 | |
| Missing | 38.3 | 42.6 | |
| Preoperative INR | >1.4 | 71.5 | 66.7 |
| ≤1.4 | 3.5 | 3.0 | |
| Missing | 25.0 | 30.3 | |
| Symptomatic status | Asymptomatic | 56.9 | 54.5 |
| Amaurosis fugax | 7.1 | 7.4 | |
| TIA | 15.7 | 16.4 | |
| Stroke | 18.4 | 20.0 | |
| Missing | 1.9 | 1.8 | |
| Preoperative modified Rankin Scale score | 0–1 | 7.5 | 8.7 |
| 2–5 | 7.8 | 7.6 | |
| Not recorded | 3.1 | 3.7 | |
| Presentation other than stroke | 81.6 | 80.0 | |
| High‐risk physiological characteristics | No | 93.3 | 94.0 |
| Yes | 4.9 | 4.7 | |
| Missing | 1.9 | 1.3 | |
| High‐risk anatomical characteristics | No | 86.3 | 87.8 |
| Yes | 11.9 | 10.8 | |
| Missing | 1.9 | 1.4 | |
| Preoperative antiplatelet agents | No | 11.2 | 10.0 |
| Yes | 88.2 | 89.7 | |
| Missing | 0.6 | 0.3 | |
| Preoperative statin | No | 19.8 | 18.4 |
| Yes | 79.6 | 81.3 | |
| Missing | 0.6 | 0.3 | |
| Preoperative β blocker | No | 43.5 | 45.9 |
| Yes | 55.6 | 53.5 | |
| Missing | 0.9 | 0.6 | |
| Ipsilateral carotid artery stenosis | Mild (<50%) | 1.2 | 1.1 |
| Moderate (50%–79%) | 28.4 | 30.8 | |
| Severe (80%–99%) or occlusion | 68.8 | 66.1 | |
| Not obtained | 1.8 | 2.0 | |
| Contralateral carotid artery stenosis | Mild (<50%) | 49.9 | 51.8 |
| Moderate (50%–79%) | 27.2 | 26.9 | |
| Severe (80%–99%) or occlusion | 10.5 | 10.4 | |
| Not obtained | 12.4 | 10.9 | |
| Admission type | Home | 93.3 | 92.7 |
| Transfer | 6.7 | 7.3 | |
| Emergent operation | … | 2.5 | 2.8 |
| ASA class | I–II | 7.4 | 5.4 |
| III | 76.0 | 74.1 | |
| IV–V | 16.5 | 20.3 | |
| Missing | 0.1 | 0.2 | |
| Revision endarterectomy | … | 0.8 | 0.5 |
Glomerular filtration rate was calculated using the Modification of Diet in Renal Disease equation. Missing data for laboratory values include patients in whom such testing was not obtained. ASA physical classification designation IV is a severe systemic disease with constant threat to life and includes, but is not limited to, recent (<3 months) myocardial infarction or cardiac stent placement, severe valve dysfunction, sepsis, and renal dysfunction. ASA class V designation is a moribund patient not expected to survive without the operation. Modified Rankin Scale score of 2 to 5 indicates at least slight disability and is only recorded among patients who presented with a stroke. ASA indicates American Society of Anesthesiologists; CHF, congestive heart failure; COPD, chronic obstructive pulmonary disease; GFR, glomerular filtration rate; INR, international normalized ratio; NSQIP, National Surgical Quality Improvement Program; PTT, partial thromboplastin time; TIA, transient ischemic attack.
Multivariable Logistic Regression
The total 30‐day cumulative incidence of stroke, death, or myocardial infarction in the derivation population was 4.0% (n=427; 95% CI, 3.6%–4.3%) overall and 5.0% (95% CI, 4.3%–5.6%) among symptomatic and 3.3% (95% CI, 2.8%–3.7%) among asymptomatic patients. A multivariable logistic regression model was constructed, evaluating the predictors of stroke, death, or myocardial infarction in the derivation population (Table 2). The model had a concordance statistic of 0.69 and was well calibrated.
Table 2.
Multivariable Logistic Regression Model Evaluating Predictors of 30‐Day Adverse Events in the Derivation Population
| Variable | Definition | OR | 95% CI | P Value |
|---|---|---|---|---|
| Age, y | 18–70 | Reference | ··· | ··· |
| 71–80 | 0.99 | 0.79–1.25 | 0.95 | |
| >80 | 1.34 | 1.03–1.75 | 0.03a | |
| Race or ethnicity | White | Reference | ··· | ··· |
| Black | 0.71 | 0.42–1.20 | 0.21 | |
| Hispanic | 0.70 | 0.37–1.36 | 0.30 | |
| Asian | 1.41 | 0.89–2.50 | 0.23 | |
| Unknown | 0.80 | 0.49–1.31 | 0.38 | |
| Diabetes mellitus | None | Reference | ··· | ··· |
| Noninsulin | 1.41 | 1.10–1.81 | 0.007a | |
| Insulin | 1.55 | 1.15–2.08 | 0.004a | |
| Body habitus | Normal weight | Reference | ··· | ··· |
| Overweight | 0.95 | 0.75–1.22 | 0.70 | |
| Class I obesity | 0.91 | 0.68–1.22 | 0.53 | |
| Class II/III obesity | 0.70 | 0.48–1.03 | 0.07 | |
| Missing | 0.76 | 0.29–2.00 | 0.59 | |
| Preoperative GFR, mL/min per 1.73 m2 | ≥40 | Reference | ··· | ··· |
| <40 | 1.36 | 1.02–1.82 | 0.04a | |
| Missing | 0.49 | 0.21–1.15 | 0.10 | |
| Preoperative white blood cell count, cells/μL | 4000–12 000 | Reference | ··· | ··· |
| >12 000 | 0.72 | 0.41–1.26 | 0.25 | |
| <4000 | 0.33 | 0.10–1.16 | 0.06 | |
| Missing | 1.54 | 0.67–3.53 | 0.31 | |
| Preoperative hematocrit, % | >36 | Reference | ··· | ··· |
| ≤36 | 1.22 | 0.97–1.53 | 0.09 | |
| Missing | 1.15 | 0.42–3.19 | 0.79 | |
| Symptomatic status | Asymptomatic | Reference | ··· | ··· |
| Amaurosis fugax | 0.96 | 0.61–1.50 | 0.84 | |
| TIA | 1.37 | 1.04–1.80 | 0.02a | |
| Stroke | 2.38 | 1.55–3.67 | <0.001a | |
| Missing | 1.00 | 0.46–2.19 | 0.99 | |
| Preoperative modified Rankin Scale score | 0–1 | Reference | ··· | ··· |
| 2–5 | 2.16 | 1.33–3.50 | 0.002a | |
| Not recorded or presentation other than stroke | 2.57 | 1.45–4.54 | 0.001a | |
| High‐risk physiological characteristics | … | 1.93 | 1.38–2.69 | <0.001a |
| Missing | 2.01 | 0.82–4.93 | 0.13 | |
| High‐risk anatomical characteristics | 1.35 | 1.02–1.78 | 0.03a | |
| Missing | 0.88 | 0.34–2.29 | 0.79 | |
| Preoperative β blocker | 1.05 | 0.85–1.29 | 0.65 | |
| Missing | 0.40 | 0.09–1.69 | 0.21 | |
| Contralateral carotid artery stenosis | Mild (<50%) | Reference | ··· | ··· |
| Moderate (50%–79%) | 1.22 | 0.96–1.54 | 0.10 | |
| Severe (80%–99%) or occlusion | 1.38 | 1.01–1.90 | 0.04a | |
| Not obtained | 1.41 | 1.04–1.92 | 0.03a | |
| Admission type | Transfer | 1.54 | 1.12–2.11 | 0.008a |
| Emergent operation | … | 1.70 | 1.06–2.71 | 0.03a |
| ASA class | I–II | Reference | ··· | ··· |
| III | 1.13 | 0.71–1.81 | 0.60 | |
| IV–V | 1.94 | 1.18–3.18 | 0.009a | |
| Concordance statistic | … | 0.69 | ||
| Hosmer‐Lemeshow test | … | 0.15 | ||
ASA indicates American Society of Anesthesiologists; GFR, glomerular filtration rate; OR, odds ratio; TIA, transient ischemic attack.
Statistically significant difference.
Additional analyses evaluated predictors of an adverse event, stratified by symptomatic status (Table S2). Statistically significant independent predictors among symptomatic patients (in descending effect size) were high‐risk physiological characteristics (P=0.001), modified Rankin Scale score of 2 to 5 (P=0.004), hospital transfer (P=0.006), hypertension (P=0.03), anemia (P=0.002), and aged >80 years (P=0.04); in asymptomatic patients, predictors were hospital transfer (P=0.005), American Society of Anesthesiologists’ class 4 to 5 designation (P=0.03), contralateral carotid imaging not obtained (P=0.02), insulin‐dependent diabetes mellitus (P=0.02), and renal insufficiency (P=0.02). The concordance statistics of the models were 0.73 for symptomatic and 0.68 for asymptomatic patients, and both models were well calibrated.
NSQIP Registry CEA Scale
The ORs from the multivariable logistic regression model of a major adverse event were used to construct the NSQIP registry CEA scale; moreover, chronic obstructive pulmonary disease was included and designated a single point because of its use in prior risk stratification scales, and anemia as well as hypertension were assigned 2 points given their significance in the subgroup analysis of symptomatic patients. The components of the scale, as well as their prevalence in the derivation and validation populations, are listed in Table 3.
Table 3.
Components of the NSQIP Registry CEA Scale Score
| Points | Variable | Stratification | Prevalence, % | |
|---|---|---|---|---|
| Derivation Population (2011–2013) | Validation Population (2014–2015) | |||
| 1 | Comorbidity | Chronic obstructive pulmonary disease | 10.0 | 10.1 |
| 2 | Comorbidity | Hypertension | 85.2 | 82.0 |
| 2 | Laboratory value | Anemia (hematocrit <36%) | 23.0 | 21.6 |
| 3 | Patient age | >80 y | 18.1 | 16.0 |
| 3 | Comorbidity | Non–insulin‐dependent diabetes mellitus | 18.2 | 19.0 |
| 3 | Laboratory value | Renal insufficiency (GFR <40 mL/min per 1.73 m2) | 10.6 | 9.1 |
| 3 | Symptomatic | Presentation with a transient ischemic attack | 15.7 | 16.4 |
| 3 | High‐risk features | High‐risk anatomical characteristics | 11.9 | 10.8 |
| 3 | Contralateral carotid artery stenosis | High‐grade stenosis (80%–99%), occlusion, or imaging not obtained | 22.9 | 21.3 |
| 4 | Comorbidity | Insulin‐dependent diabetes mellitus | 10.8 | 12.1 |
| 4 | High‐risk features | High‐risk physiological characteristics | 4.9 | 4.7 |
| 4 | Admission type | Hospital transfer, emergency department, or nursing home | 6.7 | 7.2 |
| 4 | Case urgency | Emergent | 2.5 | 2.8 |
| 4 | ASA classification | IV–V | 16.5 | 20.3 |
| 5 | Symptomatic | Presentation with a stroke | 18.4 | 20.0 |
| 5 | Modified Rankin Scale | Score 2–5 | 7.8 | 7.6 |
| 47 | Maximum potential score | … | … | |
High‐risk anatomical characteristics were defined by the NSQIP registry as prior ipsilateral CEA or carotid artery stent placement, prior ipsilateral neck dissection, contralateral carotid artery occlusion, prior radiation to the neck, or contralateral laryngeal nerve injury or palsy. High‐risk physiological characteristics were defined as New York Heart Association class III/IV congestive heart failure, left ventricular ejection fraction <30%, unstable angina, or myocardial infarction within 30 days. ASA physical classification designation IV is a severe systemic disease with constant threat to life and includes, but is not limited to, recent (<3 months) myocardial infarction or cardiac stent placement, severe valve dysfunction, sepsis, and renal dysfunction. ASA class V designation is a moribund patient not expected to survive without the operation. Modified Rankin Scale score of 2 to 5 indicates at least slight disability and is only recorded among patients who presented with a stroke. ASA indicates American Society of Anesthesiologists; CEA, carotid endarterectomy; GFR, glomerular filtration rate; NSQIP, National Surgical Quality Improvement Program.
Validation of the NSQIP Registry CEA Scale
In the validation population (2014–2015; n=8002), the total 30‐day cumulative incidence of stroke, death, or myocardial infarction was 3.7% (n=297; 95% CI, 3.3%–4.1%). Greater NSQIP registry CEA scale score was predictive of the composite end point, each end point individually, stroke or death, any complication, an extended hospitalization, a nonroutine hospital discharge, and an unplanned readmission in the validation population (Table 4, Figure). However, the NSQIP registry CEA score was not predictive of purely technical complications, which are not expected to vary by preoperative characteristics, including cranial nerve injury or an acute postoperative carotid artery thrombosis (OR, 2.00; 95% CI, 0.28–14.6; P=0.49; concordance, 0.50).
Table 4.
Predictive Capacity of the NSQIP Registry CEA Scale in the Validation Population, as Well as Stratified by Symptomatic Status
| Outcome | Population | 30‐d Rate (95% CI), % | Concordance | OR | 95% CI | P Value | HL |
|---|---|---|---|---|---|---|---|
| Stroke, death, or myocardial infarction | Total | 3.7 (3.3–4.1) | 0.67 | 1.10 | 1.08–1.11 | <0.001a | 0.22 |
| Symptomatic | 4.8 (4.1–5.5) | 0.63 | 1.08 | 1.05–1.10 | <0.001a | 0.21 | |
| Asymptomatic | 2.8 (2.3–3.3) | 0.66 | 1.14 | 1.10–1.08 | <0.001a | 0.20 | |
| Stroke or death | Total | 2.8 (2.5–3.2) | 0.69 | 1.11 | 1.09–1.13 | <0.001a | 0.16 |
| Symptomatic | 4.2 (3.5–4.8) | 0.64 | 1.08 | 1.05–1.11 | <0.001a | 0.69 | |
| Asymptomatic | 1.7 (1.4–2.1) | 0.67 | 1.15 | 1.10–1.20 | <0.001a | 0.15 | |
| Death | Total | 0.8 (0.6–1.0) | 0.76 | 1.15 | 1.11–1.19 | <0.001a | 0.92 |
| Symptomatic | 1.1 (0.8–1.5) | 0.75 | 1.14 | 1.09–1.19 | 0.001a | 0.99 | |
| Asymptomatic | 0.6 (0.3–0.8) | 0.74 | 1.21 | 1.12–1.30 | <0.001a | 0.74 | |
| Stroke | Total | 2.2 (1.9–2.5) | 0.68 | 1.09 | 1.07–1.12 | <0.001a | 0.14 |
| Symptomatic | 3.4 (2.8–4.0) | 0.61 | 1.06 | 1.03–1.09 | <0.001a | 0.34 | |
| Asymptomatic | 1.3 (1.0–1.6) | 0.64 | 1.12 | 1.06–1.19 | <0.001a | 0.35 | |
| Myocardial infarction | Total | 1.4 (1.1–1.6) | 0.62 | 1.07 | 1.04–1.10 | <0.001a | 0.59 |
| Symptomatic | 1.3 (0.9–1.6) | 0.62 | 1.08 | 1.03–1.13 | 0.001a | 0.58 | |
| Asymptomatic | 1.4 (1.1–1.9) | 0.65 | 1.13 | 1.07–1.19 | <0.001a | 0.73 | |
| Cranial nerve deficit | Total | 3.1 (2.7–3.5) | 0.51 | 1.01 | 0.98–1.03 | 0.56 | 0.17 |
| Symptomatic | 2.9 (2.4–3.5) | 0.50 | 1.00 | 0.97–1.04 | 0.89 | 0.10 | |
| Asymptomatic | 3.1 (2.6–3.7) | 0.53 | 1.03 | 0.99–1.07 | 0.16 | 0.35 | |
| Any complication | Total | 10.3 (9.7–11.0) | 0.64 | 1.09 | 1.07–1.10 | <0.001a | 0.85 |
| Symptomatic | 12.8 (11.7–13.9) | 0.62 | 1.07 | 1.05–1.09 | <0.001a | 0.99 | |
| Asymptomatic | 8.4 (7.5–9.2) | 0.63 | 1.12 | 1.10–1.15 | <0.001a | 0.70 | |
| Extended hospital stay | Total | 27.7 (26.8–28.7) | 0.73 | 1.18 | 1.16–1.19 | <0.001a | 0.06 |
| Symptomatic | 44.6 (42.9–46.2) | 0.69 | 1.13 | 1.12–1.15 | <0.001a | 0.33 | |
| Asymptomatic | 14.4 (13.4–15.4) | 0.64 | 1.13 | 1.11–1.15 | <0.001a | 0.14 | |
| Nonroutine discharge | Validation | 6.6 (6.1–7.2) | 0.83 | 1.23 | 1.21–1.35 | <0.001a | 0.09 |
| Symptomatic | 12.2 (11.1–13.2) | 0.80 | 1.21 | 1.18–1.23 | <0.001a | 0.001 | |
| Asymptomatic | 3.3 (2.7–3.9) | 0.71 | 1.18 | 1.14–1.23 | <0.001a | 0.52 | |
| Unplanned readmission | Total | 5.5 (5.0–6.0) | 0.62 | 1.07 | 1.05–1.08 | <0.001a | 0.09 |
| Symptomatic | 6.2 (5.4–7.0) | 0.58 | 1.04 | 1.02–1.07 | <0.001a | 0.81 | |
| Asymptomatic | 3.3 (2.7–3.9) | 0.65 | 1.14 | 1.10–1.17 | <0.001a | 0.26 |
An extended hospital stay was defined as longer than the upper quartile of the population (of at least 3 days). A nonroutine hospital discharge is any discharge other than to home. CEA indicates carotid endarterectomy; HL, Hosmer‐Lemeshow test; NSQIP, National Surgical Quality Improvement Program; OR, odds ratio.
Statistically significant difference.
Figure 1.
Variations in the rates (and associated SEMs) of 30‐day adverse events (stroke, death, or myocardial infarction; A), mortality (B), an extended hospitalization (of at least 3 days; C), and nonroutine hospital discharge (D) in the derivation and validation populations. CEA indicates carotid endarterectomy; NSQIP, National Surgical Quality Improvement Program.
The NSQIP registry CEA scale score was predictive of the composite end point and other expected outcomes in both symptomatic and asymptomatic patients. An additional subgroup analysis evaluated the outcomes of asymptomatic patients on preoperative antiplatelet agents and statin (n=3340/4358, 76.6%): NSQIP registry CEA scale score was also predictive of postoperative outcomes (except cranial nerve injury; Table S3). Thereafter, risk stratification categories were designated on the basis of the point allocations from the NSQIP registry CEA scale, and the 30‐day rates of adverse events are presented in Table 5. Patients with a very‐high‐risk designation had a 30‐day rate of stroke or death of >6.0%, and patients with at least a medium‐risk designation had a stroke or death rate of >3.0%, the respective recommended thresholds for symptomatic and asymptomatic patients, respectively.
Table 5.
Total 30‐Day Rates of Complications in the Validation Population, Stratified by the NSQIP Registry CEA Scale Divisions
| Outcome | Very Low Risk | Low Risk | Medium Risk | High Risk | Very High Risk |
|---|---|---|---|---|---|
| NSQIP registry CEA scale points | 0–5 | 6–8 | 9–12 | 13–16 | ≥17 |
| Overall prevalence, % | 38.9 | 20.1 | 20.6 | 11.9 | 8.5 |
| Prevalence: symptomatic patients, % | 19.9 | 17.3 | 25.7 | 19.6 | 17.6 |
| Prevalence: asymptomatic patients, % | 54.2 | 22.3 | 16.5 | 5.7 | 1.4 |
| Stroke, death, or MI, % | 1.6 | 3.4 | 4.2 | 7.2 | 8.3 |
| Stroke or death, % | 1.0 | 2.5 | 3.2 | 5.8 | 6.9 |
| Death, % | 0.2 | 0.5 | 0.6 | 2.1 | 3.1 |
| Stroke, % | 0.8 | 2.1 | 2.7 | 4.4 | 4.6 |
| MI, % | 0.8 | 1.2 | 1.6 | 2.5 | 2.4 |
| Any complication, % | 6.0 | 9.0 | 11.8 | 15.5 | 22.6 |
| Extended hospitalization, % | 12.8 | 22.0 | 32.9 | 47.9 | 69.3 |
| Nonroutine discharge, % | 1.0 | 2.7 | 6.0 | 14.7 | 32.3 |
| Unplanned readmission, % | 3.2 | 4.8 | 7.2 | 8.8 | 8.7 |
CEA indicates carotid endarterectomy; MI, myocardial infarction; NSQIP, National Surgical Quality Improvement Program.
Discussion
The NSQIP registry carotid target files provide a source of prospectively collected data from across the United States, with several pertinent variables and end points related to carotid artery revascularization. Although some studies have used the NSQIP registry before the release of the CEA target files to model outcomes,1, 7, 25, 26, 27 they were not able to use the carotid‐specific variables, many of which emerged as predictors in the NSQIP registry CEA scale. The sole publication to use the NSQIP registry carotid files to examine adverse events only analyzed a single year of the release and, thereby, only a few pertinent predictors were identified and no predictive scale was proposed.10
Risk stratification of outcomes after CEA is complex, as there are many pertinent preoperative and operative factors. Although some have published models quantifying this risk,1 including predictive scales,2, 3, 4, 5, 6, 7, 8, 9 such models have key limitations: many did not include different derivation and validation populations, and those that did include a unique validation population were restricted to asymptomatic patients,4 accrued patients from a single institution,6 or provided significant weight to limited variables (admission from a nursing home or urgent procedure).9 A validated model of outcomes after CEA that is simultaneously comprehensive, incorporating known predictors, yet also based on a straightforward calculation, would aid in risk stratification.
The NSQIP registry CEA scale underscores the importance of 14 preoperative characteristics in predicting postoperative outcomes. There are many advantages to the NSQIP registry CEA scale. First, the scale is validated and is predictive of several key postoperative outcomes. The use of the more recent years as the validation population highlights the current applicability of the score, and a separate validation population increases the external validity compared with models constructed with boostrapping.28 In the validation population, the scale was not only predictive of 30‐day stroke, death, or myocardial infarction, but it was also predictive of each outcome individually, any complication, an extended hospitalization, a nonroutine discharge, and readmission. Notably, the NSQIP registry CEA scale had greater discrimination in the validation population of death (concordance=0.76), stroke or death (concordance=0.69), an extended hospitalization (concordance=0.73), and a nonroutine discharge (concordance=0.83), compared with the composite end point (concordance=0.67) on which it was constructed. However, the scale was not predictive of purely technical outcomes that were expected to depend only on operative factors, including cranial nerve injury and acute carotid artery thrombosis, highlighting its selective prediction of end points that differ by preoperative characteristics.
In addition, the NSQIP registry CEA scale can be used to risk stratify both symptomatic and asymptomatic patients, as it was predictive of the composite end point and pertinent outcomes in both groups. This differs from some predictive scales of CEA that were restricted by symptomatic status,4 making it more parsimonious and, thereby, increasing its applicability. Although the prevalence rates of high‐ and very‐high‐risk designations are expected to be greater in symptomatic patients, given the inclusion of symptomatic status, emergent surgery, and modified Rankin Scale score in the NSQIP registry CEA scale, different thresholds of adverse events can be used for acceptable risk profile for symptomatic and asymptomatic patients. Furthermore, as maximal medical therapy is the first‐line treatment for asymptomatic stenosis,29 a subgroup analysis also evaluated asymptomatic patients taking preoperative antiplatelet and statin medications, in whom the NSQIP registry CEA scale was also predictive of outcomes.
The NSQIP registry CEA scale includes, in a single scale, many preoperative variables that are known to impact perioperative risk (including those used in other predictive scales), and weighs these factors accordingly. The components of the scale are standard preoperative characteristics that are known when the decision to pursue surgery is made, allowing the scale to risk stratify patients preoperatively. Most components from prior scales that are available in the NSQIP registry were statistically significant independent predictors of adverse events and, thereby, included in the scale; given that chronic obstructive pulmonary disease has been included in 2 prior models,1, 6 but was not a significant predictor in this model, a single point was assigned for this variable. Moreover, some variables used in prior scales are included among composite predictors that are components of the NSQIP registry CEA scale (eg, congestive heart failure,4, 8, 9 which is included among high‐risk physiological characteristics; prior neck irradiation,6 which is included among high‐risk anatomical characteristics; and cardiac valvular disease,4 which is included among American Society of Anesthesiologists’ classification 4 designation).
In addition, the scale is based on an integer‐weighting system of the effect size from multivariable logistic regression, creating a scale that can be calculated in a clinical setting with appropriate weighting without the need of a formula. Although the estimation of the effect size does lose some information (compared with a full regression equation using β coefficients), the discriminations of the linearized scale (concordance=0.67) and the entire model (concordance=0.69) were similar for the composite end point, thereby favoring the simpler model. Finally, the national enrollment of patients in the NSQIP registry increases the generalizability of results compared with single‐center or regional studies.
Nevertheless, there are several limitations of the NSQIP registry CEA scale. First, only 30‐day outcomes are available; and long‐term morbidity could not be ascertained. In addition, some pertinent variables are not included in the current NSQIP registry algorithm, including laterality of stenosis, preoperative atrial fibrillation, peripheral vascular disease, and cardiac valvular disease. Moreover, although plaque morphological characteristics are important in risk stratification, particularly of asymptomatic patients, only the severity of stenosis is recorded by the NSQIP registry. Furthermore, the deidentified NSQIP registry participant use files do not provide surgeon‐ or hospital‐level data; and differences in outcomes by surgical experience or volume could not be evaluated.
Moreover, the concordance statistic of the overall model (0.69) indicates moderate discrimination. This is not surprising as the goal was to evaluate the association of preoperative factors with adverse outcomes after CEA; however, operative and surgeon characteristics will also have a significant impact on postoperative outcomes.8 The concordance statistics are comparable with previously published risk stratification scores of CEA (ranging from 0.62 to 0.74) in their respective study populations. Finally, some have argued that the concordance statistic has limited utility in surgical outcomes research, and may have greater association with case mix and homogeneity rather than model discrimination.30
The implications of the NSQIP registry CEA scale merit further comment. As an epidemiologic tool, the scale may identify baseline differences in risk profile,18 and can be used in future observational studies and trials to quantify, compare, and adjust for perioperative risk. For clinical risk stratification, the score may be of the greatest utility among patients in the very‐high‐risk category (who had a score of at least 17 and comprised 8.5% of the validation population), where the 30‐day rate of stroke or death was 6.9% in the validation population. Guidelines recommend pursuing CEA in symptomatic patients when the risk of stroke or death is <6.0%,31 and a 3% target for stroke or death in asymptomatic patients has been suggested by the American Association of Neurology.32 Therefore, symptomatic patients in the very‐high‐risk category (17.6% of the validation population) and asymptomatic patients in the medium‐, high‐, and very‐high‐risk categories (23.6% of the validation population) should be counseled about their greater odds of adverse events; and the decision to pursue surgery should be determined on the basis of each patient and his or her physicians.
Future research will be needed to evaluate (and potentially modify) the NSQIP registry CEA scale by examining its discrimination and comparing it with other prior published scales in a validation population other than the NSQIP registry. Another potential direction would be to evaluate the predictive capability of the NSQIP registry CEA score after accounting for differences in surgeon volume, experience, and prior rates of adverse outcomes. Nevertheless, such calculations can be somewhat cumbersome in a clinical setting; and ultimate implementation of a computer program or an electronic application may optimize user efficiency.33 Moreover, additional studies that weigh the NSQIP registry CEA scale score against an individual's predicted probability of infarction from carotid stenosis, particularly among asymptomatic patients, may optimally use the NSQIP registry CEA scale for risk stratification.
Conclusions
In the present analysis, the NSQIP registry CEA scale was developed using preoperative characteristics to provide a quantifiable metric to risk stratify both symptomatic and asymptomatic patients. The scale was constructed on 14 preoperative characteristics and was predictive of the composite end point of a major adverse event (stroke, death, or myocardial infarction), an extended hospitalization, a nonroutine hospital discharge, and an unplanned readmission in the validation population, among both symptomatic and asymptomatic patients. Patients with an NSQIP registry CEA scale score of at least 17 (8.5% of patients) had a 6.9% rate of stroke or death in the validation population, highlighting patients who have the greatest odds of a perioperative adverse event.
Disclosures
Dr Aziz‐Sultan is a proctor for Covidien and Codman. Dr Gormley is a proctor for Codman. The remaining authors have no disclosures to report.
Supporting information
Table S1. The Demographics and Preoperative Characteristics of the Derivation Population (NSQIP 2011–2013) and Univariable Logistic Regression Evaluating 30‐Day Stroke, Death or Myocardial Infarction
Table S2. Multivariable Regression Evaluating the Predictors of Stroke, Death, or Myocardial Infarction in the Derivation Population, Stratified by Symptomatic Status
Table S3. Prediction of the NSQIP CEA Scale in the Validation Population Among Asymptomatic Patients on Antiplatelet Agents and Statin Preoperatively (n=3340)
Acknowledgments
The American College of Surgeons (ACS) NSQIP (National Surgical Quality Improvement Program) registry and the hospitals participating in the ACS NSQIP registry are the source of the data used herein; they have not verified and are not responsible for the statistical validity of the data analysis or the conclusions derived by the authors.
(J Am Heart Assoc. 2019;8:e013412 DOI: 10.1161/JAHA.119.013412.)
The abstract of this article was presented at the Congress of Neurological Surgeons Annual Meeting, September 24 to 28, 2016, in San Diego, CA.
References
- 1. Bekelis K, Bakhoum SF, Desai A, Mackenzie TA, Goodney P, Labropoulos N. A risk factor‐based predictive model of outcomes in carotid endarterectomy: the National Surgical Quality Improvement Program 2005–2010. Stroke. 2013;44:1085–1090. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Tu JV, Wang H, Bowyer B, Green L, Fang J, Kucey D. Risk factors for death or stroke after carotid endarterectomy: observations from the Ontario Carotid Endarterectomy Registry. Stroke. 2003;34:2568–2573. [DOI] [PubMed] [Google Scholar]
- 3. Halm EA, Hannan EL, Rojas M, Tuhrim S, Riles TS, Rockman CB, Chassin MR. Clinical and operative predictors of outcomes of carotid endarterectomy. J Vasc Surg. 2005;42:420–428. [DOI] [PubMed] [Google Scholar]
- 4. Calvillo‐King L, Xuan L, Zhang S, Tuhrim S, Halm EA. Predicting risk of perioperative death and stroke after carotid endarterectomy in asymptomatic patients: derivation and validation of a clinical risk score. Stroke. 2010;41:2786–2794. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. van Lammeren GW, Catanzariti LM, Peelen LM, de Vries JP, de Kleijn DP, Moll FL, Pasterkamp G, Bots ML. Clinical prediction rule to estimate the absolute 3‐year risk of major cardiovascular events after carotid endarterectomy. Stroke. 2012;43:1273–1278. [DOI] [PubMed] [Google Scholar]
- 6. Conrad MF, Kang J, Mukhopadhyay S, Patel VI, LaMuraglia GM, Cambria RP. A risk prediction model for determining appropriateness of CEA in patients with asymptomatic carotid artery stenosis. Ann Surg. 2013;258:534–538; discussion 538–540. [DOI] [PubMed] [Google Scholar]
- 7. Gupta PK, Ramanan B, Mactaggart JN, Sundaram A, Fang X, Gupta H, Johanning JM, Pipinos II. Risk index for predicting perioperative stroke, myocardial infarction, or death risk in asymptomatic patients undergoing carotid endarterectomy. J Vasc Surg. 2013;57:318–326. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Wimmer NJ, Spertus JA, Kennedy KF, Anderson HV, Curtis JP, Weintraub WS, Singh M, Rumsfeld JS, Masoudi FA, Yeh RW. Clinical prediction model suitable for assessing hospital quality for patients undergoing carotid endarterectomy. J Am Heart Assoc. 2014;3:e000728 DOI: 10.1161/JAHA.113.000728. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Eslami MH, Rybin D, Doros G, Farber A. An externally validated robust risk predictive model of adverse outcomes after carotid endarterectomy. J Vasc Surg. 2016;63:345–354. [DOI] [PubMed] [Google Scholar]
- 10. Bennett KM, Scarborough JE, Shortell CK. Predictors of 30‐day postoperative stroke or death after carotid endarterectomy using the 2012 carotid endarterectomy‐targeted American College of Surgeons National Surgical Quality Improvement Program database. J Vasc Surg. 2015;61:103–111. [DOI] [PubMed] [Google Scholar]
- 11. National surgical quality improvement program. https://www.Facs.Org/quality-programs/acs-nsqip.
- 12. Sellers MM, Merkow RP, Halverson A, Hinami K, Kelz RR, Bentrem DJ, Bilimoria KY. Validation of new readmission data in the American College of Surgeons National Surgical Quality Improvement Program. J Am Coll Surg. 2013;216:420–427. [DOI] [PubMed] [Google Scholar]
- 13. Bensley RP, Yoshida S, Lo RC, Fokkema M, Hamdan AD, Wyers MC, Chaikof EL, Schermerhorn ML. Accuracy of administrative data versus clinical data to evaluate carotid endarterectomy and carotid stenting. J Vasc Surg. 2013;58:412–419. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Levey AS, Coresh J, Greene T, Stevens LA, Zhang YL, Hendriksen S, Kusek JW, Van Lente F. Using standardized serum creatinine values in the modification of diet in renal disease study equation for estimating glomerular filtration rate. Ann Intern Med. 2006;145:247–254. [DOI] [PubMed] [Google Scholar]
- 15. Brott TG, Hobson RW II, Howard G, Roubin GS, Clark WM, Brooks W, Mackey A, Hill MD, Leimgruber PP, Sheffet AJ, Howard VJ, Moore WS, Voeks JH, Hopkins LN, Cutlip DE, Cohen DJ, Popma JJ, Ferguson RD, Cohen SN, Blackshear JL, Silver FL, Mohr JP, Lal BK, Meschia JF. Stenting versus endarterectomy for treatment of carotid‐artery stenosis. N Engl J Med. 2010;363:11–23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Vittinghoff E, McCulloch CE. Relaxing the rule of ten events per variable in logistic and Cox regression. Am J Epidemiol. 2007;165:710–718. [DOI] [PubMed] [Google Scholar]
- 17. Moons KG, Harrell FE, Steyerberg EW. Should scoring rules be based on odds ratios or regression coefficients? J Clin Epidemiol. 2002;55:1054–1055. [DOI] [PubMed] [Google Scholar]
- 18. Morrow DA, Antman EM, Charlesworth A, Cairns R, Murphy SA, de Lemos JA, Giugliano RP, McCabe CH, Braunwald E. TIMI risk score for ST‐elevation myocardial infarction: a convenient, bedside, clinical score for risk assessment at presentation: an intravenous nPA for treatment of infarcting myocardium early II trial substudy. Circulation. 2000;102:2031–2037. [DOI] [PubMed] [Google Scholar]
- 19. Arozullah AM, Khuri SF, Henderson WG, Daley J. Development and validation of a multifactorial risk index for predicting postoperative pneumonia after major noncardiac surgery. Ann Intern Med. 2001;135:847–857. [DOI] [PubMed] [Google Scholar]
- 20. Yeh RW, Secemsky EA, Kereiakes DJ, Normand SL, Gershlick AH, Cohen DJ, Spertus JA, Steg PG, Cutlip DE, Rinaldi MJ, Camenzind E, Wijns W, Apruzzese PK, Song Y, Massaro JM, Mauri L. Development and validation of a prediction rule for benefit and harm of dual antiplatelet therapy beyond 1 year after percutaneous coronary intervention. JAMA. 2016;315:1735–1749. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Capodanno D, Barbanti M, Tamburino C, D'Errigo P, Ranucci M, Santoro G, Santini F, Onorati F, Grossi C, Covello RD, Capranzano P, Rosato S, Seccareccia F. A simple risk tool (the OBSERVANT score) for prediction of 30‐day mortality after transcatheter aortic valve replacement. Am J Cardiol. 2014;113:1851–1858. [DOI] [PubMed] [Google Scholar]
- 22. Moons KG, Kengne AP, Woodward M, Royston P, Vergouwe Y, Altman DG, Grobbee DE. Risk prediction models, I: development, internal validation, and assessing the incremental value of a new (bio)marker. Heart. 2012;98:683–690. [DOI] [PubMed] [Google Scholar]
- 23. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21:128–138. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115:928–935. [DOI] [PubMed] [Google Scholar]
- 25. Gupta PK, Pipinos II, Miller WJ, Gupta H, Shetty S, Johanning JM, Longo GM, Lynch TG. A population‐based study of risk factors for stroke after carotid endarterectomy using the ACS NSQIP database. J Surg Res. 2011;167:182–191. [DOI] [PubMed] [Google Scholar]
- 26. Wu TY, Akopian G, Katz SG. Patients at elevated risk of major adverse events following endarterectomy for asymptomatic carotid stenosis. Am J Surg. 2015;209:1069–1073. [DOI] [PubMed] [Google Scholar]
- 27. Woo K, Garg J, Hye RJ, Dilley RB. Contemporary results of carotid endarterectomy for asymptomatic carotid stenosis. Stroke. 2010;41:975–979. [DOI] [PubMed] [Google Scholar]
- 28. Bleeker SE, Moll HA, Steyerberg EW, Donders AR, Derksen‐Lubsen G, Grobbee DE, Moons KG. External validation is necessary in prediction research: a clinical example. J Clin Epidemiol. 2003;56:826–832. [DOI] [PubMed] [Google Scholar]
- 29. Halm EA, Tuhrim S, Wang JJ, Rojas M, Hannan EL, Chassin MR. Has evidence changed practice? Appropriateness of carotid endarterectomy after the clinical trials. Neurology. 2007;68:187–194. [DOI] [PubMed] [Google Scholar]
- 30. Merkow RP, Hall BL, Cohen ME, Dimick JB, Wang E, Chow WB, Ko CY, Bilimoria KY. Relevance of the c‐statistic when evaluating risk‐adjustment models in surgery. J Am Coll Surg. 2012;214:822–830. [DOI] [PubMed] [Google Scholar]
- 31. Brott TG, Halperin JL, Abbara S, Bacharach JM, Barr JD, Bush RL, Cates CU, Creager MA, Fowler SB, Friday G, Hertzberg VS, McIff EB, Moore WS, Panagos PD, Riles TS, Rosenwasser RH, Taylor AJ. 2011 ASA/ACCF/AHA/AANN/AANS/ACR/ASNR/CNS/SAIP/SCAI/SIR/SNIS/SVM/SVS guideline on the management of patients with extracranial carotid and vertebral artery disease. Stroke. 2011;42:e464–e540. [DOI] [PubMed] [Google Scholar]
- 32. Chaturvedi S, Bruno A, Feasby T, Holloway R, Benavente O, Cohen SN, Cote R, Hess D, Saver J, Spence JD, Stern B, Wilterdink J. Carotid endarterectomy—an evidence‐based review: report of the Therapeutics and Technology Assessment Subcommittee of the American Academy of Neurology. Neurology. 2005;65:794–801. [DOI] [PubMed] [Google Scholar]
- 33. Martinez‐Perez B, de la Torre‐Diez I, Lopez‐Coronado M, Sainz‐de‐Abajo B, Robles M, Garcia‐Gomez JM. Mobile clinical decision support systems and applications: a literature and commercial review. J Med Syst. 2014;38:4. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Table S1. The Demographics and Preoperative Characteristics of the Derivation Population (NSQIP 2011–2013) and Univariable Logistic Regression Evaluating 30‐Day Stroke, Death or Myocardial Infarction
Table S2. Multivariable Regression Evaluating the Predictors of Stroke, Death, or Myocardial Infarction in the Derivation Population, Stratified by Symptomatic Status
Table S3. Prediction of the NSQIP CEA Scale in the Validation Population Among Asymptomatic Patients on Antiplatelet Agents and Statin Preoperatively (n=3340)