External Validation of Risk Prediction Models to Improve Selection of Patients for Carotid Endarterectomy

Abstract

Background and purpose:

The net benefit of carotid endarterectomy (CEA) is determined partly by the risk of procedural stroke or death. Current guidelines recommend CEA if 30-day risks are <6% for symptomatic stenosis and <3% for asymptomatic stenosis. We aimed to identify prediction models for procedural stroke or death after CEA and to externally validate these models in a large registry of patients from the United States.

Methods:

We conducted a systematic search in MEDLINE and EMBASE for prediction models for procedural outcomes after CEA. We validated these models with data from patients who underwent CEA in the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP, 2011–2017). We assessed discrimination using C-statistics and calibration graphically. We determined the number of patients with predicted risks that exceeded recommended thresholds of procedural risks to perform CEA.

Results

After screening 788 reports, 15 studies describing 17 prediction models were included. Nine were developed in populations including both asymptomatic and symptomatic patients, two in symptomatic and five in asymptomatic populations. In the external validation cohort of 26,293 patients who underwent CEA, 717 (2.6%) developed a stroke or died within 30-days. C-statistics varied between 0.52 and 0.64 using all patients, between 0.51 and 0.59 using symptomatic patients, and between 0.49 to 0.58 using asymptomatic patients. The Ontario Carotid Endarterectomy Registry (OCER) model that included symptomatic status, diabetes mellitus, heart failure and contralateral occlusion as predictors, had c-statistic of 0.64 and the best concordance between predicted and observed risks. This model identified 4.5% of symptomatic and 2.1% of asymptomatic patients with procedural risks that exceeded recommended thresholds.

Conclusions:

Of the 17 externally validated prediction models, the OCER risk model had most reliable predictions of procedural stroke or death after CEA and can inform patients about procedural hazards and help focus CEA toward patients who would benefit most from it.

Introduction

Carotid endarterectomy (CEA) aims to prevent long-term stroke and should be performed in patients who may derive greatest benefit in terms of stroke risk reduction. Symptomatic patients who have had a recent stroke or transient ischemic attack (TIA) related to an ipsilateral high-grade stenosis are recommended to undergo CEA in addition to medical therapy.^1–5 The absolute benefits of CEA have become smaller due to improvement of medical preventive therapy in patients without recent symptoms related to the carotid stenosis.^6–9 The net benefit depends not only on the long-term reductions in stroke risk, but is also determined by the procedural hazards of CEA.

Current guidelines recommend to consider CEA in patients who have a risk of 30 day stroke or death of less than 6% in symptomatic patients and 3% in asymptomatic patients who also have a life expectancy of five years.^{1, 10} The risk thresholds for 30 day stroke or death might be reduced to 4% for symptomatic and 2% for asymptomatic patients as a result of improved medical therapy.

Risk prediction models might help to inform patients about procedural hazards and possibly patient selection for CEA by providing individualized risk estimations by taking several patient and disease characteristics into account.¹¹ Before implementation of prognostic risk prediction models in clinical practice, external validation in a contemporary cohort of patients who underwent CEA is needed.

We aimed to identify available risk prediction models of procedural stroke or death after CEA and to externally validate these models in a large contemporary registry of patients who underwent CEA in the United States.

Methods

Data sharing

The data that support the findings of this study are available to researchers participating in NSQIP.

Systematic review

We conducted a systematic review according to a protocol that we registered (PROSPERO CRD42019141835), and report the results of our systematic review consistent with the Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA).¹²

Search strategy

We performed electronic searches in MEDLINE (via PubMed interface) and EMBASE (via EMBASE interface) from December 2016 to January 1, 2020 to update a previous systematic review of risk prediction models for outcomes after carotid revascularization (Table I).¹¹

Eligibility criteria

We included studies that: (1) addressed development (with or without internal or external validation) of prognostic prediction models to select patients for CEA using procedural (in-hospital or 30-days) risk of stroke or death as predicted outcome; (2) in patients with carotid artery stenosis; (3) regardless of symptomatic status; (4) using predictors that are available before the intervention and can be used for patient selection; (5) based on data from observational studies or randomized clinical trials; (6) without restrictions on baseline characteristics such as age, sex, or ethnicity and; (7) were published in peer-reviewed journals without any language restrictions.

Screening process and data extraction

Two authors (MHFP and KD) independently screened all titles and abstracts of the retrieved references and subsequently independently reviewed full-text copies for final inclusion in this study. We performed backward citation searching using the bibliographies of included studies.

Two authors (MHFP and RARH) independently extracted the following data from the included prediction models based on the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Model Studies (CHARMS) checklist:¹³ source of data, study setting, geographic area (country and continent), study years, modelling method (eg, logistic model), proportion of participants with missing data, handling of missing data, appropriateness of modelling assumptions, methods for predictor selection, shrinkage of predictor weights, number of outcome events, number of participants, degree of stenosis, number and type of predictors (diagnostic variables) used in the final model, number of outcome events per variable, presentation of model, model performance (discrimination and calibration ). In studies that reported internal validation of prediction models, we extracted the following additional data: method of internal validation (e.g., cross-validation, bootstrap); whether the model was adjusted or updated after internal validation. In studies reporting external validation of a prediction model, we extracted the following additional data: type of external validation (e.g., geographical and/or temporal distinct population); whether authors of the external validation also developed the original model; performance of the model before or after model recalibration.

Critical appraisal

One author (RARH) assessed the included prediction models for the risk of bias and applicability using the Prediction model Risk Of Bias ASsessment Tool (PROBAST) and the assessment was supervised by one author (JAAD).¹⁴ The assessment of risk of bias consisted of four domains (participants; predictors; outcome; and analysis) and the applicability consisted of three domains (participants; predictors; and outcome). Risk of bias and applicability was judged as low, high or unclear for each domain.¹⁵ Each model was evaluated separately if multiple models were developed in one study.

External validation

For external validation, we adhered to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement. The TRIPOD checklist is provided as Table II.¹⁶

External validation cohort

We used a prospectively maintained cohort of patients who were registered between January 2011 and December 2017 in the Targeted Vascular module of the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) registry for external validation of the prediction models identified in our systematic review.¹⁷ In 2017, 708 participating hospitals collected data of patients who underwent a carotid intervention, including 30-days procedural outcomes. Classification of procedures was based on Current Procedural Terminology of the American Medical Association. Trained surgical clinical reviewers in each hospital collected data from medical charts and operative case logs using strict variable definitions to maintain uniformity across hospitals. Patients were contacted by letter or phone calls at 30-days after the carotid intervention to obtain data on procedural complications after discharge, if necessary. In addition, the ACS-NSQIP performed random reliability auditing to minimize information bias. Definition of variables in ACS-NSQIP and their validity have been investigated in previous reports.¹⁸

The Targeted Vascular Module of the ACS-NSQIP recorded additional disease- and procedure-related data and outcomes that were deemed crucial by vascular surgeons in a subset of 29 participating hospital of all hospitals performing vascular interventions.

Predicted outcome

We externally validated the prediction models for stroke or death within 30-days after CEA. Procedural strokes were defined as any new acute focal neurological deficit lasting more than 24 hours.

Statistical analysis

Characteristics of the external validation cohort were summarized using standard methods.

Missing data were imputed using chained equation with the MICE package in R. The imputation model included the predicted outcome of procedural death or stroke and predictors of the included prediction models. The algorithm started with the lowest proportion of missing data. The imputation continued until convergence of each predictor with a maximum of 20 iterations for each imputed dataset. Fifteen imputed datasets were computed. The number of imputed datasets was determined by taking the highest percentage of missing values and round that to the closest multiple of five.^{19, 20} The imputation was evaluated graphically with convergence plots.

The regression formula, including the intercept and beta-coefficients (predictor weights) that allows calculation of the predicted probabilities, was used to calculate the 30-day risk of stroke or death for each patient. We contacted authors to provide the regression formula if it was not provided in the original report. If the authors could not provide the regression formula, we calculated a sum score (total points) for each participant by summing the scores of the score chart assigned to each predictor in the original reports. We matched the predictors with the variables in the external validation cohort. Proxies were used if a direct match was not available. An overview of the proxies is provided in Table III. Predictors were excluded from the external validation if no proxy was available. The risk equations to calculate the predicted probabilities used in our external validation are provided in Table IV.

We assessed the predictive performance in terms of discrimination and calibration of the included prediction models. Discrimination is the ability of the prediction model to distinguish between participants with and without stroke or death within 30-days after CEA and was assessed using c-statistics. C-statistics were calculated per imputed dataset and results were combined using Rubin’s rules.^{21, 22} Calibration is the agreement between predicted risk and observed risk and was assessed using calibration plots showing predicted risks calculated with the prediction models against the observed risks in the external validation cohort. The predicted probabilities were split in deciles to enable comparison between calibration plots and the mean predicted and observed risk with corresponding 95% confidence intervals was calculated for each decile. We used a lower number of groups (with a minimum number of 500 patients for each group to obtain precise estimates) for models that did not allow splitting in deciles because the variation of probabilities was too limited. Calibration plots were created for each imputed dataset and we found that the calibration plots did not differ materially across the imputed datasets. The calibration plots using the fifteenth imputed dataset were therefore presented. We recalibrated the prediction models to the mean incidence of 30-days stroke or death in our external validation cohort to adapt the models to current clinical practice and because some models were developed with either stroke alone or death alone as predicted outcome or were restricted to outcomes that occurred in-hospital. For this, we re-estimated the intercept (referred to as ‘recalibration-in-the-large’ or ‘updating the intercept’) by fitting a logistic model with a fixed slope and the intercept as the only free parameter.²³

Three assessments of discrimination and calibration were performed: 1) including all patients who underwent CEA regardless of symptomatic status; 2) in patients who underwent CEA for symptomatic carotid artery stenosis; and 3) for CEA in patients with asymptomatic carotid stenosis.

We calculated the number of symptomatic patients with risk of procedural stroke or death exceeding 6% and 4% and the number of asymptomatic patients with risk of procedural stroke or death exceeding 3% and 2%.

We performed sensitivity analysis in complete cases.

R version 3.5.1 was used for statistical analyses and constructing figures.

Results

After screening of 788 unique reports and assessing the full-texts of 59 for eligibility, we included 15 studies reporting 17 prediction models (Figure 1 and Table V).^24–38 Two (12%) models were developed in populations of symptomatic patients,^{33, 34} five (29%) models in populations of asymptomatic patients,^35–37 and nine (53%) in populations of both symptomatic and asymptomatic patients (Table 1). Symptomatic status was included as predictor in these nine prediction models^24–32 of which one used qualifying event as predictor.²⁸ Other predictors used frequently in the 17 included models were age in eight (47%),^{25–29, 32, 36, 38} sex in seven (41%),^{28, 32, 34, 35, 37} heart failure in eleven (65%),^{24–30, 37, 38} coronary heart disease in seven (41%),^{24, 28, 35–37} and degree of contralateral stenosis in seven (47%).^{25, 27, 29, 30, 33, 35, 37} An overview of the included predictors is provided in Figure 2. The number of predictors varied from three to eleven. The number of patients used for development varied from 218 to 39,411. Four (24%) models considered outcome events before discharge^{25–27, 38} and thirteen (76%) outcome events during 30-days after CEA.^{24, 28–37} Nine models (59%) were internally validated (Table VI).^{28, 29, 34–38}

Figure 1.

Flowchart of literature search

Table 1.

Selected characteristics of included prediction models

	First author, year of publication	Symptomatic/ asymptomatic patients	Predicted outcome(s)	N events /N patients (%)	Timeframe of outcome	Number of predictors
1.	Sridharan et al, 201824	Both	Stroke, death, MI	56 / 1496 (3.7)	30-days	7
2.	Eslami et al, 201625	Both	Stroke, MI, death or discharge to rehabilitation facility	389 / 8661 (4.5)	In-hospital	8
3.	Chaudhry et al, 201626	Both	Stroke, cardiac complications or death	1494 / 49,411 (3.0)	In-hospital	7
4.	Wimmer et al, 201427	Both	Stroke or death	213 / 12,889 (1.7)	In-hospital	7
5.	Bekelis et al, 201328	Both	Stroke, MI or death	994 / 35,698 (2.8)	30-days	11
6.	Goodney et al, 200829	Both	Stroke or death	60 / 3092 (1.9)	30-days	6
7.	Tu et al, 200330	Both	Stroke or death	362 / 6038 (5.9)	30-days	5
8.	Kuhan et al, 200131	Both	Major stroke or death	29 / 741 (3.9)	30-days	3
9.	Kucey et al, 199832	Both	Stroke or death	81 / 1280 (6.3)	30-days	7
10.	Stavrinou et al, 201633	Symptomatic	TIA, PRIND, amaurosis fugax or MI	12 / 218 (5.5)	30-days	6
11.	Rothwell et al, 199934	Symptomatic	Major stroke or death	84 / 1203 (6.9)	30-days	3
12.	DeMartino et al, 201735	Asymptomatic	Stroke	287 / 31,939 (0.9)	30-days	11
13.	Gupta et al, 201336	Asymptomatic	Stroke, MI or death	324 / 17,692 (1.8)	30-days	6
14.	Calvillo-King et al, 2010a37	Asymptomatic	Stroke or death	200 / 6553 (3.1)	30-days	8
15.	Calvillo-King et al, 2010b37	Asymptomatic	Stroke or death	200 / 6553 (3.1)	30-days	7
16.	Calvillo-King et al, 2010c37	Asymptomatic	Stroke	165 / 6553 (2.5)	30-days	7
17.	Matsen et al, 200538	NR	Death	125 / 23,237 (0.5)	In-hospital	6

MI, myocardial infarction; N, number; PRIND, prolonged reversible ischemic neurologic deficit; TIA, transient ischemic attack.

Figure 2. Overview of included predictors.

Bar chart showing the frequency of the predictors used in the included risk prediction models.

¹ Symptomatic status was included as predictor in all models that were developed in populations of asymptomatic and symptomatic patients.

Risk of bias

The overall risk of bias was deemed low in four models,^{25, 27, 35, 36} unclear in one model,²⁸ and high in twelve models.^{24, 26, 29–34, 37, 38} Concerns with applicability of the models to our population of interest was deemed low in six models,^{26, 27, 29, 30, 34, 35} and high in 11 models.^{24, 25, 28, 31–33, 36–38} Reasons for high concerns included using a different predicted outcome for development compared with our external validation or using single center data for development. An overview of the risk of bias and applicability of each model is provided in Table VII.

External validations

The validation cohort consisted of 26,293 patients who underwent CEA, of whom 702 (2.7%) developed a stroke or died within 30-days. In total, 423 (3.8%) of the 11,035 symptomatic patients developed a stroke or died within 30-days, and 267 (1.8%) of the 14,772 asymptomatic patients. In the group of patients with symptomatic carotid stenosis, the qualifying event was stroke for 5096 (20%) patients, hemispheric TIA for 4083 (16%) patients and amaurosis fugax for 1856 (7%) patients. Characteristics of the external validation cohort in the models are provided in Table 2.

Table 2.

Selected characteristics of external validation cohort

	All patients (n = 26,293)	Patients with procedural stroke or death (n = 702)	Patients without procedural stroke or death (n = 25,591)	Percentage of participants with missing data
Patient characteristics
Age (years)	71 ± 9.2	72 ± 9.5	71 ± 9.2	0
Male sex	16,136 (61%)	434 (62%)	15702 (61%)	0
Diabetes Mellitus	8082 (31%)	261 (37%)	7821 (31%)	0
Current smoker	7011 (27%)	210 (30%)	6801 (27%)	0
COPD	2650 (10%)	99 (14%)	2551 (10%)	0
Heart failure	380 (2%)	34 (5%)	346 (1%)	0
Nonwhite race	1720 (7%)	56 (9%)	1664 (7%)	9.0
Preoperative hematocrit (%)	39 ± 4.9	39 ± 5.7	40 ± 4.9	3.8
Preoperative creatinine (mg/dL)	1.1 ± 0.72	1.2 ± 0.90	1.1 ± 0.71	3.5
Preprocedural antiplatelet medication	23,426 (89%)	617 (89%)	22,809 (89%)	0.4
Antihypertensive drugs use	21,924 (83%)	606 (86%)	21,318 (83%)	0
ASA Classification				0.1
ASA I-III	20,997 (80%)	461 (65%)	20,536 (80%)	-
ASA IV-V	5262 (20%)	240 (35%)	5022 (20%)	-
Functional status				0.1
Independent	25,541 (97%)	662 (94%)	24,878 (97%)	-
Partially dependent	672 (3%)	34 (5%)	638 (2%)	-
Totally dependent	44 (0%)	5 (1%)	39 (1%)	-
Disease characteristics
Symptomatic status				1.8
Stroke	5096 (20%)	249 (36%)	4847 (19%)	-
Hemispheric TIA	4083 (16%)	142 (21%)	3941 (16%)	-
Amaurosis fugax	1856 (7%)	32 (5%)	1824 (7%)	-
Asymptomatic	14,772 (57%)	267 (38%)	14,505 (58%)	-
Ipsilateral ICA stenosis1				2.1
<50%	316 (1%)	8 (1%)	308 (1%)	-
50–79%	7875 (31%)	218 (32%)	7657 (31%)	-
80–99%	17,245 (67%)	449 (65%)	16,796 (67%)	-
100%	296 (1%)	11 (2%)	285 (1%)	-
Contralateral ICA stenosis1				11.8
<50%	13,281 (58%)	290 (46%)	12,991 (58%)	-
50–79%	7169 (31%)	218 (36%)	6951 (31%)	-
80–99%	1725 (7%)	65 (11%)	1660 (7%)	-
100%	1004 (4%)	40 (7%)	964 (4%)	-
Elective surgery	22,194 (82%)	458 (64%)	21,035 (82%)	0.1

Continuous variables are presented as mean ± SD, categorical variables are presented as N (%).

¹Measured with doppler ultrasound or CT angiography.

All patients

The c-statistics of nine prediction models varied between 0.60 and 0.64^{25−27, 29, 30, 32, 33, 37} and of eight between 0.52 and 0.59^{24, 28, 31, 34–38} in the validation population (Figure 3). The three models with the highest c-statistics were Ontario Carotid Endarterectomy (OCER) model with 0.64 (95% CI 0.62–0.66), Vascular Study Group of New England (VSGNE) model with 0.64 (95% CI 0.62–0.66), and The National Cardiovascular Data Registry (NCDR) model with 0.62 (95% CI 0.60–0.64).^{25, 27, 30} Discrimination values of the original models are provided in Table VIII.

Figure 3. Discriminative performance of risk prediction models.

The symbols represent the c-statistics of the external validation in the 15 included prediction models and the horizontal bars represent the 95% CIs. The squares represent the external validation in all patients, the circle in symptomatic patients, and the diamond in asymptomatic patients.

The calibration plots of the NCDR and VSGNE models showed overestimations of risks in the highest risk groups (Figure I). The OCER model showed the best concordance across all risk groups (Figure 4).³⁰ The OCER model was developed in a population that consisted of symptomatic and asymptomatic patients and was validated using symptomatic status (stroke or hemispheric TIA vs. amaurosis fugax or retinal infarct vs. asymptomatic), diabetes mellitus, heart failure and contralateral occlusion as predictors. Most patients (42.9%) were in the lowest risk group with a predicted and observed risk of 1.9% and 1.4%, respectively. The highest risk group of 638 (2.3%) patients showed a predicted and observed risk of 6.0% and 6.7%, respectively (Figure 4). Calibration plots of other validated models are provided in Figure I, Figure II and Table IX.

Figure 4. Calibration plots of the OCER model.

Calibration plots showing the showing the predicted against the observed risk of procedural stroke or death after carotid endarterectomy. The boxes represent the mean predicted risk of each risk group and the vertical lines represent the 95% confidence intervals. The dotted diagonal line indicates perfect calibration. Boxes above the diagonal line indicate underestimation of risk and below the diagonal line overestimation of risk.

Symptomatic patients

External validation in 11,035 patients who underwent CEA for symptomatic carotid stenosis showed c-statistics that varied between 0.51 and 0.59 (Figure 3).^24–38 Two models were developed in populations of symptomatic patients and had c-statistics of 0.59 (95% CI 0.56–0.61; Münster model) and 0.54 (95% CI 0.51–0.56; European Carotid Surgery Trial [ECST] model), but calibration was inaccurate.^{33, 34}

The Vascular Study Group of New England (VSGNE) model showed the highest discrimination values of 0.59 (95% CI 0.56–0.62)²⁵ (Figure 3). The calibration plot VSGNE model showed good concordance between predicted and observed risks in lower risk groups, but overestimated risks in the high risk group.²⁵

The OCER model showed c-statistic of 0.58 (95% CI 0.56–0.61) and the calibration plot of the OCER model showed good concordance across all risk groups.³⁰ Most patients (71.7%) were in the lowest risk group with a predicted and observed risk of 3.4% and 3.3%, respectively.

In total, 508 (4.6%) patients had a predicted risk above 6% and 3167 patients (28.2%) above 4% (Figure 4). Of these 3167 patients, 1211 (38.2%) had ipsilateral stenosis of 50–79% and 1831 (57.8%) had ipsilateral stenosis of 80–99%. Calibration plots are provided in Figure I, Figure II and Table IX.

Asymptomatic patients

External validation in 14,772 patients who underwent CEA for asymptomatic carotid stenosis showed c-statistics that varied between 0.49 and 0.58.^24–38 Models that were developed in populations of asymptomatic patients had c-statistics between 0.49 and 0.56.^35–37 The OCER model had the highest discrimination value of 0.58 (95% CI 0.56–0.59)³⁰ (Figure 3). The calibration plot showed good concordance across all risk groups. Most patients (64.0%) were in the lowest risk group with a predicted and observed risk of 1.6% and 1.4%, respectively. The highest risk group of 859 (6.2%) patients had a predicted and observed risk of 3.0% and 3.7%, respectively. Calibration plots are provided in Figure I, Figure II and Table IX.

In total, 306 (2.1%) patients had a predicted risk of above 3% and 5423 (36.0%) above 2% (Figure 4).

Sensitivity analysis

Complete case analysis showed similar results (Table X).

Discussion

Our study compared the predictive performance of 17 risk models of procedural stroke or death after CEA in a representative contemporary setting. We found that the Ontario Carotid Endarterectomy Registry (OCER) model that included symptomatic status, diabetes mellitus, heart failure and contralateral occlusion as predictors showed c-statistic of 0.64 and good concordance between predicted and observed risks of procedural stroke or death after CEA. This model could therefore reliably inform patients and clinicians about expected procedural risks of CEA in symptomatic and asymptomatic patients. The model identified 508 (4.6%) symptomatic and 306 (2.1%) asymptomatic patients with procedural risks exceeding recommended thresholds of 6% and 3% for symptomatic or asymptomatic carotid stenosis, respectively.

The OCER risk prediction model which showed in whom CEA can be performed with acceptable risk. Current procedural risk thresholds to consider CEA are 6% in symptomatic and 3% in asymptomatic patients and the OCER model only identified a small proportion of patients with procedural risks that exceeded these thresholds, but these procedural risks might be reduced in the future since the risk of stroke in medically treated patients has decreased.³⁹ If the thresholds of procedural stroke or death are to be reduced to 4% in symptomatic and 2% in asymptomatic patients, the proportion of patients with predicted risks based on the OCER model exceeding these thresholds will increase to 28% of symptomatic and 36% of asymptomatic patients.

A previous external validation of the Carotid Stenosis Trialists’ Collaboration that compared 19 prediction models with short-term outcome after CEA found poor discriminative performance.⁴⁰ Their external validation cohort consisted of 4754 patients from the EVA-3S, SPACE, ICSS, and CREST trials resulting in a more homogenous population as a result of patient selection.^41–44 This might explain the poorer discriminative performance compared with our study.

Risks of procedural stroke or death also depend on the qualifying symptom and timing of CEA.^{45, 46} Patients with ischemic strokes have higher risks compared with ocular symptoms, and possibly hemispheric TIAs. Type of symptom was only included as predictor in one of the validated models.²⁸ Procedural risks are higher when CEA is performed within 48 hours after ischemic stroke.⁴⁷ The risk of stroke recurrence is also high initially and decreases over time, reducing the benefit of CEA. The optimal timing of CEA should therefore balance the risk of recurrent events and procedural risks.

The beneficial effect is clear in symptomatic patients with 70–99% stenosis without near-occlusion and somewhat less clear in patients with 50–69% stenosis.⁴⁸ These benefits have become less clear over time since medical therapy for stroke prevention has improved and the absolute gains that individual patients might receive from CEA is smaller. The Carotid Stenosis Risk (CAR) score has been developed to predict the risk of ipsilateral stroke in patients with recently symptomatic carotid stenosis on medical therapy.⁴⁹ The recently terminated ECST-2 re-evaluates the net benefit in symptomatic patients with moderate risk of stroke recurrence, ie. <20% 5-year risk (ISRCTN97744893) calculated with the CAR score. Some predictors of the CAR score overlap with predictors of models for procedural stroke or death, indicating that these predictors identify patient at high risk of stroke rather than selecting patients for the appropriate management strategy.

The absolute risk of a first ipsilateral stroke in patients with asymptomatic carotid stenosis in patients using medical preventive therapy is presumably low, but estimates are based on historic cohorts and lack preciseness due to small sample sizes.^{39, 50, 51} In addition, not all patients are using adequate medical preventive therapy.⁵² Stratification tools aiming to identify patients with a higher risk of ipsilateral stroke despite medical therapy have been developed but not validated in contemporary cohorts.^{53, 54} The use of imaging to identify characteristics of plaques vulnerability might improve prediction, but have not been included in established risk prediction models.⁵⁵ Validated models of ipsilateral stroke risk in patients with medically treated carotid stenosis showing good predictive performance could be used in conjunction with the OCER model of procedural risks to determine who might benefit from carotid interventions.⁵⁶

Strengths and Limitations

The present study has several strengths. We conducted a comprehensive literature search to identify existing prediction models according to a prespecified protocol. A large registry representative of contemporary clinical practice was used for validation. Missing data were limited for most variables and our findings were unaffected by missing data. We also included 30-day outcome events that occurred after discharge to estimate procedural hazards reliably.^{57, 58} We performed additional analyses by symptomatic status to determine absolute risks of procedural stroke or death in those patients.

The present study also has several limitations. First, though data were collected prospectively, these were not collected primarily for the present analyses. We used proxies when a direct match between the predictors in the models and variables in the external validation cohort was not available, but proxies were not available for some predictors. This might have influenced predictive performance of the validated prediction models. We have therefore provided the linear predictor functions that we used for validation (Table IV). Second, some predictors were not available in the external validation cohort or could not be used due too many missing values (Table II). Third, some risk prediction models did not allow splitting predicted risks in deciles hampering a direct comparison of calibration plots. However, visual assessment of the calibration plots clearly showed the concordance between predicted and observed risks. Fourth, data on hospital and operator volume, possibly two of the most important determinants of procedural hazards,^{59, 60} were not available in the external validation cohort and could therefore not be validated in one model that included annual surgeon volume.³² Fifth, it is unclear whether our findings are generalizable to patients with restenosis, tandem stenosis, or who received previous cervical radiation therapy.^{61, 62} Sixth, the number of patients who were deemed ineligible for carotid intervention was not collected. Seventh, we were not able to validate some risk prediction models that were developed (partly) in the same dataset.^63–65

Implications for practice and future research

Risk prediction models help inform patients about procedural hazards and might also contribute to the calculation of the net benefit of CEA in contemporary practice. The OCER model might be further refined by adding additional predictors, such as age, medical history, type of symptom, timing of CEA, and imaging characteristics, as well as by addressing identified shortcomings (in the statistical methods).

Future research will also determine which patients have the greatest reduction in absolute risk by undergoing CEA in contemporary practice weighting short-term procedural hazards against long-term stroke rates in unoperated patients and proportional reduction in non-perioperative stroke following successful CEA. Validation of established risk prediction models and assessment of predictive value of additional imaging characteristics to determine stroke risk in medically managed asymptomatic and (low-risk) symptomatic carotid stenosis is urgently needed.⁶⁶ This together with the second Carotid Revascularization versus Stenting Trial (CREST-2; NCT02089217), the ECST-2 (ISRCTN97744893), and the Asymptomatic Severe Atherosclerotic Carotid Artery Stenosis at Higher than average Risk of Ipsilateral Stroke (ACTRIS; NCT02841098) will provide reliable evidence to guide clinical decision making.

Conclusion

This external validation study assessed the predictive performance of 17 models for procedural stroke or death after CEA. We found that the Ontario Carotid Endarterectomy Registry (OCER) model that included symptomatic status (symptomatic vs. asymptomatic), diabetes mellitus, heart failure and contralateral occlusion as predictors showed c-statistic of 0.64 and good concordance between predicted and observed risks in the calibration plot. This model can be applied to symptomatic and asymptomatic patients and can reliably inform patients and clinicians about expected risks of procedural stroke or death of CEA. Patients with stroke or hemispheric TIA as qualifying event with either contralateral stenosis or heart failure and patients with retinal infarct or amaurosis fugax with both contralateral stenosis and heart failure had risks above the recommended threshold of 6%. Asymptomatic patients with both contralateral stenosis and heart failure had risks above the recommended threshold of 3%. The OCER model might help focus CEA toward patients who benefit most from it.

Non-standard Abbreviations and Acronyms

CEA

Carotid Endarterectomy

OCER (prediction model)

Ontario Carotid Endarterectomy Registry

TIA

Transient Ischemic Attack