Assessing the fragility index of randomized controlled trials on carotid artery stenosis: a systematic review

Abstract

Background:

The fragility index (FI) measures the robustness of randomized controlled trials (RCTs) with dichotomous outcomes, calculated as the number of patients whose outcome would need to change for a significant result to become non-significant. Many RCTs have compared carotid endarterectomy (CEA) and carotid artery stenting (CAS), with variable results; thus, in this systematic review, we sought to explore the FI as a possible explanation for the variability in trial results between CEA and CAS.

Methods:

We conducted a search in MEDLINE (1946 to Apr. 22, 2024), Embase (1974 to Apr. 22, 2024), and PubMed (up to Apr. 22, 2024) for RCTs comparing CEA and CAS. We included RCTs with statistically significant results and dichotomous primary outcomes.

Results:

Our literature search identified 11 RCTs involving 5296 patients (n = 2640 CEA, n = 2656 CAS). All studies except 1 favoured CEA, with the exception favouring CAS. The median FI was 11 (interquartile range 2 to 22). Interestingly, 100% of the studies had loss to follow-up (LTFU) greater than its FI.

Conclusion:

A small number of events is required to change the findings of RCTs comparing CEA to CAS from statistically significant to statistically nonsignificant. All studies that reported LTFU had LTFU greater than its FI, calling into question the robustness of these results.

Carotid artery stenosis is a serious risk factor for ischemic stroke. Atherosclerotic plaque formation narrows the carotid arteries, restricting blood flow and risking embolization of cerebral vessels.¹ The prevalence of carotid stenosis increases with age, and may present as an incidental finding or with an ischemic event.² Two primary interventional treatment options for carotid stenosis are carotid artery stenting (CAS) and carotid endarterectomy (CEA). It is worth noting that optimized medical therapy has increasingly been recognized as a primary management strategy for many patients, including those with asymptomatic disease and those with symptomatic disease at low-to-intermediate predicted risk of recurrent stroke.³ Although many randomized controlled trials (RCTs) have compared the efficacy, safety, and long-term outcomes of the 2 interventional modalities, findings have varied. Trials such as the Stenting versus Endarterectomy for Treatment of Carotid-Artery Stenosis (CREST) and Endarterectomy versus Stenting in Patients with Symptomatic Severe Carotid Stenosis (EVA-3S) trials have favoured CEA for superior stroke prevention and CAS for shorter recovery time and fewer postoperative cardiac complications.^4,5 Other trials — such as the Stent-Protected Angioplasty versus Carotid Endarterectomy (SPACE) and Second Asymptomatic Carotid Surgery (ACST-2) trials — have found no difference in the risk of postoperative ipsilateral stroke or death.^6,7 One explanation for the discrepancy could be the fragility of these results.

The fragility index (FI) is a measure of robustness of clinical trials with dichotomous end points. It is calculated as the number of patients whose outcome would need to change for a significant result to become nonsignificant. A higher FI indicates more robust and less fragile results. The FI can be useful in evaluating the results of RCTs to better understand their significance.⁸ The primary objective of this systematic review is to analyze the FI of RCTs comparing CAS and CEA that showed statistically significant results for dichotomous outcomes.

Methods

We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist to systematically screen studies.⁹ This review was not registered.

Search strategy and eligibility

One reviewer (E.L.) conducted a search in MEDLINE (1946 to Apr. 22, 2024), Embase (1974 to Apr. 22, 2024), and PubMed (up to Apr. 22, 2024) using the keywords “carotid artery stenosis,” “carotid endarterectomy,” or “carotid artery stenting” combined with “randomized controlled trials” limited to human participants and English articles (Appendix 1, available at www.canjsurg.ca/lookup/doi/10.1503/cjs.008525/tab-related-content). We included RCTs that compared CEA and CAS in the treatment of carotid artery stenosis. Included studies must have had a primary outcome that was dichotomous and statistically significant (p < 0.05). We excluded studies if they were not RCTs (e.g., observational studies, review articles, technique articles, commentaries), if they did not compare CEA and CAS, if the primary outcome was not dichotomous (e.g., continuous, categorical), or if the primary outcome was not statistically significant (p < 0.05).

Study selection

Two reviewers (A.Z. and N.T.) independently screened the titles and abstracts of the studies identified through the literature search. Relevant articles were retrieved and rescreened for eligibility based on the full-text articles. We resolved any disagreements at the title and abstract stage or the full-text review stage through discussion with a third reviewer (E.L.).

Data collection

Two reviewers (A.Z. and N.T.) extracted data from included studies in duplicate. Disagreements in the data extraction stage were resolved through discussion with a third reviewer (E.L.). Data extracted included study characteristics (title, author, publication year), trial characteristics (index trial, follow-up study, subanalysis, source of funding), primary outcome (single or combined, number of patients achieving primary outcome in each group, p value), and loss to follow-up (LTFU) (number of patients lost to follow-up in each arm).

We used the Cochrane risk-of-bias tool to assess the quality of RCTs. We rated the studies to be of low, high, or unclear risk on 7 domains, including random sequence generation, assignment concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective reporting, and other bias.¹⁰

Statistical analysis

We calculated means, standard deviations (SDs), confidence intervals (CIs), and ranges. Where appropriate, we also calculated weighted means and SDs, which consider the different sample sizes of studies. We used the Mann–Whitney U test to compare sample means from nonparametric data. The FI is a simple numerical measure that describes how many event status changes would be required to change the statistical significance of a trial’s primary outcome. We calculated the FI by creating 2-by-2 contingency tables from the primary outcome, adding events from the group with fewer events, and subtracting nonevents from the same group until a 2-tailed Fisher exact test changes from statistically significant (p < 0.05) to a statistically nonsignificant result (p ≥ 0.05). The number of event changes required is the FI of the study, which corresponds to the number of patients whose outcomes would need to change to convert the primary outcome from statistically significant to nonsignificant.⁸ We also calculated the fragility quotient, which is the FI divided by the sample size, multiplied by 100.

Table 1 and Table 2 illustrate this with a fictional example. In the experimental group, 5 of 100 patients had the primary event whereas in the control group, 15 of 100 patients had the primary event (p = 0.03) (Table 1). To calculate the FI, we would add 1 event and subtract 1 from nonevent in the experimental group to obtain a new 2-by-2 contingency table (Table 2). We then reperform the Fisher exact test, for a p value of 0.06, rendering the results statistically nonsignificant. The FI of this fictional trial is 1, meaning that 1 patient needs to change their event status to change the trial results from statistically significant to nonsignificant.

Table 1.

Example contingency table

Group	Event	Nonevent
Experimental	5	95
Control	15	85

Table 2.

Example contingency table after changing event status

Group	Event	Nonevent
Experimental	6	94
Control	15	85

Ethics approval

Institutional ethics approval was not required for this systematic review.

Results

Included studies

Our literature search identified 3138 studies, of which we included 11 RCTs for review. The 11 RCTs included a total of 5296 patients, of whom 2640 underwent CEA and 2656 underwent CAS. For subanalyses or follow-up analyses of the same trial, we counted only the manuscript with the greatest number of patients included. Study selection and exclusion are summarized in the PRISMA flowchart (Figure 1).

Fig. 1.

Study flow chart. CAS = carotid artery stenting; CEA = carotid endarterectomy. See Related Content tab for accessible version.

Study characteristics

Characteristics of included studies are summarized in Table 3. Studies were published between 1998 and 2023. All included studies were RCTs. This included 2 index trials, 1 interim analysis, 6 subanalyses, and 2 follow-up studies. Six of the studies had nonindustry funding and 5 had both industry and nonindustry funding. Follow-up times ranged from 3 days to 5 years.

Table 3.

Characteristics of included studies

Study	Study design	Trial analysis	Funding*	Follow-up time	No. of patients
					Total	CEA	CAS
Altinbas 201411	RCT	Subanalysis	Both	3.2 d	1585	819	766
Blackshear 201112	RCT	Subanalysis	Both	4 yr	2502	1240	1262
Bonati 200913	RCT	FU study	Nonindustry	5 yr	413	213	200
Bonati 201014	RCT	Subanalysis	Both	1 mo	231	107	124
Bonati 201815	RCT	Subanalysis	Both	4 yr	1530	793	737
Ederle 201016	RCT	Interim analysis	Both	120 d	1710	857	853
Mas 20065	RCT	Index study	Nonindustry	30 d	520	259	261
Mas 200817	RCT	FU study	Nonindustry	4 yr	527	262	265
McCabe 200518	RCT	Subanalysis	Nonindustry	1 yr	347	174	173
Naylor 199819	RCT	Index study	Nonindustry	30 d	17	10	7
Wagner 202320	RCT	Subanalysis	Nonindustry	3 d	127	58	69

CAS = carotid artery stenting; CEA = carotid endarterectomy; FU = follow-up; RCT = randomized controlled trial.

^*Industry or nonindustry funding.

Fragility index and loss to follow-up

Primary outcomes were single in 6 studies and composite in 5 studies. The primary outcomes included a composite of stroke or death (3 studies); stroke, death, or myocardial infarction (1 study); restenosis (3 studies); new lesion on postoperative diffusion-weighted imaging (2 studies); hemodynamic complications (1 study); and myocardial infarction (1 study). All included studies favoured CEA except for 1 study of the CREST cohort that used myocardial infarction as the outcome, which favoured CAS (Table 4).

Table 4.

Fragility index and lost-to-follow-up data

Study	Primary outcome	Index trial	Single or composite	Results favour	CEA		CAS		p value	FI	LTFU	FI > LTFU
					No. of patients	No. of events	No. of patients	No. of events
Altinbas 201411	Hemodynamic complications	ICSS	Composite	CEA	819	59	766	105	< 0.001	27	NR	NR
Blackshear 201112	Myocardial infarction	CREST	Single	CAS	1240	28	1262	14	0.03	2	NR	NR
Bonati 200913	≥ 70% restenosis	CAVATAS	Single	CEA	213	20	200	53	< 0.001	19	51	Yes
Bonati 201014	New hyperintense DWI lesion on the post-treatment scan	ICSS	Single	CEA	107	18	124	62	< 0.001	22	148	Yes
Bonati 201815	> 50% restenosis	ICSS	Single	CEA	793	217	737	274	< 0.001	40	183	Yes
Ederle 201016	120-d rate of stroke, death, or procedural myocardial infarction	ICSS	Composite	CEA	857	44	853	72	0.006	8	NR	NR
Mas 20065	Any stroke or death	EVA-3S	Composite	CEA	259	10	261	25	0.01	3	23	Yes
Mas 200817	Any periprocedural stroke or death and any nonprocedural ipsilateral stroke during follow-up (up to 4 yr)	EVA-3S	Composite	CEA	262	15	265	29	0.03	1	3	Yes
McCabe 200518	> 70% restenosis	CAVATAS	Single	CEA	174	9	173	32	0.0001	11	NR	NR
Naylor 199819	Death or stroke	Unnamed	Composite	CEA	10	0	7	5	0.003	2	6	Yes
Wagner 202320	At least 1 new brain lesion on postprocedure DWI	ACST-2	Single	CEA	58	12	69	40	< 0.001	11	NR	NR

ACST-2= Second Asymptomatic Carotid Surgery trial; CAS = carotid artery stenting; CEA = carotid endarterectomy; CAVATAS = Carotid and Vertebral Artery Transluminal Angioplasty Study; CREST = Stenting versus Endarterectomy for Treatment of Carotid-Artery Stenosis; DWI = diffusion-weighted imaging; EVA-3S = Endarterectomy versus Stenting in Patients with Symptomatic Severe Carotid Stenosis; FI = fragility index; ICSS = International Carotid Stenting Study; LTFU = loss to follow-up; NR = not reported.

The median FI was 11 (range 1 to 40, interquartile range [IQR] 2 to 22). There was no relationship between single or composite outcomes and fragility (median 15 v. 3, p = 0.2). There was also no relationship between funding type and fragility (median 22 v. 7, p = 0.2). The median fragility quotient was 2.6 per 100 (IQR 0.47 to 8.66).

Loss to follow-up was not reported in 6 of the 11 studies. The median LTFU was 37 patients (range 3 to 183, IQR 6 to 148) and was greater than the FI in all studies that reported LTFU (Table 4 and Figure 2).

Fig. 2.

Fragility index and loss to follow-up.

Quality assessment and risk of bias

We used the Cochrane risk-of-bias tool to assess the included RCTs. For subanalyses or follow-up analyses on the same trial, we referred to the index trial for clarification on trial design. Overall, the studies included low selection bias in random sequence generation and assignment concealment, and low reporting bias. There was high performance bias as surgeon and patient could not be blinded, given the nature of the studies. There was variable detection bias, attrition bias, and other bias owing to industry funding (Appendix 2, available at www.canjsurg.ca/lookup/doi/10.1503/cjs.008525/tab-related-content).

Discussion

The FI is a measure of the robustness of results in RCTs, defined as the number of patients whose outcome would need to change for a significant result to become nonsignificant. We analyzed the FI of RCTs that compared CEA and CAS in the treatment of carotid artery stenosis and found that a median of 11 event conversions would change outcomes in these RCTs to nonsignificant.

In 2014, Walsh and colleagues published the index paper on FI in which they surveyed 306 RCTs in high-impact journals and found a median FI of 8, with the total LTFU exceeding FI in 52.9% of the trials.⁸ A 2022 review of neurosurgical RCTs by Volovici and colleagues found the median FI to be 4.5, with LTFU exceeding FI in 51% of the cases.²¹ At first glance, the absolute FI in this systematic review is higher than in both of those studies. However, an interesting finding in our study is that LTFU was higher than FI in 100% of studies that reported LTFU data. The LTFU is critical when interpreting FI results. Because the FI represents the number of outcome changes required to nullify statistical significance, when LTFU exceeds the FI, there is a theoretical possibility that, if the outcomes of those missing patients were known, the trial’s result could shift from significant to nonsignificant. Importantly, the impact of LTFU is amplified in trials with a small FI (e.g., FI < 5), where even a small number of unaccounted patients could completely overturn the primary finding. Conversely, in trials with higher FIs (e.g., > 20), the influence of LTFU is less dramatic but still relevant, especially when LTFU numbers are very high. This finding calls into question the robustness of statistically significant results in carotid stenosis literature, as it suggests the results of these trials could become nonsignificant if the patients were not lost to follow-up.

Landmark trials comparing CEA and CAS have recognized both as safe and effective treatment options, with varying results regarding their long-term outcomes.^4,5,6,7 Early RCTs such as the North American Symptomatic Carotid Endarterectomy Trial originally established CEA as the gold standard for symptomatic severe (> 70%) carotid stenosis, with significantly reduced rates of stroke or death over 5 years compared with medical therapy.²² However, CREST demonstrated no difference between CEA and CAS in 10-year rates of stroke, myocardial infarction, or death, with both treatments proving effective for stroke prevention.⁴ Subsequent studies have had conflicting results depending on patient population.^4–7 In symptomatic patients, CREST found a higher risk of perioperative stroke with CEA and higher rates of myocardial infarction with CAS.⁴ Asymptomatic patients had a slightly higher risk of stroke with CAS than with CEA in the EVA-3S trial.⁵ The current clinical guideline, as per the Canadian Stroke Best Practices, recommends CEA in most patients, especially those older than 70 years, with CAS as a secondary treatment option for poor surgical candidates, and those aged 70 years and younger.²³

In this systematic review, we explored FI as a possible explanation for the variability in trial results between CEA and CAS. We found the median FI to be 11 and the LTFU to be greater than the FI in 100% of the studies. We know from previous studies that sample size is weakly correlated with the FI and the total event number is moderately correlated with the FI.^8,24 This means that larger sample sizes and more common events are correlated with higher FIs. Unfortunately for fragility analyses and fortunately for patients, the primary outcomes in carotid stenosis RCTs are relatively rare events such as stroke or death, which makes the results more fragile and vulnerable to attrition bias. For example, in our review, we found the median fragility quotient to be 2.6 per 100, which means 2.6 events per 100 participants are needed for the results to change from statistically significant to nonsignificant. This also means that included RCTs need less than a 2.6% LTFU rate to have an LTFU that does not exceed the FI, which can be difficult to accomplish methodologically. Overall, the relative fragility of the RCTs comparing CEA and CAS introduces ambiguity to the results and opens the studies to be influenced by attrition bias.

Limitations

The FI is a well-established statistical measure that has been used in many specialties to assess the robustness of RCT results. Despite its utility in analyzing RCT fragility, the FI is limited in that it assesses only dichotomous outcomes. This is useful for outcomes such as stroke and death, which are commonly used in carotid stenosis literature, but we had to exclude studies reporting outcomes such as degree of restenosis or duration without symptoms postoperatively. Additionally, the FI includes only statistically significant trials; RCTs with statistically nonsignificant results need to be assessed using the reverse FI, which is calculated in a similar way, except events are subtracted from groups with fewer events and nonevents are added to the same group until statistical significance is achieved.⁴ Additionally, there is no accepted FI classification system and what is considered an acceptable value. Results are compared with published historical results and LTFU data. Another limitation to this study is the variable follow-up time in the included studies, which ranged from 3 days to 5 years. The FI is moderately related to total event number, which means that more common events are likely to be less fragile. Trials with longer follow-up periods are more likely to accumulate events, which make them less fragile. Trials with shorter follow-up periods are less likely to accumulate events. However, it also depends on the specific event that the trial is measuring and how common it is.

Conclusion

Our systematic review of RCTs comparing CEA and CAS found that a small number of outcome changes would be sufficient to change the results from statistically significant to nonsignificant. All studies that reported LTFU had LTFU greater than its FI, calling into question the robustness of these results. These findings underscore the need for robust trial designs with minimized LTFU, transparent reporting, and consideration of fragility metrics when interpreting evidence for carotid revascularization strategies.