Enrollment bias: frequency and impact on patient selection in endovascular stroke trials

Abstract

Background

Selection bias may have affected enrollment in first generation endovascular stroke trials. We investigate, evaluate, and quantify such bias for these trials at our institution.

Methods

Demographic, clinical, imaging, and angiographic data were prospectively collected on a consecutive cohort of patients with acute ischemic stroke who were enrolled in formal trials of endovascular stroke therapy (EST) or received EST in clinical practice outside of a randomized trial for acute cerebral ischemia at a single tertiary referral center from September 2004 to December 2012.

Results

Among patients considered appropriate for EST in practice, 47% were eligible for trials, with rates for individual trials ranging from 17% to 70%. Compared with trial ineligible patients treated with EST, trial eligible patients were younger (67 vs 74 years; p<0.05), more often treated with intravenous tissue plasminogen activator (53% vs 34%; p<0.01), and had shorter last known well to puncture times (328 vs 367 min; p<0.05). Focusing on the largest trial with a non-interventional control arm, compared with trial eligible patients treated with EST outside the trial, enrolled patients presented later (274 vs 163 min; p<0.001), had higher National Institutes of Health Stroke Scale scores (20 vs 17; p<0.05), and larger strokes (diffusion weighted imaging volumes 49 vs 18; p<0.001).

Conclusions

The majority of patients felt suitable for EST at our institution were excluded from recent trials. Formal entry criteria succeeded in selecting patients with better prognostic features, although many of these patients were treated outside of trials. Acknowledging and mitigating these biases will be crucial to ongoing investigations.

INTRODUCTION

Endovascular stroke therapy (EST) is a promising intervention for acute ischemic stroke. Three first generation randomized controlled trials, however, failed to demonstrate that EST yielded an improvement in final clinical outcomes over medical therapy.^1–3 Several reasons for EST failure in these trials other than lack of efficacy have been proposed, including use of early generation and less effective endovascular interventions, prolonged intervals from presentation to endovascular treatment, and failure of some studies to require imaging confirmation of large vessel occlusion prior to enrollment.^4,5 In addition, failure to enroll all patients who met study entry criteria has been suggested as a potential important contributor to the neutral results.⁶

Missing from this discussion, however, have been hard data on the frequency of non-enrollment of eligible patients in EST clinical trials, the reasons for non-enrollment, and the differences among enrolled and non-enrolled patient cohorts. There are several potential causes of non-enrollment of eligible patients in endovascular trials. First, at many sites, multiple trials were underway concurrently, testing both EST and non-EST interventions, and competing for patients.⁷ In addition, many investigators, referring physicians, and/or patients and family held a conviction that EST was beneficial. With trial enrollment seen as potentially withholding a therapy felt to be effective, some patients were treated outside of trials in accordance with routine clinical practice.⁸

With the recent publication of results by the Multicenter Randomized Clinical trial of Endovascular treatment for Acute ischemic stroke in the Netherlands (MR CLEAN) and other EST trials, equipoise in randomization against EST has become more elusive.^9–12

In this study, we analyzed the rates and reasons for endovascular treatment of acute ischemic stroke patients outside of active trials at our institution, to determine the factors that influence this process. These aspects will be important to recognize in the design and implementation of future acute stroke trials, and to better understand the context of the previous ones.

METHODS

Demographic, clinical, imaging, and angiographic data were prospectively collected on a consecutive cohort of patients who were enrolled in formal trials of EST or received EST in clinical practice outside of a randomized trial for acute cerebral ischemia at a single tertiary referral center from September 2004 to December 2012. Patients were identified in a prospectively maintained registry of all endovascular treated patients as well as through trial records. Inclusion and exclusion criteria for the five endovascular clinical trials active at the institution during this time period were obtained from the trial protocols and clinicaltrials.gov entries.^13–17 Patient eligibility for each trial was determined by reviewing their presenting demographic, clinical, and imaging data, and entry criteria for trials active at the time of their presentation. Logs for each of these trials for the duration of their active period at the institution were reviewed, and reasons for non-enrollment were recorded. Clinical outcomes were determined by data captured in the institutional database. Good neurological outcome was considered a modified Rankin Scale (mRS) score of ≤3 at discharge, consistent with comparable studies.¹⁸

In individual patients treated outside of formal trials, the decision to proceed with endovascular therapy was made jointly by the attending non-invasive vascular neurologist and the attending neurointerventionalist based on patient demographics, clinical history, and examination, as well as imaging demonstration of large vessel occlusion that would be amenable to endovascular therapy. Patients with large infarct volumes more than one-third of middle cerebral artery territory on initial imaging were generally not offered EST. Conversely, patients with substantive clinical improvement on the way to the interventional suite after intravenous tissue plasminogen activator (IV tPA) were typically observed closely but not treated with EST without recurrence of symptoms or worsening shown on imaging to be due to progressive ischemia. While no formal criteria were used in the selection of patients treated with EST outside of trials, these guidelines are also largely representative of the practices of other high volume centers.^19–22

The study was approved by the institutional review board of the local institution, and was conducted in compliance with the Health Information Portability and Accountability Act.

RESULTS

Timelines for the trial enrollment periods for the five multicenter trials of EST at the institution during the study years are shown in figure 1. During the study period, 319 patients were approved for potential endovascular therapy, including 36 (11%) enrolled in randomized trials with a non-interventional control arm (18 randomized to intervention and 18 to control), 19 (6%) enrolled in trials with only interventional arms, 20 (6%) enrolled in other trials (18 in a prehospital trial of neuroprotection and 2 in an earlier generation EST trial), and 244 (77%) treated with EST outside of any trial. Among the 319 patients eligible for EST (those treated with EST within or outside of trials plus patients randomized to non-interventional control arms of EST trials), median age was 72 years, 57% were female, median National Institutes of Health Stroke Scale (NIHSS) score was 17, and 44% received IV tPA prior to endovascular intervention. Table 1 shows the demographic and clinical characteristics for all patients. Median door to groin puncture time was 125 min, and 47% of patients achieved Thrombolysis in Cerebral Infarction (TICI) 2b/3 reperfusion.

Figure 1.

Timeline of active enrollment periods of endovascular trials at our institution, and presentation characteristics of non-enrolled eligible patients versus enrolled patients in the Mechanical Retrieval and Recanalization of Stroke Clots Using Embolectomy (MR RESCUE) trial. (A) Timeline of enrollment periods of endovascular trials at our institution. Box and whisker plots for (B) National Institutes of Health Stroke Scale (NIHSS), (C) time from last known well (LKWT) to emergency department (ED) arrival, (D) LKWT to initiation of the endovascular procedure, and (E) diffusion weighted imaging (DWI) volumes for the cohorts of patients who were eligible but not enrolled compared with those that were enrolled. Data are displayed as box and whisker plots centered on the median, with IQR as the edges of the box, and minimum and maximum values as the whiskers. IMS III, Interventional Management of Stroke III trial; Multi-Merci, Multi-Mechanical Embolus Removal in Cerebral Ischemia trial. *p<0.05, ***p<0.001, Mann–Whitney U test.

Table 1.

Patient characteristics, eligibility, and enrollment in endovascular stroke trials

		Endovascular trials
	All endovascular eligible (treated or randomized non-interventional arm)	IMS III	MR RESCUE	SWIFT	TREVO2	Multi-Merci	p Value
Total eligible (n)	319	33	109	10	7	32
Enrolled (n)	75**	14	22	5	1	13
Enrolled and randomized to non-interventional arm (n)	18	5	13	N/A	N/A	N/A
Proportion of trial eligible patients enrolled in trials (% (n))	50 (75/150)	42 (14/33)	20 (22/109)	50 (5/10)	14 (1/7)	41 (13/32)
Proportion of treatment eligible patients enrolled (% (n))	24 (75/319)	7 (14/211)	8 (22/271)	25 (5/20)	9 (1/11)	28 (13/46)
Proportion of treatment eligible patients eligible for trial (% (n))	47 (150/319)	17 (33/211)	40 (109/271)	50 (10/20)	64 (7/11)	70 (32/46)
Age (years)	72 [57–83]	68 [54–78]	67 [54–77]	58 [56–73]	74 [61–80]	75 [58–84]	0.05
Female (n (%))	181 (57)	18 (55)	57 (52)	3 (30)	3 (43)	17 (53)	0.59
Median NIHSS	17 [12–21]	19 [14–22]	17 [14–21]	16 [15–21]	15 [14–22]	19 [15–21]	0.54
Received IV tPA (n (%))	139 (44)*†	33 (100)*§‡¬	54 (50)§√	9 (90)†√∫	4 (57)‡	14 (44)¬∫	<0.001
Target vessel (n (%))							0.1
ICA	59 (19)	8 (24)	20 (18)	1 (10)	4 (57)	7 (22)
MCA	238 (75)	24 (73)	89 (82)	9 (90)	3 (43)	23 (72)
Basilar	14 (4)	0 (0)	0 (0)	0 (0)	0 (0)	2 (6)
Door to puncture time (min)	125 [94–168]	139 [114–172]	117 [88–155]	103 [85–153]	158 [126–229]	95 [82–160]	0.03
LKWT to ED arrival time (min)	210 [65–300]*	210 [65–300]*†	197 [68–269]†	260 [44–304]	82 [62–214]	180 [65–253]	<0.001
LKWT to puncture time (min)	335 [249–430]*	323 [242–390]*†§	315 [229–387]†	345 [191–408]	268 [226–321]	323 [255–358]§	<0.001
TICI 2b or 3 (n (%))	142 (47)	14 (42)	44 (40)	5 (50)	3 (43)	15 (47)	0.99
Discharge destination (n (%))							0.65
Home	63 (20)	10 (30)	22 (20)	2 (20)	2 (29)	5 (16)
SNF/rehab	189 (59)	16 (49)	74 (68)	7 (70)	4 (57)	19 (59)
Death	66 (21)	7 (21)	13 (12)	1 (10)	1 (14)	8 (25)
Discharge mRS (n (%))							0.28
0–3	101 (32)	11 (33)	38 (35)	6 (60)	2 (29)	12 (38)
4–6	218 (68)	12 (67)	71 (65)	4 (40)	5 (71)	20 (62)

Note p value reflects analysis by ANOVA or χ² of all conditions.

^{*,†,§,‡,¬,√,∫}indicate pairs found to be significant (p<0.05) after Dunn’s multiple corrections test or pairwise Fisher’s exact test.

^**Value includes patients enrolled in prehospital neuroprotection trial.

ED, emergency department; ICA, internal carotid artery; IMS III, Interventional Management of Stroke III trial; IV tPA, intravenous tissue plasminogen activator; LKWT, last known well time; MCA, middle cerebral artery; MR RESCUE, Mechanical Retrieval and Recanalization of Stroke Clots Using Embolectomy; mRS, modified Rankin Scale; Multi-Merci, Multi-Mechanical Embolus Removal in Cerebral Ischemia; NIHSS, National Institutes of Health Stroke Scale; SNF, skilled nursing facility; SWIFT, Solitaire FR With the Intention For Thrombectomy; TICI, Thrombolysis in Cerebral Infarction; TREVO2, TREVO versus Merci retrievers for thrombectomy revascularization of large vessel occlusions in acute ischemic stroke.

Less than half of all patients considered appropriate for EST in clinical practice at our institution met eligibility criteria for active clinical trials; 47% in total, ranging from 17% to 70% for individual trials. Of these patients meeting trial eligibility criteria treated with EST, half (50%) were enrolled in an EST trial (range 14–50% for the five trials). Overall, patients enrolled in formal clinical trials represented 24% of all patients considered appropriate in clinical practice for EST at our center during the study period, ranging in individual trials from 7% to 28%.

Patients who would have been eligible for each of the five trials and all patients considered appropriate for EST in clinical practice showed no differences with respect to gender, NIHSS score, target vessel, or final recanalization outcomes. There was a trend towards younger patients eligible for the Solitaire FR With the Intention For Thrombectomy (SWIFT) trial compared with the entire cohort, as well as reduced last known well (LKWT) to emergency department (ED) arrival times, and LKWT to puncture times in the subgroups of patients eligible for the Interventional Management of Stroke (IMS) III and Mechanical Retrieval and Recanalization of Stroke Clots Using Embolectomy (MR RESCUE) trials compared with the entire cohort. More patients eligible for IMS III and SWIFT received IV tPA compared with the entire cohort. Of note, inclusion criteria for IMS III mandated receipt of IV tPA, and early generations of MR RESCUE protocols excluded any patients that received IV tPA. There was no difference in discharge destination or discharge mRS for any of the subgroups of patients divided by trial eligibility.

Table 2 outlines the characteristics and outcomes of patients treated with EST but not eligible for a trial, those eligible for a trial but not enrolled, and those enrolled in any trial including those randomized to the medical therapy arms. Note that patients enrolled in non-EST trials who received EST were also included in this final category, and that all patients listed in the ‘eligible for a trial but not enrolled’ column received EST. Compared with patients eligible for trials and treated with EST but not enrolled, patients treated with EST who were ineligible for a trial were older and presented more often with basilar occlusions. Compared with patients enrolled in a trial, patients who were ineligible for a trial were less often treated with IV tPA, and had longer door to groin puncture and LKWT to groin puncture times. There were no differences in discharge outcomes of these three groups.

Table 2.

Characteristics of patient enrollment

	Treated, not eligible for any trial	Eligible for a trial but not enrolled	Enrolled in a trial (including non-interventional arm)	p Value
Number of patients	171	73	75
Age (years)	75 [60–86]*	65 [50–77]*	72 [59–81]	<0.001
Female (n (%))	103 (60)	39 (53)	39 (52)	0.49
Median NIHSS	17 [9–21]	17 [14–21]	17 [14–21]	0.23
Received IV tPA (n (%))	58 (34%)†*	37 (51)*	40 (53)†	<0.01
Target vessel (n (%))				0.03
ICA	31 (18)	18 (25)	10 (13)
MCA	123 (72)	54 (74)	63 (84)
Basilar	16 (9)*	1 (1)*	2 (3)
Door to puncture time (min)	133 [103–182]*	120 [90–160]	117 [86–155]*	0.033
LKWT to ED arrival time (min)	235 [79–330]	204 [63–285]	210 [65–385]	0.09
LKWT to puncture Time (min)	367 [263–500]*	330 [257–393]	328 [220–390]*	<0.01
TICI 2b or 3 (n (%))	80 (47)	33 (45)	31 (41)	0.35
Discharge destination (n (%))				0.42
Home	32 (19)	17 (23)	14 (19)
SNF/rehab	95 (56)	44 (60)	51 (68)
Death	44 (25)	12 (16)	10 (13)
Discharge mRS (n (%))				0.35
0–3	51 (30)	25 (34)	25 (33)
4–6	120 (70)	48 (66)	50 (67)

Note p value reflects analysis by ANOVA or χ² of all conditions.

^{*, †}indicate pairs found to be significant (p<0.05) after Dunn’s multiple corrections test or pairwise Fisher’s exact test.

ED, emergency department; ICA, internal carotid artery; IV tPA, intravenous tissue plasminogen activator; LKWT, last known well time; MCA, middle cerebral artery; mRS, modified Rankin scale; NIHSS, National Institutes of Health Stroke Scale; SNF, skilled nursing facility; TICI, Thrombolysis in Cerebral Infarction.

Because our cohort included patients randomized to a non-EST trial in the prehospital setting, we re-examined the cohort after excluding 18 patients who were ineligible for EST trials based on prehospital enrollment as well as 2 patients enrolled in an earlier generation EST trial. Eligibility for each of the trials as well as the characteristics of those enrolled can be found in online supplementary tables S1 and S2.

We performed a more granular analysis of non-enrollment reasons for the single trial with the most eligible and enrolled patients, MR RESCUE. As shown in the online supplementary table S3, the most common reasons for non-enrollment in patients who had been screened included enrollment in another randomized trial (43%) and the fact that the patient had been transferred from an outside facility specifically for EST (23%). In these cases, justification for transfer was contingent on offering the patient EST, which represented a higher level of care the referring facility could not offer. Other reasons included inability to successfully complete the imaging requirement, inability to randomize within the time window, as well as inability to obtain consent from the patient or surrogate.

Comparing the 87 patients who were eligible but not enrolled in MR RESCUE against the 22 enrolled patients (including those who were randomized to the ‘control’ arm and did not undergo endovascular therapy), patients who were eligible but not enrolled were similar in age (69 vs 67 years; p=0.43) but had lower NIHSS scores on presentation, as shown in figure 1. These patients presented earlier to the ED, had similar door to groin puncture times (117 vs 137 min; p=0.68), and shorter LKWT to puncture time. Infarct volume on presentation was significantly less for the group of eligible patients who were not enrolled (18 vs 49; p<0.001, Mann–Whitney U test). TICI 2b/3 recanalization rates were similar between the two groups (eligible vs enrolled 47% vs 44%; p=1.0, Fisher’s exact test).

Characteristics of the subset of eligible but not enrolled patients who were not enrolled specifically because their hospital transfer was contingent on EST (total of 20 patients) are shown in figure 2. We focused on this subset as it allows for direct evaluations of equipoise for the treatment modality. Compared with enrolled patients, there was no difference in age (70 vs 69 years; p=0.32) or door to groin time (104 vs 120 min; p=0.64). These patients were treated earlier, with smaller infarct volumes on MRI. There were no differences in NIHSS.

Figure 2.

Presentation characteristics of patients not enrolled due to refusal of randomization versus enrolled patients in the Mechanical Retrieval and Recanalization of Stroke Clots Using Embolectomy (MR RESCUE) trial. Box and whisker plots for (A) National Institutes of Health Stroke Scale (NIHSS), (B) time from last known well (LKWT) to emergency department (ED) arrival, (C) LKWT to initiation of endovascular procedure, and (D) diffusion weighted imaging (DWI) volumes for the subset of patients that were eligible but not enrolled due to refusal for randomization compared with enrolled patients. Edges of the box represent IQR, with a line at the median value. Whiskers denote minimum and maximum values. *p<0.05, **p<0.01, Mann–Whitney U test.

Considering patient outcomes, discharge mRS was significantly better for the cohort of MR RESCUE eligible patients treated with EST outside of the trial compared with all patients enrolled in the trial (medical and endovascular arms), as shown in online supplementary figure S1 (mRS at discharge ≤3 37% vs 5%; p<0.001, Fisher’s exact test). There was a trend towards improved outcome for the eligible patients treated outside of the trial with EST compared with those enrolled in the trial treated with EST (mRS at discharge ≤3 37% vs 11%; p=0.08, Fisher’s exact test).

DISCUSSION

In this analysis of a large cohort of acute ischemic stroke patients considered appropriate for EST at a single tertiary care center over an 8 year period, less than half were eligible for a formal clinical trial, among whom half were enrolled. Rates of eligibility for individual trials were as low as 17% in the case of IMS III. Overall, about one in six patients considered appropriate for treatment in our practice was enrolled in a formal EST trial. Patients treated outside of combined non-randomized and randomized EST trials (regardless of trial eligibility) were older, less often treated with intravenous thrombolysis, more often had basilar involvement, had longer LKWT to puncture than patients enrolled in trials, and did not differ in clinical outcomes. However, patients who were eligible for the largest randomized trial with a non-interventional control group but were not enrolled presented with lower NIHSS, arrived at the ED earlier, were treated earlier from symptom onset, and had better clinical outcomes at discharge compared with patients who were enrolled.

A notable finding of this study was that the large majority of patients treated at our center with EST in routine clinical practice were not eligible for formal EST trials. As such, the outcomes of these patients treated with EST never entered the analyses of the major EST trials. Indications for treatment in clinical practice in our center, as in many, were broad and left substantial discretion to the attending neurologist and neurointerventionalist, generally consisting of having a target large vessel occlusion and imaging or clinical evidence suggesting treatment benefit would outweigh risk. As such, while our routine clinical practice cohort was not selected with formal criteria as is done in clinical trials, it reflects the biases and preferences of real world practice. The finding that patients treated in practice substantially outnumber patients treated in formal trials highlights the need for supplementation with large pragmatic trials and registries. Among the trials analyzed, IMS III was found to have the most stringent criteria by requiring randomization prior to IV tPA, a stipulation which eliminated patients transferred from outside hospitals as potential enrollees.

Compared with patients treated outside of trials, patients enrolled or eligible for formal trials were younger, presented earlier, and were more often treated with IV tPA. These findings likely reflect the intended purposes of trial entry criteria that were designed to select for patients that may demonstrate benefit after EST. However, in spite of these tailored selection criteria, we found evidence of significant bias in the actual enrollment of these patients when a trial had a medical control arm and the potential for patients to be assigned to non-interventional therapy. Focusing on the largest trial with a non-interventional control arm, we found that patients who were eligible for enrollment but were treated outside the trial presented with less severe strokes, were treated earlier, and had better outcomes than patients who were enrolled. Door to puncture times were comparable, and as such screening and randomization of these patients likely did not contribute significantly to delays in treatment. Further, rates of successful TICI 2b/3 reperfusion were comparable, so the difference in outcomes likely reflects the more favorable baseline prognostic factors of less initial severity and earlier presentation enabling earlier treatment.

How then do we ensure that more patients, and particularly those with a chance for substantial improvement, are enrolled in formal stroke trials? Doing so would allow for rapid recruitment, earlier trial conclusions, and significant reductions in the cost of clinical investigation. We found that the primary obstacle for our cohort was competing clinical trials, followed by a frequent patient/legally authorized representative, referring physician, or treating physician preference to be treated outside of a trial. A policy of only offering EST to patients treated within the bounds of investigational trials has been suggested as one solution, and was the practice at several sites enrolling in MR RESCUE.⁸ Such policies are likely meant to counterbalance a natural decrease in the urgency to enroll patients in clinical trials when a robust practice of treating outside the trial exists.

In addition to these approaches at an institutional level, a potential solution at the national level is for regulatory authorities and payers to only reimburse for patients actually enrolled in formal clinical trials or registries. A similar policy was employed in MR CLEAN.¹² This study represents the first randomized controlled trial demonstrating benefit for EST in acute ischemic stroke, and while multiple features distinguish it from its three negative predecessors, including the almost exclusive use of modern stent retriever devices, it is possible that the nationwide restriction on EST outside of the trial also played a role in its success. The trialists in the MR CLEAN study further limited non-enrollment by designing their study with broad entry criteria, which minimized the population of trial ineligible patients who may have seemed appropriate for EST in routine clinical care. These analyses of our single center experience suggest that there may have existed an enrollment bias in the first generation studies that affected their outcomes, a possibility that had previously been suggested but never demonstrated in a quantifiable manner.²³

It is worth noting, however, that a policy of limiting reimbursement to patients enrolled in trials is not without its drawbacks. In our study, a substantial proportion of eligible patients were treated outside of clinical trials because their transfer to the hospital was contingent on performing EST. This transfer requirement may have stemmed from a belief in the benefit of EST by the referring physicians, but also relates to the financial justification of patient transfer by providing higher levels of service not available at the referring institution.²⁴

This study has limitations. The results are from a single academic medical center. Customary care at this center, with preferential MRI imaging for all possible acute strokes, and multiple concurrent stroke trials, may not be representative of all institutions enrolling in EST trials. Further, the decision making used in offering and proceeding with EST is likely not representative of all centers capable of this practice. Larger multi-institutional analyses are needed to confirm the generalizability of the findings. During the study period, a prehospital neuroprotective trial enrolled patients before they could be considered for EST trials, a factor not present at most centers. However, sensitivity analysis removing these patients did not alter the main findings, and prehospital trials will continue in the future. We identified reasons for non-enrollment from screening logs and patient medical records. Studies performing detailed interviews with patients, families, referring physicians, and treating physicians are desirable to provide more granular ethnographic insights.

In this introspective study, we laid bare our institutional experiences of enrollment in recent EST trials, by discovering, highlighting, and quantifying biases in this process. We found that these biases led to the generation of importance differences among patients that were enrolled in randomized trials and those treated outside of them. Although our study was limited to a single institution, this discovery provides a rich description of the setting in which first generation EST trials were conducted, and raises the question of how these biases may have affected their outcome. Looking forward, the results of MR CLEAN and other concurrent EST trial results will likely place EST squarely into the mainstream of acute ischemic stroke care. While the era of equipoise for EST has passed,²⁵ lessons learnt from EST trial conduct will have important ramifications to novel interventions developed in the future. For this reason, identifying and acknowledging biases, as we do in this study, will be crucial for ongoing and future investigations.