Using Epidemiological Data to Inform Clinical Trial Feasibility Assessments: A Case Study

Abstract

Background:

Clinical trial enrollment and completion is challenging, with nearly half of all trials not being completed or not completed on time. In 2014, NIH StrokeNet in collaboration with stroke epidemiologists from the Greater Cincinnati/Northern Kentucky Stroke Study (GCNKSS) began providing proposed clinical trials with formal trial feasibility assessments. Herein, we describe the process of prospective feasibility analyses using epidemiological data that can be used to improve enrollment and increase the likelihood a trial is completed.

Methods:

In 2014, DEFUSE 3 trialists, NIH StrokeNet and stroke epidemiologists from the Greater Cincinnati/Northern Kentucky Stroke Study collaborated to evaluate the initial inclusion and exclusion criteria for the DEFUSE 3 study. Trial criteria were discussed and an assessment was completed to evaluate the percent of the stroke population that might be eligible for the study. The DEFUSE 3 trial was stopped early with the publication of DAWN, and the Wilcoxon rank sum statistic was used to analyze if the trial would have been stopped had the proposed changes not been made, following the DEFUSE 3 statistical analysis plan.

Results:

After initial epidemiological analysis, 2.4% of acute stroke patients in the GCNKSS population would have been predicted to be eligible for the study. After discussion with PIs and modifying 4 key exclusion criteria (upper limit of age increased to 90, baseline mRS broadened to 0-2, time since last well expanded to 16 hours, and decreased lower limit of NIHSS to <6) the number predicted to be eligible for the trial increased to 4%. At the time of trial conclusion, 57% of enrolled patients qualified only by the modified criteria and the trial was stopped at an interim analysis which demonstrated efficacy. We estimated that the Wilcoxon rank sum value for the unadjusted predicted enrollment would not have crossed the threshold for efficacy and the trial not stopped.

Conclusions:

Objectively assessing trial inclusion and exclusion criteria using a population-based resource in a collaborative and iterative process including epidemiologists can lead to improved recruitment and can increase the likelihood of successful trial completion.

Graphical Abstract

Introduction:

Clinical trials are a cornerstone of medical science, testing the safety and efficacy of new therapeutics and technologies in the clinical setting. Numerous barriers exist to successful clinical trial completion, including substantial cost, prolonged duration and difficulties with recruiting and retaining participants. Fewer than half of all clinical trials complete their target enrollment on time or at all. ¹ Acute stroke trial recruiting is particularly challenging, with a systematic review showing a tendency to reduced rates of recruitment among more recently performed trials. ² Estimates of average cost related to performing clinical trials vary widely, ranging from $19 to 30 million.^{3 4} In addition to cost, various issues have been identified as barriers to recruitment of patients into clinical trials, including lack of awareness, lack of access and/or fear/distrust of research. ^5–8

With increasing cost and complexity involved in clinical trial implementation, careful planning of trial design and eligibility is paramount to ensuring successful completion. Subjective and anecdotal assessments done by clinical trialists may underestimate, but more likely overestimate the eligible population for their trial leading to unexpected difficulty with enrollments and with completing the trial within the planned timeframe. In September of 2013, the National Institutes of Health (NIH) established the NIH StrokeNet trial network to facilitate the rapid initiation and efficient implementation of multisite stroke clinical trials and biomarker validation studies. The network incorporated a process for objectively assessing clinical trial enrollment feasibility using a longstanding, NIH-funded, population-based study of stroke. Here we highlight the impact of this process on enrollment in a pivotal clinical trial, Endovascular Therapy Following Imaging Evaluation for Ischemic Stroke 3 (DEFUSE 3). ⁹

Methods:

The data that support the findings of this study are available from the corresponding author upon reasonable request. The DEFUSE 3 trial was approved by the StrokeNet central institutional review board and the Food and Drug Administration (FDA) for an investigational device exemption (IDE G150028) and the Greater Cincinnati/Northern Kentucky Stroke Study (GCNKSS) was approved by the institutional review boards of all participating hospitals with a waiver from informed consent.

Greater Cincinnati-Northern Kentucky Stroke Study (GCNKSS)

Population:

GCNKSS is a population-based stroke study occurring in 5 contiguous counties in Southern Ohio and Northern Kentucky that abut the Ohio river. The study population size is approximately 1.3 million people, and its composition is similar to the United States overall in terms of black race, age, education, and income. The study has been ascertaining and characterizing stroke in this population since 1993-94, approximately every five years.

Case ascertainment:

Methods for case ascertainment in the GCNKSS have been published previously.¹⁰ Briefly, all potential stroke cases in the study region were identified using ICD codes (ICD9 430-436 in 2005). Cases were obtained from all local hospitals, local public health care clinics, and a random sample of local physicians’ offices. For each potential case, a trained research nurse abstracted relevant clinical data with a standardized case report form, and these cases were then adjudicated by a trained study physician. Cases were categorized as ischemic strokes, transient ischemic attack (TIA), intracerebral hemorrhages (ICH), or subarachnoid hemorrhages (SAH);^10–12 ischemic strokes were further subtyped into cardioembolic, small vessel, large vessel occlusive, other cause (including dissection, cancer, etc.), and undetermined cause.¹³ For this analysis, we included only ischemic strokes presenting to an emergency department.

DEFUSE-3 Case Study

Feasibility assessment:

The DEFUSE-3 trial was proposed to NIH StrokeNet in 2014 and is the first study to have been developed and completed fully within NIH StrokeNet. In collaboration with the DEFUSE 3 trial investigators, prior to trial launch, we assessed patient eligibility using the initially proposed inclusion and exclusion (I/E) criteria within our GCNKSS database. A complete list of the I/E criteria, including which were able to be assessed in the GCNKSS is included in Table 1. As the first estimate of the candidate population for the trial was smaller than desirable for efficient recruitment, the proposed I/E criteria were then evaluated specifically for possible adjustments which might be expected to increase enrollment but, in the opinion of the trial investigators, would not affect the scientific goals of the study. Adjustments yielded a predicted higher enrollment rate and based in part upon this feasibility analysis, the revised I/E criteria were chosen for the DEFUSE 3 trial.

Table 1:

Complete list of Inclusion and Exclusion Criteria for DEFUSE 3

Clinical Inclusion Criteria	Neuroimaging Inclusion Criteria
1. Signs and symptoms consistent with the diagnosis of an acute anterior circulation ischemic stroke*	1. ICA or MCA-M1 occlusion (carotid occlusions can be cervical or intracranial; with or without tandem MCA lesions) by MRA or CTA AND
2. Age 18-90 years*	2. Target Mismatch Profile on CT perfusion or MRI (ischemic core volume is < 70 ml, mismatch ratio is > 1.8 and mismatch volume* is > 15 ml) Alternative neuroimaging inclusion criteria (if perfusion imaging or CTA/MRA is technically inadequate):
3. Baseline NIHSSS is ≥ 6 and remains ≥6 immediately prior to randomization*	A) If CTA (or MRA) is technically inadequate: Tmax>6s perfusion deficit consistent with an ICA or MCA-M1 occlusion AND Target Mismatch Profile (ischemic core volume is < 70 ml, mismatch ratio is >1.8 and mismatch volume is >15 ml as determined by RAPID software) 6
4. Endovascular treatment can be initiated (femoral puncture) between 6 and 16 hours of stroke onset. Stroke onset is defined as the time the patient was last known to be at their neurologic baseline (wake-up strokes are eligible if they meet the above time limits).*	B) If MRP is technically inadequate: ICA or MCA-M1 occlusion (carotid occlusions can be cervical or intracranial; with or without tandem MCA lesions) by MRA (or CTA, if MRA is technically inadequate and a CTA was performed within 60 minutes prior to the MRI) AND DWI lesion volume < 25 ml
5. modified Rankin Scale less than or equal to 2 prior to qualifying stroke (functionally independent for all ADLs)*	C) If CTP is technically inadequate: Patient can be screened with MRI and randomized if neuroimaging criteria are met.
6. Patient/Legally Authorized Representative has signed the Informed Consent form.
Clinical Exclusion Criteria	Neuroimaging Exclusion Criteria
1. Other serious, advanced, or terminal illness (investigator judgment) or life expectancy is less than 6 months.	ASPECT score <6 on non-contrast CT (if patient is enrolled based on CT perfusion criteria)
2. Pre-existing medical, neurological or psychiatric disease that would confound the neurological or functional evaluations	2. Evidence of intracranial tumor (except small meningioma) acute intracranial hemorrhage, neoplasm, or arteriovenous malformation
3. Pregnant	3. Significant mass effect with midline shift
4. Unable to undergo a contrast brain perfusion scan with either MRI or CT^	4. Evidence of internal carotid artery dissection that is flow limiting or aortic dissection
5. Known allergy to iodine that precludes an endovascular procedure	5. Intracranial stent implanted in the same vascular territory that precludes the safe deployment/removal of the neurothrombectomy device
6. Treated with tPA >4.5 hours after time last known well	6. Acute symptomatic arterial occlusions in more than one vascular territory confirmed on CTA/MRA (e.g., bilateral MCA occlusions, or an MCA and a basilar artery occlusion)
7. Treated with tPA 3-4.5 hours after last known well AND any of the following: age >80, current anticoagulant use, history of diabetes AND prior stroke, NIHSS >25
8. Known hereditary or acquired hemorrhagic diathesis, coagulation factor deficiency; recent oral anticoagulant therapy with INR > 3 (recent use of one of the new oral anticoagulants is not an exclusion if estimated GFR > 30 ml/min).*
9. Seizures at stroke onset if it precludes obtaining an accurate baseline NIHSS
10. Baseline blood glucose of 400mg/dL (22.20 mmol)
11. Baseline platelet count < 50,000/uL*
12. Severe, sustained hypertension (Systolic Blood Pressure >185 mmHg or Diastolic Blood Pressure >110 mmHg)
13. Current participation in another investigational drug or device study
14. Presumed septic embolus; suspicion of bacterial endocarditis
15. Clot retrieval attempted using a neurothrombectomy device prior to 6 hours from symptom onset
16. Any other condition that, in the opinion of the investigator, precludes an endovascular procedure or poses a significant hazard to the subject if an endovascular procedure was performed.

^*indicates criteria which were able to be analyzed for the feasibility analysis. Of note, modified Rankin scores were calculated retrospectively, unless clearly documented during the stroke evaluation. This method has been shown to be reliable and unbiased in prior work. ¹⁷

^{^}indicates that surrogate markers (patients with pacemakers or automated implanted cardiac defibrillators) were used to estimate the number of patients unable to undergo MRI.

Estimated enrollment with initially proposed versus revised eligibility criteria:

Actual monthly enrollment in the trial using the revised criteria was plotted and estimated by a non-linear function using R (version 4.10, The R Foundation for Statistical Computing). We then calculated the monthly enrollment numbers, if the initial I/E criteria were not modified, by estimating the fraction of the actual enrollment computed based on the feasibility assessment. Estimates were projected out to reach a sample size of 200 (the first of two planned interim analyses). The DEFUSE 3 statistical analysis plan (SAP)⁹ was reviewed to obtain details on the Data and Safety Monitoring Board (DSMB) stopping rules for the early interim analysis. The DSMB chose to use a two-sided efficacy boundary computed using the O’Brien-Fleming spending function with a maximum sample size of 376. We applied this same stopping rule using the estimated enrollment using the initially proposed criteria at the time the trial was suspended to determine if the boundary would have been crossed with the reduced sample size. The SAP specified that the primary endpoint, 90-day mRS, was tested using the Wilcoxon rank sum statistic using the PROC NPAR1WAY procedure in SAS (SAS Institute Inc. Cary, NC).

Results:

DEFUSE-3 Feasibility Assessment

Population eligibility based on initial eligibility criteria:

In 2005, there were 1843 ischemic stroke patients arriving to an emergency department in our five-county area, with a median age of 73 years old of whom 56% were female and 23% were black. An estimated 2.4% (46 of 1843) of stroke patients would have qualified for the trial based on initial I/E criteria, not including the proposed perfusion imaging parameters (as these were not available in the GCNKSS database from 2005). The impact of key inclusions/exclusion criteria in reducing study eligibility is shown in Table 2. The complete initial assessment presented to StrokeNet is included in Supplementary Material.

Table 2:

Final Exclusion Criteria and Numbers expected to be Excluded from GCNKSS 2005

Exclusion Criterion	Initial and revised exclusion	Number of GCNKSS patients that would have been ineligible by only this criterion before revision	Number of GCNKSS patients that would have been ineligible by only this criterion after revision
Age	>80; >90	10	4
Baseline mRS	0-1; 0-2	56	27
Time since LSN	<6 hrs, >12 hrs; <6 hrs, >16 hrs	236	210
NIHSS	≤8 or > 25; <6	225	141
Total Eligible Including all Criteria		46/1843 (2.4%)	74/1843 (4.0%)

Recommendations for revision of eligibility criteria:

Initially, three proposed criteria were identified as potentially being too exclusive and later modified based on epidemiological feedback: upper age limit was increased to 90 years old from 80 years old; pre-stroke disability was expanded from mRS ≤1 to mRS ≤ 2; and NIHSS on presentation was decreased from ≥ 8 to of ≥ 6. After discussion with the PIs and a StrokeNet survey was completed, another criterion, time from last seen normal, was increased from 12 hours to 16 hours. The result of this analysis is presented in Table 2, including the proportion excluded only by the revised entry items. After these criteria were revised based on the epidemiology feedback, an estimated 4.0% (74 of 1843) of stroke patients would have qualified for the trial.

Clinical trial performance in consideration of initial and final eligibility criteria:

DEFUSE 3 recruited at goal during its study period, enrolling 182 patients at 38 centers over 13 months. Overall, 57% of the patients enrolled in the study qualified based on the newly adjusted study entry criteria (aged 81-90 years: 40 patients ; mRS of 2: 13 patients; time since last seen well between 12 and 16 hours: 60 patients; NIHSS 6 or 7: 34 patients; of note, patients may have been ineligible by multiple initially proposed criteria simultaneously). The first of two planned interim analyses was to occur after 200 patients had data on 90-day outcomes. Figure 1 demonstrates the predicted enrollment if the I/E criteria had not been revised as compared to the actual enrollment. We estimate that if the initial I/E criteria had not been modified, the first planned interim analysis would have occurred thirteen months later in July of 2018 rather than in June 2017. Actual enrollment was held on May 27, 2017 due to the reporting of the DAWN study results, which suggested a large treatment effect in a comparable patient population and prompted the interim analysis to be performed early.

Figure 1:

Enrollment in DEFUSE 3: Initial Vs Final with Epidemiological Feedback

Actual enrollment of DEFUSE 3 seen in black, which was modified based on epidemiological feedback. Expected enrollment if initial I/E criteria were not modified is represented in red.

To accommodate the early interim analysis, the DSMB chose to use the O’Brien-Fleming spending function with a maximum sample size of 376 which gives a two-sided efficacy boundary Z-score of 3.018. This boundary is based on the amount of information available at the time of the interim look which is a fraction of the total sample size at that time (182/376). If the initial I/E criteria had not been modified, the estimated fraction of the sample size would have been much lower at 0.21 (80/376 estimated) resulting in a much higher O’Brien-Fleming boundary Z-score of 4.717 (Table 3). Using the distribution of the 90-day mRS scores reported in the DEFUSE 3 paper, the Wilcoxon rank sum statistic Z-score is 3.729. Therefore, the actual trial using the revised eligibility criteria was stopped because the efficacy boundary was crossed (3.729 > 3.018 boundary), but we estimate that the trial may not have been stopped if the enrollment had been much lower due to stricter I/E criteria, leading to a reduced fraction of information at the time of trial suspension (3.729 < 4.717 boundary).

Table 3.

DEFUSE 3 Actual and Estimated Data and Safety Monitoring Board Stopping Rules.

	Actual		Estimated I/E if criteria not expanded
Look	Information Fraction	2-sided Efficacy Boundary (Z-scale)	Information Fraction	2-sided Efficacy Boundary (Z-scale)
Interim	0.48 (182/376)	±3.018	0.21 (80/376)	±4.717
Final	1.0	±1.967	1.0	±1.960

Discussion:

In this descriptive analysis, we demonstrate that prospective feasibility assessment of a clinical trial using population-based epidemiological data broadened the eligibility of trial participants and thereby accelerated trial enrollment. Our estimates indicate that, if the DEFUSE 3 trial I/E criteria had not been revised, the study would not have demonstrated efficacy at the time of the interim analysis. Furthermore, by increasing recruitment by 57%, the feasibility assessment and expanded I/E criteria saved approximately 13 months of recruitment time to achieve the same sample size, which might have translated to about $3 million additional dollars beyond the total actual study expenditures of $5.3 million.

Future studies should consider utilizing epidemiologic data for feasibility assessments. Critically appraising inclusion and exclusion criteria and informing decision making with an objective population-based epidemiological resource can lead to faster, broader trial recruitment and cost savings. Additionally, using a population-based resource such as the GCNKSS, which does not have the same referral and diagnostic biases of data from single academic centers, can lead to a more inclusive study with more widely generalizable results. A significant point, and what makes this process unique to our knowledge, is that I/E criteria are debated and considered based on actual quantitative estimates of the effect of a given criteria, rather than the typically utilized “experienced trialist opinion” of what are the best I/E criteria.

Despite numerous changes to methodology and processes, recruitment rates specifically in stroke trials have been poor and may be worsening over time.¹⁴ Low recruitment rates prolong trial duration and frequently lead to trial amendments,¹⁵ costing as much as ~$500,000 and further delaying patient recruitment and trial completion.¹⁶ Studying these issues in clinical trial implementation can be difficult as many trials don’t publish their screening logs, and thus the effect of particular criteria on overall recruitment may be difficult to assess. Feasibility assessments will not eliminate issues related to randomized clinical trial implementation and have limitations of their own if they are unable to include key imaging or clinical data in the database used to assess eligibility. Furthermore, this analysis does not assess individual patient’s willingness to participate, the availability of trial staff to enroll, competing trials which may detract from a single trial’s enrollment or language barriers that inhibit enrollment. But, if used in a conscientious and objective manner, feasibility assessments employing well-characterized and representative databases can expedite enrollment and help to ensure successful trial completion.

This study does have limitations that require acknowledgment. DEFUSE 3 was stopped early due to external data (prompted by the results of DAWN), and an early interim analysis demonstrated efficacy. This limits our ability to describe how final enrollment figures and timelines might have been affected if the trial ran its full course. Additionally, while population-based studies such as GCNKSS collect a substantial amount of clinical data associated with stroke cases, they often do not have all data regarding specific variables relevant for full estimation of trial eligibility. For DEFUSE 3, as an example, CT perfusion imaging was not routinely performed clinically at the time of the feasibility assessment and thus rates of favorable estimates were informed by the literature and expert opinion. Finally, this is a single example of our process described herein. The DEFUSE 3 trialists may have come to the same conclusions and made similar adjustments without GCNKSS data or epidemiological input. We intend to evaluate other trials from NIH StrokeNet that have used this process as they are completed to see if a similar benefit was provided.

In summary, we show that a clinical trial feasibility assessment, evaluating inclusion and exclusion criteria using an objective, population-based epidemiological resource through a collaborative and iterative process with epidemiologists, improves recruitment into clinical trials, likely reducing the time required to perform trials and thereby substantially reducing cost.

Non-standard Abbreviations and Acronyms

NIH

National Institutes of Health

DEFUSE 3

Endovascular Therapy Following Imaging Evaluation for Ischemic Stroke 3

GCNKSS

Greater Cincinnati/Northern Kentucky Stroke Study

ICD

International Classification of Diseases

I/E criteria

inclusion/exclusion criteria

SAP

statistical analysis plan

DSMB

Data Safety and Monitoring Board

NIHSS

National Institutes of Health Stroke Scale

mRS

modified Rankin Scale