Factors Associated with Time to Site Activation, Randomization, and Enrollment Performance in a Stroke Prevention Trial

Abstract

Background and Purpose

Multicenter clinical trials attempt to select sites that can move rapidly to randomization, and enroll sufficient numbers of patients. However, there are few assessments of the success of site selection.

Methods

In the Carotid Revascularization and Medical Management for Asymptomatic Carotid Stenosis Trials (CREST-2), we assess factors associated with the time between site-selection and authorization to randomize, the time between authorization to randomize and the first randomization, and the average number of randomizations per site per month. Potential factors included characteristics of the site, specialty of the Principal Investigator (PI), and site type.

Results

For 147 sites, the median time between site-selection to authorization to randomize was 9.9 months (Interquartile Range (IQR) 7.7, 12.4), and factors associated with early site activation were not identified. The median time between authorization to randomize and a randomization was 4.6 months (IQR 2.6, 10.5). Sites with authorization to randomize in only the Carotid Endarterectomy (CEA) study were slower to randomize, and other factors examined were not significantly associated with time to randomization. The recruitment rate was 0.26 (95% CI: 0.23 – 0.28) patients per site per month. By univariate analysis, factors associated with faster recruitment were authorization to randomize in both trials, principal investigator specialties of interventional radiology and cardiology, pre-trial reported performance > 50 Carotid Angioplasty and Stenting (CAS) procedures per year, status in the top half of recruitment in the CREST trial, and classification as a private health facility. Participation in StrokeNet was associated with slower recruitment as compared to the non-StrokeNet sites.

Conclusion

Overall, selection of sites with high enrollment rates will likely require customization to align the sites selected to the factor under study in the trial.

Introduction

Critical for success of large multi-center Phase III clinical trials is the identification of clinical sites that can: 1) quickly move through the steps required for approval of randomization, 2) rapidly initiate randomization, and 3) enroll a large number of patients to the trial, including women and minorities. To meet this challenge, the leadership of the Carotid Revascularization and Medical Management for Asymptomatic Carotid Stenosis Trials (CREST-2) created a Site Selection Committee (SSC). Similar to the approach taken in many other trials,^1–3 the SSC established criteria presumed to be associated with capabilities of quick site initiation and large enrollment, and the committee also developed a site questionnaire for the collection of data to evaluate potential study sites. In addition, CREST-2 established linkage to the National Institute of Neurological Disorders and Stroke (NINDS)-funded StrokeNet network, consisting of 25 regional coordinating centers designed to maximize efficiencies of conducting high quality, multi-site clinical trials in stroke.^{4, 5}

Few studies have formally assessed if information available at the time of site selection reliably identifies sites with good performance. Those doing so primarily focused on a single factor as a potential predictor of performance.^6–8 At the time of site selection, we could not find a report of a comprehensive assessment of information available with either the time to initiate randomization or the number of patients recruited, which is the goal of these analyses.

Methods

The CREST-2 protocol includes a pair of randomized clinical trials.⁹ One trial compares outcomes of approximately 1,240 patients randomized to carotid endarterectomy plus intensive medical management (IMM) versus IMM alone (the CEA study). The other trial compares the outcomes of approximately 1,240 patients randomized to carotid stenting plus IMM versus IMM alone (the CAS study).

The SSC has evaluated applications from 192 sites during 60 Committee meetings from March 5^th 2014 to January 9^th, 2017. The committee identified factors presumed to predict quicker initiation of randomization and higher enrollment: specialty of the site principal investigator (interventional radiology/neuroradiology, neurology, neurosurgery, vascular surgery or cardiology), site type (private hospital, private office, Veteran’s Affairs medical center, or academic medical center), presence of affiliate or satellite recruiting sites (yes, no), membership in StrokeNet (yes, no), use of a central IRB (yes, no), presence of the full complement of investigators (full, partial), the reported annual number of CEA procedures performed at the site (<10, 10–25, 26–50, or >50), the reported annual number of CAS procedures performed at the site (<10, 10–25, 26–50, or >50), seeking approval for CAS only, CEA only, or both studies, and participation and performance in the Carotid Revascularization, Endarterectomy versus Stent Trial (CREST).¹⁰ Potential to enroll women and minorities were additional criteria.

The time between a site being approved as a CREST-2 site and the first randomization can be divided into two segments: 1) the period from approval to the time of meeting the requirements to be given authorization to randomize, and 2) the period between receiving authorization to randomize and the time of performing the first randomization. With the approval by the SCC, a site may begin the process of completing the prerequisites required for authorization to randomize patients. This includes securing IRB approval, completion of contracts with the Clinical Coordinating Center (CCC) at Mayo Clinic in Florida, submission and review of information to qualify both surgeons to perform the CEA and the interventionists to perform the CAS, and the training and certification of the clinical center staff on matters of the protocol and data management. With the completion of these steps, the site is given authorization to begin randomization (given the “green light” letter). Because a site may qualify surgeons but not interventionists, or interventionists but not surgeons, they can be given authorization to begin the CEA or CAS study at different times.

For this report, the time between approval and authorization to randomize, the time between authorization to randomize and the first randomization, and the average monthly enrollment per site were analyzed as indices of site performance. The association between potential factors associated with performance on the two time intervals was assessed using standard time-to-event (i.e., survival) methodology. The Kaplan-Meier method was used to estimate the proportion with authorization to randomize as a function of time since approval. The significance of the predictive factors was estimated using proportional hazards analysis, with a plan to perform multivariable analysis should more than one factor be significant on univariate testing. An identical approach was used to assess the factors associated with the time between authorization to randomize and first randomization. The association between potential factors associated with recruitment volume was assessed using Poisson regression, which was used to both estimate the recruitment per clinic per month (with 95% confidence intervals) and the recruitment ratio between different strata of the potential predictor. Multivariable analysis was utilized if more than one factor was significant on univariate testing.

Results

As of January 9, 2017, 147 of the 192 sites reviewed were approved for participation in CREST-2. Of these, 122 (83%) completed regulatory and training requirements and were permitted to randomize (i.e., received a “green light” letter); 89 of these 122 (73%) sites had randomized one or more patients. The characteristics of all sites, the sites which have been approved for randomization, and those which have randomized one or more patients, are summarized in Table 1. Over 1,713 clinic-months, the 122 permitted sites had performed 437 randomizations, for an overall recruitment rate of 0.26 (95% CI: 0.23 – 0.28) patients per site per month.

Table 1.

Description of sites overall, sites approved for randomization (“green-lighted”) and sites that have randomized (n and (%) of sites).

		All	Approved for Randomization	Randomized 1 or more patients
Number of Sites		147	122	89
PI Specialty	Interventional Radiology/Neuroradiology	6 (4%)	6 (5%)	4 (4%)
	Neurology	43 (29%)	33 (27%)	21 (24%)
	Neurosurgery	14 (10%)	8 (7%)	7 (8%)
	Vascular Surgery	44 (30%)	40 (33%)	31 (35%)
	Cardiology	40 (27%)	35 (29%)	26 (29%)
Site Type	Private Hospital	50 (34%)	39 (32%)	24 (27%)
	Private office	18 (12%)	17 (14%)	16 (18%)
	VA Medical Center	8 (5%)	7 (6%)	6 (7%)
	Academic	71 (48%)	59 (48%)	43 (48%)
Affiliates	Yes	45 (31%)	33 (27%)	22 (25%)
	No	102 (69%)	89 (73%)	67 (75%)
StrokeNET Membership	StrokeNET	57 (39%)	46 (38%)	33 (37%)
	Non-StrokeNET	90 (61%)	75 (62%)	56 (63%)
CIRB	Yes	75 (51%)	62 (51%)	46 (52%)
	No	72 (49%)	60 (49%)	43 (48%)
Team Complement	Full	123 (84%)	105 (86%)	77 (87%)
	Partial	24 (16%)	17 (14%)	12 (13%)
Annual number of CEA	<10	10 (7%)	8 (7%)	7 (8%)
	10–25	14 (10%)	11 (9%)	5 (6%)
	26–50	35 (24%)	30 (25%)	20 (22%)
	>50	88 (60%)	73 (60%)	57 (64%)
Annual number of CAS	<10	29 (20%)	23 (19%)	15 (17%)
	10–25	48 (33%)	40 (33%)	31 (35%)
	26–50	41 (28%)	35 (29%)	26 (29%)
	>50	29 (20%)	24 (20%)	17 (19%)
Study approval by the site selection committee	CAS only	8 (5%)	5 (4%)	5 (6%)
	CEA only	11 (7%)	8 (7%)	4 (4%)
	Both	128 (87%)	109 (89%)	80 (90%)
Study participation to randomize	Neither	25 (17%)	0 (0%)	0 (0%)
	CAS only	9 (6%)	9 (7%)	7 (8%)
	CEA only	39 (27%)	39 (32%)	19 (21%)
	Both	74 (50%)	74 (61%)	63 (71%)
Participation in CREST	Top Half	36 (24%)	35 (29%)	29 (33%)
	Bottom Half	35 (24%)	29 (24%)	22 (24%)
	Not in CREST	76 (52%)	58 (48%)	39 (44%)

The proportion of clinical sites with authorization to randomize as a function of time since approval is provided in Figure 1A. The minimum time between approval and authorization to randomize was 3.3 months, with 10% of sites receiving authorization in 5.2 months, 25% in 7.7 months, 50% in 9.9 months, 75% in 12.4 months, and 90% in 17.3 months.

Figure 1.

Figure 1A and 1B: The proportion of sites approved for randomization (green lighted sites) as a function of time since approval by the site selection committee (left panel), also shown by participation in the previously-conducted CREST trial (right panel).

None of the factors associated with the time interval between site approval and the authorization to randomize met the p < 0.05 criteria. Only prior participation in CREST approached (p = 0.055) significance. Compared to sites not in the CREST study, those sites that were in the top half of CREST recruitment had significantly longer time between approval and authorization to randomize (HR = 0.59; 95% CI: 0.38 – 0.91), with those in the bottom half of CREST recruitment having non-significantly longer time to approval to randomize (HR = 0.88; 95% CI: 0.56 – 1.38). The Kaplan-Meier estimates of the proportion with authorization to randomize shown by CREST participation status is shown in Figure 1B. Because only one factor was marginally significant, no multivariable analysis was performed, and because the other results were non-significant, they are detailed in the supplemental material.

Of the 122 sites with authorization to randomize, the proportion performing a randomization as a function of time since that authorization is shown in Figure 2A. The shortest time between authorization to randomize and a randomization was 0.2 months, with 10% of the sites performing a randomization within 0.9 months, 25% within 2.6 months, 50% within 4.6 months, and 75% within 10.5 months.

Figure 2.

Figure 2A and 2B: The proportion of sites approved for randomization that had randomized one or more patients as a function of time since approval for randomization (left panel), also shown by whether the site was approved for randomization only in the CEA plus intensive medical management vs intensive medical management study, only in the CAS plus intensive medical management versus intensive medical management study, or in both studies.

The only factor significantly associated with the time between approval and randomization was whether the site was approved to randomize to both CEA and CAS, or to CEA-only, or to CAS-only (p = 0.014). As shown in Figure 2B, compared to those with authorization to randomize in both the CEA and CAS study, those with authorization for CEA-only were much slower to randomize (HR = 0.47; 95% CI: 0.28 – 0.80), while those with authorization for CAS-only were non-significantly slower (HR = 0.92; 95% CI: 0.42 – 2.02). The estimates for the impact of other factors on the time between approval and randomization are provided in Supplemental Table I (all p ≥ 0.15). Because only one factor was significant, no multivariable analysis was performed.

Table 2 provides the description between the associated factors and the average monthly recruitment. On univariate analysis, several factors had a large impact on the recruitment rate. Recruitment was higher among those with authorization to randomize in both trials (0.30/month; 95% CI: 0.27 – 0.33) or to CAS-only (0.30/month; 95% CI: 0.22 – 0.42) compared to those with authorization to randomize only to CEA (0.13; 95% CI: 0.10 – 0.16). Among the principal investigator specialties, the highest monthly recruitment rates were for interventional radiologists (0.34/month; 95% CI: 0.24 – 0.47) and cardiologists (0.33/month; 95% CI: 0.29 – 0.33); with sites led by neurologists having almost half the recruitment rate (0.18/month; 95% CI: 0.14 – 0.23). Participation in StrokeNet was also associated with slower recruitment (0.17/month; 95% CI: 0.14 – 0.21) as compared to the non-StrokeNet sites (0.30/month; 95% CI: 0.27 – 0.33). Sites that reported performing more than 50 CAS procedures per year recruited substantially faster (0.37/month; 95% CI: 0.31 – 0.44) than sites reporting less than 50 CAS procedures per year (all ≤ 0.23/month). Those in the top half of recruitment in the CREST trial had a recruitment rate much higher (0.30/month) than either those in the bottom half of CREST recruitment (0.24/month; 95% CI: 0.27 – 0.33) or those not in the CREST study (0.22/month; 95% CI: 0.18 – 0.26). Recruitment was much faster in sites that were either a private hospital (0.31/month; 95% CI: 0.26 – 0.36) or private office (0.31/month; 95% CI: 0.26 – 0.38) compared to academic sites (0.20/month; 95% CI: 0.17 – 0.24). There were not significant univariate differences in the recruitment rate (p > 0.05) depending on whether sites had affiliates, used a central IRB, had their full complement of investigators, had a larger number of CEA procedures performed at their site annually, or what trial they were approved to conduct by the Site Selection Committee.

Table 2.

Statistically significant (either univariate or multivariable) factors associated with the number of patients recruited per site per month. Data by columns are: 1) number of patients recruited and cumulative number of recruitment months, 2) estimated recruitment rate per month (with 95% CI), 3) relative recruitment rates (with 95% CI), 4) univariate test of differences in recruitment rates, 5) multivariable relative recruitment rates, and 6) multivariable test of differences in recruitment rates. Analysis was also performed for whether the site had affiliates recruitment partners and whether the site was at full staff complement, with neither of these factors being significant (p > 0.1) in either univariate or multivariable analysis (these factors also remain in the multivariable model).

Factor	Level	Univariate				Multivariable
		Recruit/Clin Mths	Recruit Rate per Month (95% CI)	Recruitment Ratio (95% CI)	P-Value	Recruitment Ratio (95% CI)	P-Value
PI Specialty	Interventional Radiology or Neuroradiology	170 513	0.34 (0.24 – 0.47)	1.01 (0.71 – 1.45)	<0.001	1.26 (0.79 – 2.01)	0.39
	Neurology	36 107	0.18 (0.14 – 0.23)	0.55 (0.42 – 0.72)	0.001	0.84 (0.57 – 1.24)
	Neurosurgery	73 404	0.23 (0.16 – 0.33)	0.69 (0.46 – 1.04)		0.92 (0.54 – 1.57)
	Vascular Surgery	27 118	0.23 (0.19 – 0.27)	0.69 (0.55 – 0.87)		1.12 (0.78 – 1.61)
	Cardiology	131 571	0.33 (0.29 – 0.39)	1.00 (ref)		1.00 (ref)
Site Type	Private Hospital	148 482	0.31 (0.26 – 0.36)	1.52 (1.22 – 1.89)		1.48 (1.08 – 2.03)	<0.001
	Private office	95 303	0.31 (0.26 – 0.38)	1.55 (1.20 – 1.99)		1.37 (0.91 – 2.05)
	VA Medical Center	26 97	0.27 (0.18 – 0.40)	1.33 (0.88 – 2.01)		3.71 (2.14 – 6.46)
	Academic	168 831	0.20 (0.17 – 0.24)	1.00 (ref)		1.00 (ref)
StrokeNET participation	StrokeNET	106 608	0.17 (0.14 – 0.21)	0.58 (0.47 – 0.72)	<0.001	1.00 (0.73 – 1.36)	0.98
	Non-StrokeNET	331 1,105	0.30 (0.27 – 0.33)	1.00 (ref)		1.00 (ref)
Number of CEA	<10	48 133	0.36 (0.27 – 0.48)	1.43 (1.06 – 1.95)	0.079	1.88 (1.24 – 2.85)	0.020
	10–25	22 107	0.20 (0.13 – 0.31)	0.81 (0.53 – 1.25)		1.51 (0.92 – 2.50)
	25–50	93 388	0.24 (0.20 – 0.29)	0.95 (0.75 – 1.20)		1.14 (0.88 – 1.49)
	>50	274 1,085	0.25 (0.22 – 0.28)	1.00 (ref)		1.00 (ref)
Number of CAS	<10	65 310	0.21 (0.16 – 0.27)	0.56 (0.42 – 0.76)	<0.001	0.57 (0.38 – 0.85)	<0.001
	10–25	128 556	0.23 (0.19 – 02.7)	0.62 (0.49 – 0.79)		0.58 (0.42 – 0.79)
	25–50	103 467	0.22 (0.18 – 0.27)	0.59 (0.46 – 0.77)		0.42 (0.31 – 0.57)
	>50	141 380	0.37 (0.31 – 0.44)	1.00 (ref)		1.00 (ref)
Approval by site selection committee	CAS only	25 72	0.35 (0.23 – 0.51)	1.37 (0.91 – 2.05)	0.22	2.20 (1.20 – 4.04)	0.04
	CEA only	13 66	0.20 (0.12 – 0.34)	0.78 (0.45 – 1.36)		1.63 (0.74 – 3.59)
	Both	399 1,575	0.25 (0.23 – 0.28)	1.00 (ref)		1.00 (ref)
Green light authorization for randomization	CAS only	35 115	0.30 (0.22 – 0.42)	1.01 (0.72 – 1.43)	0.006	0.54 (0.34 – 0.85)	<0.001
	CEA only	56 446	0.13 (0.10 – 0.16)	0.42 (0.32 – 0.55)		0.28 (0.19 – 0.41)
	Both	346 1,152	0.30 (0.27 – 0.33)	1.00 (ref)		1.00 (ref)
CREST Participation	Top Half	198 650	0.30 (0.27 – 0.35)	1.40 (1.13 – 1.74)	<0.001	1.64 (1.27 – 2.11)	0.001
	Bottom Half	98 415	0.24 (0.19 – 0.29)	1.08 (0.84 – 1.40)		1.58 (1.16 – 2.15)
	Not in CREST	141 648	0.22 (0.18 – 0.26)	1.00 (ref)		1.00 (ref)

The results of the multivariable analysis of the monthly recruitment rate is also provided in Table 2, with the powerful factors being the site type (with VA medical centers showing a dramatic increase), the number of CAS procedures per year at baseline, what trials into which the site has authorization to randomize (CEA and CAS; CEA-only; CAS-only), and CREST participation and performance. In this multivariable analysis, the association between principal investigator specialty and StrokeNet membership became non-significant. These changes in the magnitude of association arise from a substantial collinearity between the predictor variables. Specifically, comparing StrokeNet to non-StrokeNet sites, 67% versus 37% were academic, 47% versus 18% had a neurologist PI, 7% versus 28% had more than 50 CAS procedures annually, 42% versus 17% could randomize only to CEA, and 17% versus 29% were in the top half of recruitment in CREST. As such, the StrokeNet sites had a higher frequency of all the powerful factors associated with slow recruitment, and after adjustment for these differences these factors there was no difference between the StrokeNet and non-StrokeNet sites. Conversely, univariately the VA medical centers recruited non-significantly faster than academic centers (recruitment ratio = 1.33; 95% CI: 0.88 – 2.01). However, they achieved this marginally higher recruitment rate despite having a higher frequency of a powerful factor predicting low recruitment, specifically with 0.0% with > 50 CAS procedures compared to 20% in academic centers, and 75% being approved for only CEA randomization compared to 30% in academic centers.

Discussion

In what we believe is one of the first comprehensive assessments, we report time from site selection to randomization, assess factors associated with rapid or slow activation, and examine site characteristics associated with early enrollment. Site activation was prolonged with one-year required for activation of three-quarters of the sites, and only prior participation in CREST was near significance (p=0.055) as a predictor of early activation. Initiation of enrollment was also prolonged with 10.5 months required for 75% of the sites to achieve enrollment of one or more patients. In contrast to site activation, factors associated with early enrollment were identified and include site and investigator characteristics as will be will be discussed below.

Complexity of the start-up process impedes identifying factors associated with the selection of sites that initiate randomization quickly. For example, the use of a central IRB may speed IRB approval, but would not affect the time to establish contracts or credential clinicians to perform study-approved procedures. Participation in an established network may speed time for contracts, but would have no effect on the time to credential clinicians. Importantly, qualifying for randomization is a multi-step process and tactics to accomplish individual steps does not speed the process unless that step is on the “critical path” to overall approval. For example, credentialing of clinicians to perform the procedures was a remarkable barrier to beginning randomization in CREST-2, so the benefit of quickly moving through IRB and contract approval (factors not on the critical path) was of little value to getting the site quickly approved for randomization

Unlike the absence of factors associated with time-to-randomization, there were a substantial number of factors associated with the average monthly recruitment per site. There seems to be a consistent pattern of associated factors that may inform the selection of sites for future trials. For example, a powerful predictor of high recruitment was an ability to qualify for randomization in the CAS trial. CAS-sites may have more incentive to enroll patients due to the limited reimbursement for CAS outside of clinical trials. The low recruitment in sites led by neurologists versus cardiologists is likely a reflection of this same factor, as cardiologists would be more likely to qualify for the CAS trial and hence would have an advantage in qualifying for both trials. Those sites reporting a large number of CAS procedures are more likely to qualify for the CAS study. The low recruitment in StrokeNet sites could be similarly associated with low recruitment in sites led by neurologists, as 47% of the StrokeNet CREST-2 principal investigators were neurologists (compared to 18% in non-StrokeNet sites). All of this raises the observation that if a trial is studying a particular procedure/technology, expertise and clinical practice in that procedure/technology arena is a paramount qualification for participation.

Apparently, only a few other investigations have examined whether information available as part of the site selection process was predictive of performance in the trial. In a treatment trial of myasthenia gravis, the mean time for sites to achieve regulatory approval was reported to be shorter for sites in the United States (9.7 ± 0.7 months) than for non-US sites (13.4 ± 1.0 months).¹¹ The Antihypertensive and Lipid Lowering Treatment to Prevent Heart Attack Trial (ALLHAT) reported a median number of randomizations of 40 subjects at university sites, compared to 79 subjects at Veteran’s Affairs Sites, and 37 at private group practices; however, these estimates were not adjusted for the average number of months of recruitment at these different types of sites.⁸

This study has strengths and weaknesses. Perhaps the greatest strength is the heterogeneity of sites, allowing an analysis of factors associated with successful site start-up and performance. In addition, the large number of sites in CREST-2 allows reasonable power to detect differences in all three indices of performance. For example, with 122 sites receiving authorization to randomize, a hazard ratio of 1.66 can be detected with 80% power for a predictor factor that is approximately 50% prevalent, and a hazard ratio of 1.80 can be detected for a predictor factor that is 25% prevalent. However, there are also shortcomings. This paper is offered when 18% (437/2480) of the patients anticipated for recruitment have been randomized. We suggest that this is a minor weakness, as adequate power to detect differences in the recruitment rates was demonstrated by the large number of significant associations. Site selection bias, particularly for the rate of enrollment analysis, is a potential limitation. The array of potentially unique complexities of each trial and of each trial site are such that some caution should be taken during attempts to generalize the findings to other trials. Researchers planning future studies, of this type, may also consider investigating the association of site, site teams, and site investigators’ cumulative past trial experience with time to activation, randomization, and enrollment performance metrics.

Conclusion

Few studies have reported whether it is possible to preferentially select participating sites that are most likely to succeed in a large multi-center clinical trial. These findings suggest that much more needs to be learned about factors associated with sites that will move rapidly to randomization. However, powerful factors associated with sites’ early recruitment rate in CREST-2 were identified. Collectively these observations suggest that the selection of sites for high recruitment may need to be targeted and tailored to the treatment under assessment. Targeting sites in this manner could improve the efficiency of future clinical trials.