Relation of Study Design to Recruitment and Retention in CTN Trials

Abstract

Background

Recruitment and retention in randomized clinical trials are difficult in general and particularly so in trials of substance abuse treatments. Understanding trial design characteristics that could affect recruitment and retention rates would help in the design of future trials.

Objective

To test whether any of the following factors are associated with recruitment or retention: type of intervention, type of therapy, duration of treatment, total duration of trial, number of treatment sessions, number of follow-up visits, number of primary assessments, timing of primary assessments, number of case report form (CRF) pages at baseline, and number of CRF pages for the entire trial.

Methods

Recruitment and retention data from 24 Clinical Trials Network (CTN) trials conducted and completed between 2001 and 2010 were analyzed using single-factor analysis of variance and single-predictor regression methods to test their association with trial design characteristics.

Results

Almost all of the analyses performed did not show statistically significant patterns between recruitment and retention rates and the trial design characteristics considered.

Conclusion

In CTN trials, the relationship between assessment burden on participants and length of trial, on the one hand, and recruitment and retention, on the other, is not as strong and direct as expected. Other factors must impinge on the conduct of the trial to influence trial participation.

Scientific Significance

Researchers may deem slightly more justifiable to permit inclusion of some of the design features that previously were assumed to have a strong, negative influence on recruitment and retention, and should consider other strategies that may have a stronger, more direct effect on trial participation.

INTRODUCTION

Recently, the National Drug Abuse Treatment Clinical Trials Network (CTN) celebrated 10 years of conducting multisite clinical trials comparing the effectiveness of interventions for substance use disorders (1). Since 2001, the CTN has recruited more than 12,000 individuals into a series of studies that tested various substance abuse treatments in community treatment program settings. Descriptions of these trials and their main results have been published elsewhere (1–4). In general, the most frequent experimental designs fit into one of the following categories (3): (1) intervention (Tx) is compared to treatment as usual (TAU): Tx versus TAU; (2) intervention is added to TAU and compared to TAU alone: Tx + TAU versus TAU; or (3) intervention is added to TAU and compared to a control condition added to TAU: Tx + TAU versus control + TAU. A fourth design category applied in some CTN trials is comparing the intervention (Tx) to a standardized control treatment: Tx versus standard control (5). In CTN studies, each trial establishes prespecified targets for recruitment and retention, and the main eligibility criterion for participants is either being in treatment or seeking treatment. All participants are screened for eligibility according to criteria set by the trials and complete a process of written and verbal consent. During this process, investigators explain to participants the trial’s objectives, duration, assessments, and outcome schedules. Although investigators do not directly explore the “burden” of the study with participants during this process, the detailed explanations of study requirements present that information.

Much has been written about improving recruitment and retention of participants in clinical trials, and estimates suggest that 80% of randomized clinical trials struggle with recruitment and retention issues (6). Indeed, recruitment is often difficult and occurs more slowly than intended, leading to longer and more costly trials. Poor retention can limit the availability of outcome data and, therefore, the impact of the trial and interpretability of results. Many strategies to improve recruitment and retention have been reported (7), and these activities, including site selection strategies (8), are set forth as part of trial implementation. This article focuses on trial design characteristics, such as duration of treatment, duration of the entire trial, type of intervention, type of therapy, number of assessments, data-gathering procedures, and overall participant burden, to find out whether they are associated with the recruitment and retention of participants in CTN trials.

METHODS

Data on the first 24 completed CTN clinical trials were used for this analysis. The trials were conducted between January 2001 and September 2010. Their full name and a brief description of each have been published elsewhere (1,2,4).

In order for National Institute on Drug Abuse staff to monitor study progress, CTN’s Data and Statistics Center prepares monthly monitoring reports for all studies conducted. These reports include information regarding recruitment, demographics (gender, race/ethnicity, age group), availability of the primary outcome measure(s), treatment exposure, and attendance at follow-up visits. We analyzed data from these reports using single-factor analysis of variance (ANOVA) and single-predictor regression methods to test their association with trial design characteristics. All analyzed pairs of variables were plotted first to visually inspect any linear or nonlinear patterns and to identify outliers.

In this analysis, we defined recruitment in two ways: (1) the actual number of randomizations per site per week and (2) the ratio of actual to planned recruitment rate. The former reflects the trial’s ability to recruit participants and the latter reflects its ability to recruit participants as compared to what was planned. Retention is represented by three variables. The first is the availability of the primary outcome measure(s), which is calculated as the actual total count of non-missing primary endpoints divided by the total count of expected primary endpoints, across all assessment periods and all participants, regardless of whether they dropped out of the treatment or were lost to follow-up. It is a measure of the completeness of the data for the primary intention-to-treat analysis. Three trials (CTN0015, CTN0028, and CTN0032) had two co-primary endpoints. Because the inclusion of both primary endpoints in statistical models would violate the assumption of independent observations, only the first recorded co-primary endpoint was used, that is, drug use for CTN0015 and CTN0028, and receipt of HIV test result at 1 month for CTN0032. The second retention variable is treatment exposure, which is calculated as the total number of treatment sessions (in psychosocial interventions) received, or the number of tablets (in medication interventions) consumed by participants, divided by the total number of sessions participants were expected to receive, or the total number of tablets they were expected to consume. It is a measure of the treatment “dose” actually received as compared to the planned dose. The third measure of retention is attendance at follow-up visit(s), which is calculated as the total number of follow-up sessions participants actually attended divided by the total number of follow-up sessions participants were expected to attend. It is a measure of retention following the active treatment phase.

All studies were approved by local institutional review boards, and all participants signed informed consent prior to participating in the trials. For our analysis, we divided the studies into three intervention categories based on whether they primarily consisted of a medication treatment, a psychosocial treatment, or a combination of both. We also divided trials into three categories based on the type of therapy offered: individual, group, or combination. In some cases, this classification was a judgment call based on separating the added interventions specifically for the trial from what occurred as background treatment (TAU).

One trial (CTN0030-POATS) was conducted in two phases. Participants who relapsed at any time during the first phase were randomized again for the second phase. It was therefore more appropriate to use data of the first phase for analyses on recruitment, and of the second phase for analyses on retention.

RESULTS ON RECRUITMENT

Table 1 lists the trials considered, along with the type of intervention, type of therapy, number of participants randomized (sample size), number of participating sites, planned recruitment rates, actual recruitment rates, and range of actual recruitment rates across sites. The numbering sequence is incomplete, not because we excluded some clinical trials, but because some studies were surveys or other types of CTN research projects. The actual recruitment rate across trials ranged from .4 to 6.7 participants per site per week, and the overall recruitment rate for all trials combined was 1.0 participant per site per week. The actual recruitment rates from all 190 trial sites that participated in the 24 trials ranged from .2 to 8.3, reflecting a wide array of prespecified goals among CTN trials.

TABLE 1.

Recruitment information on trials included in the analyses.

Study	Type of intervention	Type of therapy1	Number of participants randomized	Number of participating sites	Planned randomizations per site per week	Actual randomizations per site per week	Range of actual randomizations across sites (min–max)
CTN0001-BUP1 inpatient	Medication	Individual	113	6	2.0	.7	.4–1.0
CTN0002-BUP2 outpatient	Medication	Individual	230	6	2.0	.9	.6–1.2
CTN0003-BUP3 taper	Medication	Individual	516	11	1.0	.6	.5–1.1
CTN0004-MET	Psychosocial	Individual	496	6	2.0	.9	.6–1.3
CTN0005-MI	Psychosocial	Individual	423	5	2.0	1.7	.8–2.1
CTN0006-MIEDAR drug-free	Psychosocial	Combination	454	8	2.0	.8	.5–1.1
CTN0007-MIEDAR methadone	Psychosocial	Combination	403	6	2.0	.8	.6–1.0
CTN0009-smoking	Combination	Combination	225	7	2.1	1.3	.7–2.22
CTN0010-BUP adolescent	Combination	Combination	154	6	.7	.4	.2–.7
CTN0011-TELE	Psychosocial	Individual	339	4	4.0	3.2	2.7–3.8
CTN0013-MET pregnant	Psychosocial	Individual	200	4	.7	.6	.4–.7
CTN0014-BSFT	Psychosocial	Combination	480	8	1.2	.7	.6–.9
CTN0015-seeking safety	Psychosocial	Group	353	7	2.0	.7	.2–1.4
CTN0017-HIV	Psychosocial	Individual	632	7	2.0	1.7	.7–3.3
CTN0018-safe sex for men	Psychosocial	Group	594	14	.9	1.1	.4–2.0
CTN0019-safe sex for women	Psychosocial	Group	515	12	.9	.9	.5–1.4
CTN0020-job seekers	Psychosocial	Group	628	11	.9	1.1	.7–1.5
CTN0021-Spanish MET	Psychosocial	Individual	462	6	1.5	1.2	1.0–1.7
CTN0027-START	Medication	Individual	1269	9	2.0	1.0	.4–1.3
CTN0028-ADHD adolescent	Combination	Individual	303	11	.5	.4	.2–.5
CTN0029-ADHD adult	Combination	Individual	255	6	.6	.6	.4–.7
CTN0030-POATS phase I	Combination	Individual	653	11	.7	.7	.3–1.2
CTN0031-STAGE12	Psychosocial	Combination	471	10	1.0	.9	.7–1.4
CTN0032-HIV rapid testing	Psychosocial	Individual	1281	12	8.0	6.7	3.0–8.3
Overall 24 trials			11449	1903		1.0	.2–8.3

Notes: CTN, Clinical Trials Network.

¹Based on the interventions being compared, not on the interventions received as part of treatment as usual that are common to all treatment conditions.

²Three sites that recruited a total of 31 participants during a total of 13 days were excluded for this calculation.

³Some sites participated in more than one trial. The total of 190 represents 120 different sites.

Of the 24 clinical trials analyzed, 19 (79%) had actual recruitment rates lower than the corresponding target; in three trials (13%), the actual and target randomization rates were equal; and in only two trials (8%), the actual recruitment rate was higher than planned. As expected, the correlation between planned and actual recruitment rates was high (.95).

As shown in Table 2, the 24 CTN multisite clinical trials recruited 11,449 individuals with the following characteristics: 59% male and 41% female; 57% white, 22% African American, and 7% multi-raced; 17% Hispanic and 82% non-Hispanic; 6% 17 years old or younger and 90% between the ages of 18 and 55 years.

TABLE 2.

Demographic composition of trial participants (n = 11,449).

	Count	Percent
Gender
Male	6795	59
Female	4646	41
Missing	8	<.1
Race
White	6476	57
African American	2465	22
Multi-race	797	7
American Indian	169	1
Other	775	7
Missing or choose not to answer	767	7
Ethnicity
Non-Hispanic	9392	82
Hispanic	1966	17
Missing or choose not to answer	91	1
Age
≤17 years	729	6
18–55 years	10,321	90
>55 years	389	3
Missing	10	<.1

Using single-predictor regression for continuous independent variables and single-factor ANOVA for categorical independent variables, recruitment rates were modeled individually against the following independent variables: type of intervention (medication, psychosocial, or combination), type of therapy (individual, group, or combination), duration of treatment, duration of trial, number of treatment sessions, number of follow-up visits, number of case report form (CRF) pages at baseline, and total number of CRF pages for the entire trial. Table 3 shows the range of these trial design characteristics across the 24 trials. The number of CRF pages is a rough proxy for the complexity of the clinical trial and its burden on participants. Because the amount of information collected on any one CRF page varies across instruments and trials, it is not a precise representation of participation burden.

TABLE 3.

Recruitment analyses (n = 24).

Trial design characteristic (independent variable)	Range	Actual recruitment rate			Ratio of actual to planned recruitment rate
		Parameter estimate1	Test statistic (df)	P-value	Parameter estimate1	Test statistic (df)	P-value
Type of intervention	Medication	−.757	F(2,21) = 1.22	.315	−.305	F(2,21) = 2.38	.117
	Psychosocial	.000			.000
	Combination	−.905			−.033
Type of therapy	Individual	.000	F(2,21) = .63	.544	.000	F(2,21) = 3.13	.065
	Group	−.509			.260
	Combination	−.664			−.130
Number of CRF pages at baseline2	22–131	−.001	t(22) = −.13	.899	.002	t(22) = .94	.355
Number of treatment sessions	1–24	−.066	t(22) = −2.01	.057	−.020	t(22) = −3.49	.002
Duration of treatment	1 day–24 weeks3	−.064	t(22) = −1.49	.152	−.012	t(22) = −1.39	.178
Number of follow-up visits	1–8	−.129	t(22) = −.68	.504	−.040	t(22) = −1.05	.304
Duration of trial (weeks)	4–54	−.011	t(22) = −.47	.644	−.008	t(22) = −1.79	.087
Total number of CRF pages for trial2	67–917	−.002	t(22) = −1.57	.131	−.0003	t(22) = −1.07	.296

Notes:

¹Value of additional effect (for categorical variables in ANOVA) or of slope (for continuous variables in regression). For categorical variables (ANOVA), the statistical software used (SAS^®) sets the parameter value at 0 for the last category, after ordering the categories alphabetically.

²The amount of information collected on one case report form (CRF) page varies across instruments and trials.

³In one trial, booster treatment sessions occurred up to 52 weeks post-randomization.

Results of the ANOVA and regression models show that all factors had a corresponding p-value greater than.05 (not statistically significant), except for the number of treatment sessions on the ratio of actual to planned recruitment rate (p-value = .002) (Table 3). In this case, the negative slope (−.020) indicates a decrease in the ratio of actual to planned recruitment rate as the number of treatment sessions increases. However, because multiple statistical tests were performed, this statistically significant result should be interpreted with caution.

Quadratic terms (independent variables squared) were also tested in all regression models, and none was found to be statistically significant.

RESULTS ON RETENTION

Table 4 shows the values of the variables used to represent retention (expressed in percent): (1) availability of the primary outcome measure(s); (2) treatment exposure; and (3) attendance at follow-up visits.

TABLE 4.

Retention based on three criteria across time points.

Study	Availability of the primary outcome measure(s) (%)	Treatment exposure (%)	Attendance at follow-up visits (%)
CTN0001-BUP1 inpatient	40	74	69
CTN0002-BUP2 outpatient	59	73	61
CTN0003-BUP3 taper	72	79	47
CTN0004-MET	53	69	68
CTN0005-MI	64	89	76
CTN0006-MIEDAR drug-free	60	88	69
CTN0007-MIEDAR methadone	77	95	83
CTN0009-smoking	81	82	79
CTN0010-BUP adolescent	62	88	61
CTN0011-TELE	71	56	72
CTN0013-MET pregnant	98	71	77
CTN0014-BSFT	76	59	73
CTN0015-seeking safety	611	54	61
CTN0017-HIV	71	77	64
CTN0018-safe sex for men	78	50	71
CTN0019-safe sex for women	75	45	67
CTN0020-job seekers	86	59	84
CTN0021-Spanish MET	88	75	82
CTN0027-START	67	64	47
CTN0028-ADHD adolescent	862	80	71
CTN0029-ADHD adult	87	91	82
CTN0030-POATS phase II	92	76	60
CTN0031-STAGE12	80	67	70
CTN0032-HIV rapid testing	983	100	96

Notes: PTSD, Post-Traumatic Stress Disorder; ADHD, Attention Deficit Hyperactivity Disorder.

¹For CTN0015, there were two co-primary endpoints: drug use (61%) and PTSD severity (61%). The analysis used drug use only.

²For CTN0028, there were two co-primary endpoints: drug use (86%) and ADHD (76%). The analysis used drug use only.

³For CTN0032, there were two co-primary endpoints: receipt of HIV test result at 1 month (98%) and risky sexual behaviors at 6 months (89%). The analysis used receipt of HIV test result only.

Many interventions required multiple assessments to calculate a primary outcome, for example, drug use assessed every week over a 6-week period. The second column in Table 4 provides the availability of the primary outcome measure(s) across all assessments on which the primary outcome measure was based. Across the 24 trials, this retention measure ranged from 40% to 98%. The third column in Table 4 indicates treatment exposure across all treatment sessions. It ranged from 45% to 100%. The fourth column in Table 4 shows attendance at follow-up visits across all follow-up visits. It ranged from 47% to 96%.

Through simple single-factor ANOVA, we tested for any pattern between the type of intervention and type of therapy, on the one hand, and the overall percent of available primary outcome assessments, the overall percent of treatment sessions attended, and the overall percent of follow-up visits attended, on the other hand (top two rows in Table 5). Two of these analyses yielded p-values less than .05: (1) the type of therapy on treatment exposure (p-value = .002), indicating that group therapy yielded the lowest attendance; and (2) the type of intervention on the attendance at follow-up visits (p-value = .010), indicating that follow-up visits were attended more often in trials with psychosocial interventions. This latter result was driven mostly by two medication trials (CTN0003 and CTN0027) with particularly low attendance at follow-up visits.

TABLE 5.

Retention analyses (n = 24).

Trial design characteristic (independent variable)	Range	Availability of the primary outcome measure(s)			Treatment exposure			Attendance at follow-up visits
		Parameter estimate1	Test statistic (df)	P-value	Parameter estimate1	Test statistic (df)	P-value	Parameter estimate1	Test statistic (df)	P-value
Type of intervention	Medication	−.161	F(2,21) = 3.36	.054	.023	F(2,21) = 1.63	.220	−.182	F(2,21) = 5.82	.010
	Psychosocial	.000			.000			.000
	Combination	.059			.132			−.037
Type of therapy	Individual	.000	F(2,21) = .05	.952	.000	F(2,21) = 8.78	.002	.000	F(2,21) = .14	.874
	Group	.005			−.247			.016
	Combination	−.021			.031			.029
Number of primary assessment sessions	1–12	−.010	t(22) = −.99	.331
Time of last primary assessment (days post-randomization)	11–365	−.0003	t(22) = −.84	.412
Number of treatment sessions	1–24				−.0001	t(22) = −.02	.987
Duration of treatment	1 day – 24 weeks2				.0000	t(22) = .01	.992
Number of CRF pages during treatment3	0–702				.0002	t(22) = 1.19	.248
Number of follow-up visits	1–8							−.005	t(22) = −.30	.766
Time of last follow-up visit (days post-randomization)	84–407							−.0003	t(22) = −1.16	.257
Number of CRF pages during follow-up visits3	10–203							.0003	t(22) = .65	.524

Notes:

¹Value of additional effect (for categorical variables in ANOVA) or of slope (for continuous variables in regression). For categorical variables (ANOVA), SAS sets the parameter value at 0 for the last category, after ordering the categories alphabetically.

²In one trial, booster treatment sessions occurred up to 52 weeks post-randomization.

³The amount of information collected on one case report form (CRF) page varies across instruments and trials.

The following analyses were also performed: the percent of participants who provided the last primary outcome assessment was modeled separately against the number of primary assessments expected and the time (days post-randomization) of the last planned primary outcome assessment. Similarly, the percent of participants who attended the last treatment session was modeled separately against the number of planned treatment sessions, the time (days post-randomization) of the last planned treatment session, and the total number of CRF pages expected to be completed with participants during all treatment sessions. Finally, the percent of participants who attended the last follow-up visit was modeled separately against the number of planned follow-up visits, the time (days post-randomization) of the last planned follow-up visit, and the total number of CRF pages expected to be completed with participants during all follow-up visits.

None of these models showed a statistically significant trend (Table 5). All p-values were greater than .2. Quadratic terms (independent variables squared) were also tested in all regression models, and none was found to be statistically significant.

DISCUSSION

This analysis is an important step for the CTN to critically evaluate the design models chosen for its trials. An examination of the recruitment and retention rates of CTN trials shows that some trials achieved very high rates. But what are the factors that contribute to this success?

When potential participants are approached to join a clinical trial, the investigators provide a description of what the study entails during the informed consent process, including the duration of the trial, the type of intervention, and the number of treatment sessions expected. These trial design characteristics may influence potential participants to enter the trial or not, and in turn could affect the recruitment rate. As time passes, and the participant experiences the many requirements of the trial, she/he may be influenced on whether to return for all the treatment sessions or every follow-up visit.

Common sense suggests that the heavier the burden on participants, the lower the recruitment rate and retention of participants. But our analysis indicates that the relationship between assessment burden on participants and length of trial, on the one hand, and recruitment and retention, on the other, is not as strong and direct as we had expected. It may be that these trial design characteristics do influence recruitment and retention, but that their influence is subtle, intertwined with, or obscured by the influence of other study characteristics.

The rate at which a trial enrolls participants more likely depends on many factors: the target population (e.g., adolescents, pregnant women), the inclusion/exclusion criteria, the popularity of the treatments offered, the location of the participating sites (rural vs. urban, distance traveled to site), the size of the participating sites (number of patients regularly seen), and many others. Likewise, retention may be influenced by these factors, as well as by the empathy of the counselors, the severity of the participants’ addiction, the primary drug of abuse, involvement in the criminal justice system, and other, sometimes unpredictable, life circumstances. For example, Magruder et al. (9) conducted a secondary analysis of several CTN trials and reported that retention rates for opiate users were higher than those for polydrug users. Incentives offered for participation, such as money paid to participants to come to the clinic for assessment, may also affect attendance. For example, in the Buprenorphine for Adolescents trial (CTN0010), the data show that clinic visits, during which participants were paid more, were attended more frequently (10).

There are other possible explanations as to why the factors considered in these analyses showed no association with recruitment and retention:

1. The sample size is small. Observations from 24 trials may have been too few for any of the analyses to have enough power to detect a pattern that may exist. Additional analyses as more trials are completed could be informative.

2. All trials within the CTN are complex, multisite, effectiveness/efficacy trials with multiple assessments, endpoints, and secondary outcomes. The range of complexity in the trials considered here may be small compared to the spectrum of all clinical trials. Although our analyses showed no pattern within this cohort, the same factors could have shown associations with recruitment and retention if a larger range of trials – including more simple or more complex designs – were analyzed. For example, a study on 10,038 phase 1–4 protocols conducted between 1999 and 2005 found that recruitment and retention rates decreased as the frequency of procedures per protocol and work burden on sites increased (11).

3. The relative effect of trial design is small. The ratio of signal (influence of the considered trial design factors on recruitment and retention) to noise (influence of other trial factors not considered here, as well as general variability among trials and among participants that are unrelated to trial design) is too small to detect.

Our analysis also indicates that the extent to which CTN trials struggle to meet their planned recruitment rate is consistent with the 80% figure previously reported (6). However, this comparison has limitations, because most authors do not report the number of subjects recruited per week.

Regarding retention, most studies only report the number of participants that completed the treatment phase, as opposed to the percent of treatment sessions attended (used in this analysis). For example, Bisaga et al. (12) reported that 49% of participants completed treatment in a single-site medication study for cocaine dependence conducted in an outpatient clinic, whereas Fals-Stewart and Lam (13) reported that 92% of participants completed treatment in a psychosocial study conducted in a long-term residential program, and Heinzerling et al. (14) reported that 38% of participants completed treatment in a medication study for methamphetamine abuse conducted in two clinical research sites.

This article provides in one place recruitment and retention numbers on 24 multisite clinical trials on substance abuse treatment in community treatment programs, all conducted within the CTN and all with a common definition of recruitment and retention. It provides to investigators planning similar trials a rough idea of what to expect in terms of recruitment rates (randomizations per site per week) and retention (availability of primary outcome measure, treatment exposure, and attendance at follow-up visits). It may help sponsors and monitoring board members (of similar trials) gauge whether the trial they are monitoring is in line with past recruitment and retention experience.

The main limitation of this article is that it represents a post hoc analysis, in the sense that when the 24 trials were designed and conducted, there were no a priori goals to evaluate the factors that affect recruitment and retention. For example, there is no information in the trials’ databases on why some declined to join the trial, why some dropped out, or how the length of the assessments influenced their decision to drop out. A study that directly seeks from participants their reasons for dropping out of the trial or for missing visits is a more valid approach to better understand the factors that affect recruitment and retention. Several CTN trials have estimated the time required to administer each assessment. This measure could have served as a better predictor of participant burden than the number of CRF pages. However, this information was not available for all trials, and therefore could not be used here.

This analysis only considered trial design characteristics that are easily quantified or classified; but other qualitative, hard-to-quantify, factors may explain variability in recruitment and retention. For example, matching the protocol to the usual kinds of clinical operations, knowledge base, attitudes, and skills of the participating sites may play an important role in recruitment and retention. Maintaining staff morale and team spirit could also impact recruitment and retention, especially in studies that take place over several years, involve many staff, and have unanticipated challenges. Future research on these and other qualitative design factors would be worthwhile.

As a final note, it is important to put recruitment and retention in perspective. Although critical, they do not in and of themselves make a clinical trial “successful.” The ultimate goal of course is to design clinical trials that produce clinically meaningful findings, which will have an impact on improving addiction treatment. A clinical trial with great recruitment and retention, but fundamentally flawed design, is unlikely to provide valid or useful results.