The potential impact of recruitment method on sample characteristics and treatment outcomes in a psychosocial trial for women with co-occurring substance use disorder and PTSD

Abstract

Background

Recruitment method can impact the sample composition of a clinical trial and, thus, the generalizability of the results, but the importance of recruitment method in substance use disorder trials has received little attention. The present paper sought to address this research gap by evaluating the association between recruitment method and sample characteristics and treatment outcomes in a substance use disorder trial.

Method

In a multi-site trial evaluating Seeking Safety (SS), relative to Women’s Health Education (WHE), for women with co-occurring PTSD (either sub-threshold or full PTSD) and substance use disorders, one site assessed the method by which each participant was recruited. Data from this site (n=106), which recruited participants from newspaper advertising and clinic intakes, were analyzed.

Results

Participants recruited through advertising, relative to those from the clinic, had significantly higher levels of baseline drug use and higher rates of meeting DSM-IV-TR criteria for full PTSD. Results suggest that the effectiveness of SS in decreasing PTSD symptoms was greater for participants recruited through advertising relative to those recruited from the clinic. Conversely, the results revealed a significant treatment effect in the clinic-recruited participants, not seen in the advertising-recruited participants, with SS, relative to WHE, participants being more likely to report past week drug use during the follow-up phase.

Conclusion

Recruitment method may impact sample composition and treatment effects. Replication of this finding would have important implications for substance use disorder efficacy trials which often utilize advertising to recruit participants.

1.0 Introduction

Multiple differences between efficacy and effectiveness research, which impede translation of research into practice, have been delineated including differences in study sample composition (Glasgow et al., 2003). In the substance abuse field, efficacy trials are often conducted in research settings that rely heavily on advertising to recruit participants. Effectiveness trials also use advertising to recruit difficult samples such as in research with co-occurring disorders. The association between recruitment method and sample characteristics has been evaluated in clinical trials of depressed elderly patients (Schlernitzauer et al., 1998; Stack et al., 1995), smokers (Harris et al., 2003; Hoving et al., 2007), and Alzheimer’s disease patients (Andersen et al., 2010) with all but one study (Schlernitzauer et al., 1998) finding significant differences in sample characteristics as a function of recruitment method. The association between recruitment method and treatment outcomes has been less well studied. Two studies evaluated the association between recruitment method and study outcomes in depressed elderly patients with both finding no significant difference in outcomes (Schlernitzauer et al., 1998; Stack et al., 1995).

The degree to which participants recruited through advertising are representative of patients seeking treatment at substance abuse treatment centers, has not, to our knowledge, been evaluated and, thus, the extent to which research findings from trials utilizing advertising for recruitment are generalizable to practice is unclear. It is particularly important to understand the potential impact on external validity in effectiveness trials since this is one of their primary goals (Brigham et al., 2009). A recent National Institute on Drug Abuse (NIDA) Clinical Trials Network (CTN) study evaluated the effectiveness of Seeking Safety (SS), relative to a Women’s Health Education (WHE) control group, for women with co-occurring PTSD and substance use disorders (Hien et al., 2009). One of the participating sites recruited from clinic intakes and newspaper advertisements. The present analyses evaluated the degree to which recruitment method was associated with sample characteristics and treatment outcomes. Finding that different recruitment methods lead to different sample characteristics but to no difference in treatment outcome would support the use of different recruitment methods in that the study samples could be recruited more quickly and would be more diverse, resulting in quicker trial completion with potentially more generalizable findings. A finding that recruitment method is associated with treatment outcomes might suggest the need to stratify on recruitment method or on the sample characteristics leading to differential treatment outcomes or, possibly, that participants should not be recruited through advertising due to a lack of generalizablity to patients seeking treatment, which is the population of ultimate interest.

2.0 Method

2.1 Participants

Participants were women ages 18–65 who reported drug and/or alcohol use within the prior 6 months and who met DSM-IV-TR (APA, 2000) criteria for either full or sub-threshold PTSD and drug or alcohol abuse or dependence. Exclusion criteria included serious medical disease, impaired cognition, psychosis, and significant suicidal/homicidal risk; a full description of study eligibility criteria can be found in Hien et al. (2009). The participants for the present analysis were the 106 participants randomized at the Maryhaven site in Columbus, Ohio.

2.2 Measures

As reported by Hien et al. (2009), the primary outcome measures for the trial included two PTSD measures: the PTSD Symptom Scale-Self Report (PSS-SR; Foa et al., 1993), and the Clinician Administered PTSD Scale (CAPS; Blake et al., 1995). Self-reported substance use was assessed using the Timeline Follow-Back procedure (Fals-Stewart et al., 2000; Sobell and Sobell, 1992). Substance use was transformed into a dichotomous variable of no days of substance use versus at least one day of substance use in the past seven days. Self-reported abstinence was confirmed with qualitative urine drug screens and alcohol saliva tests. Research visits were completed at screening/baseline, weekly during the treatment phase, and at one week, three, six, and twelve months following the end of the treatment phase. The CAPS was completed at baseline and during study follow-up while the other measures were collected at each research visit. In addition, the Maryhaven site collected information on whether participants were recruited through advertisements (AD) or the clinic.

2.3 Procedures

See Hien et al. (2009) for a full description of study procedures. Briefly, women who met full eligibility criteria were randomized to SS or WHE. Participants received two SS or WHE sessions per week for approximately six weeks. The 12 SS/WHE sessions were added to a range of other therapeutic activities offered at the treatment site.

2.4 Data analysis

Stata SE (version 11.1, College Station, TX) was used to conduct all of the analyses. Baseline characteristics by recruitment method were compared using chi-square or Fisher’s exact test for categorical variables and t-tests for continuous variables. Changes of the primary outcomes were compared using generalized estimating equations (GEE) including the main effect for treatment and the baseline value of the outcome variable, as well as the interaction between treatment and time. The first set of models tested whether there was a treatment effect at the site, irrespective of recruitment source. A second set of models were run for each recruitment group (AD or Clinic) separately in order to generate coefficients and odds ratios that were directly interpretable as the effect of SS compared to WHE. Piecewise linear splines were used to test whether there was a treatment effect either during the active phase of treatment (Weeks 0–6) or during the follow-up phase (Weeks 7–54) for the outcome variables collected at every study visit. The CAPS was only collected at baseline and follow-up, therefore a linear spline was not used. The covariates for the analyses included: age, education level, race/ethnicity (White, Black and Other) and treatment dose (number of sessions attended; 0–12). Time was modeled as the assigned study week, rather than the actual week. Robust variance estimates were used as a more conservative approach in the event that the covariance structure was not properly specified (Fitzmaurice et al., 2004). The self-report substance use and biologically-verified abstinence outcome variables are dichotomous (no/yes) and the results of these multivariable models are presented as odds ratios.

3.0 Results

3.1 Sample Characteristics

Over half (66%, n=70) of the sample was recruited via advertisements and 34% (n=36) were recruited from the clinic. The AD and clinic groups were similar in respect to sociodemographic characteristics (see Table 1); of note, the AD group was significantly older, had significantly higher levels of drug use and CAPS scores at baseline, was significantly less likely to meet DSM-IV-TR criteria for cocaine SUD, and was significantly more likely to meet full DSM-IV-TR criteria for PTSD compared to the clinic group. Forty percent (n=28) of participants in the AD group completed the treatment phase of the study, compared to 41.7% (n=15) in the CTP group (p=0.87). Recruitment source was not associated with differential rates of treatment dose (p=0.48) or completion.

Table 1.

Baseline participant characteristics by recruitment source

	Advertising(N=70)	Clinic(N=36)	Advertising vs. Clinic
			p value
Age	44.9 (6.6)	37.0 (8.2)	<0.001
Race/Ethnicity %			0.65
African American/Black	22.9	30.6
Caucasian	55.7	52.8
Hispanic/Multi-racial/Other	21.4	16.7
Marital Status %			0.06
Never Married	22.9	44.4
Married	17.1	16.7
Divorced/Separated/Widowed	60.0	38.9
Education	13.4 (3.1)	12.6 (2.5)	0.18
Employment %			0.71
Employed	35.7	27.8
Unemployed	57.1	66.7
Student/Retired/Disabled	7.1	5.6
Controlled Environment in Past 30 Days%	8.6	22.2	0.05
Number of Days Used Drugs Past Week	3.9 (2.7)	0.8 (2.0)	<0.001
Abstinent Week Prior to Study Entry %	12.9	61.1	<0.001
Prior AOD Treatment Episodes	6.4 (12.4)	5.7 (6.1)	0.73
SUD Diagnoses %
Cocaine	51.4	77.8	0.009
Stimulants	5.7	8.3	0.69
Opiates	24.3	19.4	0.57
Marijuana	14.3	30.6	0.05
Alcohol	60.0	55.6	0.66
PTSD Diagnosis % meeting full criteria	90.0	66.7	0.003
Total CAPs Severity Score	68.0 (16.9)	58.2 (21.3)	0.01
PSSR-SR Score	50.9 (15.8)	46.4 (18.6)	0.19
Lifetime Traumatic Experiences %
Child Physical Abuse	55.7	61.1	0.59
Adult Physical Abuse	88.6	94.4	0.49
Child Sexual Abuse	65.2	83.3	0.05
Adult Sexual Abuse	78.6	83.3	0.56
Transportation Accident	82.9	88.9	0.57
Life Threatening Illness	45.7	66.7	0.04
Exposed to Violent Death	48.6	36.1	0.22

Note. Where not specifically indicated, numbers represent means (standard deviations).

AOD=Alcohol or drug; SUD=Substance use disorders

3.2 Overall Treatment Effects

SS, relative to WHE, was associated with decreased PTSD symptoms (PSSR Active Treatment Coef.=−0.43, p<0.001; PSSR Follow-up Coef.= −0.04, p<0.001; TCAPs Coef. −0.18, p<0.001), independent of recruitment source. The SS intervention was not associated with decreased rates of weekly drug use (Active treatment OR=1.02, 95% CI: 0.99–1.05, p=0.26; Follow-up OR=1.00, 95% CI: 1.00–1.01, p=0.05) or abstinence (Active treatment OR=0.98, 95% CI: 0.94–1.01, p=0.18; Follow-up OR: 1.00, 95% CI: 0.99–1.00, p=0.53).

3.3 PTSD Outcomes by Recruitment Source

As can be seen in Table 2, the treatment effect sizes for PTSD symptom reduction as measured by both the PSS-SR and CAPS scores were greater for participants recruited by advertisement relative to participants recruited from the clinic, which reflected a greater decrease in symptoms in the SS, relative to WHE, group.

Table 2.

Multivariable analysis of PTSD and drug use outcomes as a function of recruitment method

a. PTSD VARIABLESa
	PSS-SR Score						CAPS Scoreb
	Advertising-recruited (n=70)			Clinic-recruited (n=36)			Advertising-recruited (n=70)			Clinic-recruited (n=36)
	Coef. (SE)	p	d	Coef. (SE)	p	d	Coef. (SE)	p	d	Coef. (SE)	p	d
Treatment	5.78 (0.2.27)	0.01		1.12 (4.36)	0.80		1.37 (4.20)	0.75		7.08 (7.57)	0.35
Treatment*Active Phase	−0.86 (0.22)	<0.001	0.96	−0.76 (0.37)	0.04	0.74	N/A			N/A
Treatment *Follow-up	−0.08 (0.03)	<0.001	0.72	−0.01 (0.05)	0.90	0.04	−0.36 (0.04)	<0.001	2.20	−0.27 (0.06)	<0.001	1.61
b. DRUG USE VARIABLESa
	Past Week Drug Use						Past Week Abstinence
	Advertising-recruited (n=70)			Clinic-recruited (n=36)			Advertising-recruited (n=70)			Clinic-recruited (n=36)
	OR (CI)	p	d	OR (CI)	p	d	OR (CI)	p	d	OR (CI)	p	d
Treatment	1.04 (0.49, 2.18)	0.93		4.48 (1.02, 19.7)	0.05		0.99 (0.45, 2.22)	0.99		0.53 (0.18, 1.60)	0.26
Treatment*Active Phase	1.03 (0.97, 1.09)	0.37	0.22	0.99 (0.86, 1.14)	0.93	0.03	0.95 (0.88, 1.03)	0.25	0.28	1.04 (0.91, 1.18)	0.60	0.19
Treatment *Follow-up	1.01 (0.99, 1.02)	0.10	0.40	1.03 (1.01, 1.06)	0.01	0.91	1.00 (0.98, 1.01)	0.67	0.11	0.99 (0.97, 1.01)	0.35	0.34

Note. Coef.=Coefficient; SE=Standard error; OR=Odds ratios; CI= 95% Confidence intervals; d=Cohen’s effect size

^aThe multivariable models included the following covariates: baseline value of the outcome variable, race/ethnicity, age, education and treatment dose.

^bThe CAPS assessment was only collected at screening and each follow-up assessment

3.4 Drug Use Outcomes by Recruitment Source

As can be seen in Table 2, during the active treatment phase, there were no treatment group differences in either past week drug use or abstinence for either the advertising or clinic-recruited participants. During the follow-up phase, there was a significant treatment effect for the clinic-recruited participants for past week drug use (see Table 2) which reflected greater odds of past week drug use by the SS, relative to WHE, participants.

3.5 Sensitivity Analysis

With the exception of PTSD diagnosis, the other statistically significant differences at baseline were controlled for by the pre-specified models. To evaluate the association between PTSD diagnosis and PTSD and drug use outcomes, this variable was added to the models as an interaction term; this term was not statistically significant for any of the outcomes (data not shown).

4.0 Discussion

This study found baseline and treatment effectiveness differences based on how participants were recruited into the trial. Participants recruited from advertisements had significantly greater baseline drug use days and PTSD severity compared to participants recruited from the clinic. For both recruitment strategies, SS was associated with a reduction in PTSD symptoms. However, the effectiveness of SS in reducing PTSD symptoms may be greater for participants recruited from advertisements as indicated by the treatment effect sizes for both the active treatment and follow-up phases.

For drug use, there was no significant effect for SS, relative to WHE, for the sample as a whole. The evaluation of the association between recruitment method and substance use outcomes revealed no significant differences during the active treatment phase. However, during the follow-up phase, there was a significant treatment effect in the clinic-recruited participants, not seen in the advertising-recruited participants, with SS, relative to WHE, participants being more likely to report past week drug use.

Overall, SS was effective in reducing PTSD symptoms at the Maryhaven site irrespective of recruitment source. The main study did not find a treatment effect and there are several potential explanations for the difference. First, this study used data from a single site whereas the main study included data from seven sites and it is possible that SS was more effective at this particular site. Second, variation in the statistical approach may explain some of the difference. These studies used slightly different methods to test for a treatment effect during the active and follow-up phases of the study. The statistical model for the main study included a site effect and tested for several interaction effects using stepwise elimination; neither approach was used in this secondary analysis.

The present results raise methodological issues and it may be important to understand why recruitment sources yield clinically different study samples. The study participants recruited from advertisements were not in treatment at the time of recruitment which could explain their greater clinical severity at baseline compared to participants recruited from the clinic. In this study, 61% of the participants recruited from the clinic were already abstinent at the onset of the trial. In other words, only 39% of the participants (n=14) could potentially achieve abstinence during the clinical trial. While randomization should ensure equitable distribution between treatment arms, the high rate of drug abstinence at baseline may inadvertently under-power the study to detect a substance use reduction. Additional research is needed to better understand the reasons for these clinical differences, in order to understand how this may impact the design and outcomes of clinical trials. The present results should be considered with the limitation that the data were collected in a clinical trial not specifically designed to evaluate the importance of recruitment source. In addition, post-hoc analyses suggested that this study was adequately powered to detect moderate to large effects but was under-powered to detect small effects. Finally, the extent to which the clinical differences based on recruitment source found in this single-site study are generalizable to other clinical sites or studies is unknown.

In conclusion, different recruitment sources may produce clinically different samples and may result in treatment effect differences. Additional research is needed to understand the extent to which these findings are representative of drug treatment clinical trials and whether recruitment source needs to be taken into consideration during the design and analysis of trials. If replicated, this finding could have important implications for treatment development and testing in that it would suggest that the typical reliance on advertising-recruited participants in earlier stage testing might not be an effective approach given the potential lack of generalizability to treatment-seeking patients, which is the population of ultimate interest.