Recruitment Methods, Inclusion, and Successful Participation in a Longitudinal Clinical Trial Using Ecological Momentary Assessment

Abstract

Underrepresentation of historically marginalized populations in clinical trials continues to threaten the validity of health intervention research. Evidence supports the merits of intercept and other proactive forms of recruitment for achieving more equitable representation. However, researchers also report lower retention and adherence to protocols among these populations, particularly in longitudinal studies. Few studies have compared recruitment methods for longitudinal randomized trials testing health interventions, with even fewer having done so for trials involving ecological momentary assessment (EMA). As intervention research integrates EMA and other data collection approaches requiring substantial participant effort, it is critical to better understand the effectiveness and implications of strategies to improve the representativeness of health research. This secondary data analysis compared outcomes of proactive and reactive recruitment strategies (mobile lab intercepts and internet/flyer advertising, respectively) in study inclusion, task completion, and retention within a 14-day randomized controlled trial that used EMA to evaluate cigarette package health messages. Proactive recruitment resulted in higher proportions of participants with low income and education, limited health literacy, and of diverse racial/ethnic makeup. However, this recruitment method also resulted in lower task completion, especially in the second week of the trial period, and lower retention, although group differences were not explained by participant sociodemographic characteristics targeted by inclusion efforts. We conclude that proactive recruitment via intercepts is an effective strategy for health intervention research that aims to include stakeholders from historically marginalized groups but that researchers and funders must recognize these methods require additional resources, considerations, and capacity to address non-trivial challenges to successful participation.

Introduction

Cigarette smoking remains the leading cause of preventable disease in the United States, with individuals of low socioeconomic status (SES) and Black, Indigenous, and People of Color (BIPOC) communities bearing a disproportionate amount of the burden. BIPOC and low SES individuals are overrepresented among U.S. smokers (Cornelius et al., 2020), are disproportionately targeted by tobacco marketing (Balbach et al., 2003; Carroll et al., 2020; Mills et al., 2022), and tend to suffer more burden from and are more likely to die from tobacco-related disease (Giovino & Gardiner, 2016; Henley et al., 2016; Hiscock et al., 2012). These disparities are driven by structural racism and other systemic differences in the social, economic, and environmental conditions in which low SES and BIPOC populations live and work (National Academies of Sciences, Engineering, and Medicine [NASEM], 2017; Williams et al., 2008). However, despite these heightened health and social burdens of tobacco, these populations have long been underrepresented or inadequately reported on in research used to inform tobacco policy and interventions (Cwalina et al., 2021) and in clinical trials more broadly (NASEM, 2022).

To address the historical underrepresentation of these populations in health research, the U.S. National Institutes of Health (NIH) developed guidelines on the inclusion of women and racial/ethnic minorities as participants in clinical research (National Institutes of Health, 1994). These guidelines require appropriate inclusion, disallow cost as a reason for exclusion in NIH-funded studies, and point to SES as an additional health-related factor to consider in sampling designs.

A recent comprehensive NASEM (2022) report summarizes many of the factors contributing to underrepresentation. These include hesitancy among researchers to recruit from historically marginalized populations due to concerns about their willingness and capacity to participate or the additional time and resources it may be required, site selection limited to less diverse areas (e.g., in or near academic institutions), and studies developed without consideration of the life challenges faced by historically marginalized populations (e.g., overly restrictive inclusion/exclusion criteria, burdensome interventions) or without incorporating their values (e.g., research questions and interventions lacking cultural relevance). Additional factors include distrust among BIPOC populations given a legacy of abuses in clinical research and perceptions that studies will benefit white populations or research institutions more than the underrepresented individuals who enroll. Limited health literacy may also reduce interest in participation (Kripalani et al., 2019).

Proactive recruitment strategies such as in-person intercepts at locations frequented by BIPOC and low SES individuals or partnerships with organizations serving historically marginalized populations tend to require more time to reach sampling goals and incur higher costs than reactive recruitment methods where targeted individuals must respond to advertisements, but they also tend to be more inclusive (Graham et al., 2011; Greiner Safi et al., 2019; Harris et al., 2003).

Researchers report, however, that participants from these groups are less likely to adhere to protocols or complete studies (Blumenthal et al., 1995; Bonevski et al., 2014; Schüz et al., 2014). These reports point to a variety of issues associated with BIPOC identities or low SES—including those that stem from structural, historical racism—such as lack of access to transportation and child care, inflexible work schedules, and other competing demands of daily living, whose impacts may be stronger for studies with relatively burdensome protocols. Lower SES participants also are more likely to have fluid housing and communication resources (e.g., consistent active phone numbers) making it difficult to maintain contact over lengthy study periods. Readiness for behavior change (e.g., tobacco quit intentions) may also play a role in retention, although findings are inconsistent (cf. (Prochaska et al., 2001; Woods et al., 2002),

Few studies have directly compared both recruitment and retention rates between recruitment methods administered concurrently within a longitudinal randomized controlled trial (RCT; see Bonevski et al., 2014, for a review). Even fewer have done so with RCTs involving ecological momentary assessment (EMA; Frandsen et al., 2014, 2016). Given the greater participant effort required by EMA, it offers an important test of the relative merits of two recruitment strategies regarding inclusion, protocol adherence, and retention amid considerable participant burden.

This study compared rates of inclusion, daily task, and study completion using concurrent proactive and reactive recruitment strategies within a 14-day EMA RCT on cigarette package labels with contrasting health messages. This study also took advantage of a natural experiment introduced by COVID-19-related restrictions on research protocols which required the intercept recruitment team to pivot to internet-based recruitment roughly halfway through data collection. This enabled post hoc comparisons between methods during two meaningfully different periods—pre-COVID-19, where recruitment methods differed substantially (intercept vs. internet), and post-COVID-19-onset where recruitment methods were similar (both internet).

Based on existing literature, we developed several exploratory hypotheses (Hs). First, we predicted that compared with reactive recruitment via internet ads and flyers, proactive recruitment via intercepts would be more inclusive of historically underrepresented populations (i.e., higher proportions of low-education, low-income, and BIPOC participants, relative to city-level census estimates; H1) and of participants with lower health literacy (H2). We also hypothesized that intercept recruitment would result in lower study (H3) and daily task (H4) completion rates than reactive recruitment. We further examined whether differences in sociodemographic and smoking characteristics explained group differences in study (RQ1) and task completion rates (RQ2) and explored the role of time in task completion attrition (RQ3).

Method

Overview

Data were gathered as part of a 14-day RCT of cigarette package warning labels and informational inserts. We recruited adult cigarette smokers from areas of central New York (NY) State and central South Carolina (SC) June 2019 to June 2021 with a substantial pause during the initial outbreak of the COVID-19 pandemic (NY: 12/2019-12/2020; SC: 3/2020-8/2020; Table 1). Research teams at each site recruited individuals via either intercepts at smoke shops in six urban and rural areas (35% of the sample) or through online advertisements and flyers (65%). Eligible recruits were at least 21 years old, smoked at least 10 cigarettes per day, and had smoked at least 100 cigarettes in their life (N=397, M_age=44.0, SD=12.3).

Table 1.

Recruitment and Orientation Characteristics.

Recruitment		Orientation
Group	Strategy	Immediacy	Locale	Materials delivery
NY1: Pre-COVID	Proactive	Immediate	Intercept site	At immediate orientation
SC1: Pre-COVID	Reactive	Scheduled	Office visit	At scheduled orientation
NY2: Post-Onset	Reactive	Scheduled	Online	Separate scheduled delivery
SC2: Post-Onset	Reactive	Scheduled	Online	Separate scheduled delivery

Participants completed a 45-min orientation and received a 14-day supply of their preferred cigarettes with packs modified to reflect their randomized labeling condition. They also received a mobile phone on which they were asked to log every cigarette they smoked (tap a button, answer one question), complete brief (1–3 min) surveys that appeared after 4 to 5 of each day’s logs (selected semi-randomly to appear throughout each day), and complete a 1- to 3-min evening report at the end of each day. Participation concluded with a 20-min follow-up survey and 30-min interview.

We incorporated recommended practices for inclusive research (e.g., Bonevski et al., 2014; NASEM, 2022) in a variety of ways: designed the study protocols to minimize burden (e.g., survey lengths, required tasks, transportation requirements) and to be appropriate for a sixth-grade level of literacy; piloted the protocols with members of BIPOC and low SES populations; provided a phone number where participants could address questions, challenges, and concerns with staff who were trained to be sensitive and accommodating; monitored task completion and reached out to participants who might be experiencing difficulties; minimized eligibility requirements; provided $150 when participants completed the follow-up survey; and made exceptions for participants who had made a reasonable effort but were prevented from completing the study.

Pre-COVID-19 Recruitment

Proactive Recruitment

The NY team (Proactive-NY1) recruited individuals outside smoke shops in six low-SES areas in a university-branded mobile lab (n=139 recruits, M_age=40.3, SD=11.8). They used city-level census data to identify candidate sites where the median household income was lower than the state median of US$68,000. Sites were selected based on driving distances and the presence of smoke shops willing and capable of hosting the mobile lab (median incomes for the selected sites ranged US$34K–$44K). Research staff posted signs and invited individuals to participate as they entered or left the shops. Eligible recruits immediately completed the orientation and received the study materials.

Reactive Recruitment

The SC team (Reactive-SC1) posted online advertisements (Facebook, Craigslist) and distributed flyers in a major metropolitan area which briefly described the study and pointed individuals to an online eligibility questionnaire (n=89 recruits, M_age=45.5, SD=12.0). They called eligible respondents to provide additional information about the study, confirm eligibility, and schedule a visit to a university office for the orientation and study materials.

COVID-19 Recruitment

After the onset of the COVID-19 pandemic, both sites advertised the study through social media, similar to the strategy used for Reactive-SC1, targeting the same cities as during the pre-COVID-19 period (Reactive-NY2: n=84 recruits, M_age=47.4, SD=11.3; Reactive-SC2: n=85 recruits, M_age=45.2, SD=12.9). They called respondents to confirm eligibility and schedule a visit to a university or other central area near their residence, but only to receive the study materials. During this phase, all orientations were administered virtually through a web conferencing platform and, to meet sampling requirements, recruitment prioritized smokers intending to quit within the next 6 months.

Measures and Analytic Procedures

Study Inclusion (H1 and H2)

These analyses included every case in which the participant received the study materials. We used sociodemographic information from each recruitment site (city-level census data) to calculate expected population proportions for our comparisons of education, income (U.S. Census Bureau, 2019), and race/ethnicity (U.S. Census Bureau Population Estimates Program, 2019). We calculated educational attainment as whether participants aged ≥25 years had completed high school to match U.S. census reporting. We coded participants as “low-income” if they earned less than US$30,000/year. This threshold was approximated but was somewhat lower than the range of census estimates for median household income across sites. This allowed us to compare observed proportions of low-income participants in each recruitment group to the 50% proportion inherent in census median estimates.

For each of the two NY recruitment groups, we weighted observed case counts for each NY recruitment site by census proportions for each respective city and summed them into a single proportion for each group. To test H1, we used chi-square tests to compare the proportions of participants enrolled in each recruitment group with low education, low income, and of each race/ethnicity to census estimates.

We administered the six-item Newest Vital Sign questionnaire to participants during study orientation to assess health literacy (Weiss et al., 2005). To test H2, we constructed an indicator variable for participants with a “high likelihood” or “possibility” (0–3 correct responses) of limited health literacy. We conducted between-group chi-square tests to compare the proportions of participants with limited health literacy enrolled in each recruitment group.

Study Completion (H3 and RQ1)

We calculated study completion for each recruitment group as the proportion of participants who completed the follow-up survey after the 14-day study period. To test H3, we conducted chi-square tests comparing the proportions of participants who completed the survey within each recruitment group. To address RQ1, we analyzed two adjusted models using logistic regression to test whether demographic or smoking characteristics explained study completion rates by recruitment group. The first (Base) model included terms for the randomized experimental condition to account for the potential effects of the RCT intervention on study completion. In the second (Full) model, we added participant sociodemographic and smoking characteristics—intention to quit (International Agency for Research on Cancer, 2008) and nicotine dependence (Heatherton et al., 1991)—to assess whether any observed differences by recruitment method were accounted for by differences in sample composition.

Task Completion (H4, RQ2, and RQ3)

We developed two measures of task completion—evening report and cigarette log completion. For both analyses, we limited the sample to participants who completed the study, at least one evening report, and one cigarette log on or after the first full day of participation. Because participants began the study at different times on the first day, we also limited our analyses to the 13 subsequent full days of participation, referred to hence as days 1 to 13. Results for both measures were substantively equivalent. For simplicity, we present the results for evening reports.

We constructed day- and aggregate-level measures of evening report completion. Day-level completion was a binary outcome (completed or not) for each participant on each day of the study. Reports were complete if participants responded to the final question. Aggregate-level completion was the proportion (P) of possible evening reports (X) completed (x) by each participant (i) in the 13-day period we analyzed: P(X_i) = x_i / 13. To test H4, we conducted a series of t-tests to compare the average rate of evening report completion of each recruitment group, making use of the aggregate-level measure of evening report completion.

To explore RQ2 and RQ3, we analyzed three adjusted models using multi-level logistic regression. The first two models (Base, and Full to address RQ2) matched those of the study completion analysis except for the addition of study time (Days 1–13) to both models, to account for differences in reporting over time. In a third model (addressing RQ3), we added interactions between recruitment group and time and plotted the results of model predictions in a figure.

We used R v.4.1.2 to analyze task completion. We used the lme4 package for multi-level models (Bates et al., 2015), the lm function for linear models (R Core Team, 2018), and ggplot2 for data visualization (Wickham, 2016). We conducted analyses of study inclusion and completion with IBM SPSS Statistics v.27.0 (IBM Corp, 2020).

Results

Inclusion (H1 and H2)

Intercept recruitment (Proactive-NY1) included higher proportions of low-income area residents than census estimates. All other groups are underrepresented low-income residents (Table 2). Proactive-NY1 also included a higher proportion of area residents aged 25+ years who had not completed high school. All other recruitment groups appeared to underrepresent this population by roughly half, although two of the three differences were not statistically significant. Sample proportions for race/ethnicity in Proactive-NY1 were not statistically different from census proportions for area residents identifying only as Black or White, and those of Hispanic/Latina/o/x descent. All three reactive recruitment groups underrepresented residents identifying only as Black and appeared to underrepresent those of Hispanic/Latina/o/x descent, though overall counts were very low. Reactive recruitment groups also overrepresented participants identifying only as White.

Table 2.

Study Inclusion by Recruitment Group: Sample and Census Proportions by Education, Income, and Race/Ethnicity Within Each Recruitment Group.

Participant characteristic	Sample, %	Census, %	Test result (X2)
Education (≥ 25 yrs, < HS diploma)
Proactive-NY1	27.5	13.5	(1, N=131) = 21.93, p < .001
Reactive-SC1	5.8	10.6	(1, N=86) = 2.08, p = .15
Reactive-NY2	4.8	14.7	(1, N=84) = 6.62, p = .01
Reactive-SC2	6.0	10.6	(1, N=84) = 1.92, p = .17
Household Income
Proactive-NY1	74.6 < $30k	50 < $34k-$44k	(1, N=130) = 31.51, p < .001
Reactive-SC1	25.3 < $30k	50 < $47,286	(1, N=87) = 21.25, p < .001
Reactive-NY2	34.1 < $30k	50 < $34k-$44k	(1, N=82) = 8.24, p = .004
Reactive-SC2	32.1 < $30k	50 < $47,286	(1, N=81) = 10.38, p < .001
White (solely)
Proactive-NY1	69.8	73.2	(1, N=139) = 0.83, p = .36
Reactive-SC1	76.4	49.2	(1, N=89) = 26.35, p < .001
Reactive-NY2	92.9	74.2	(1, N=84) = 15.27, p < .001
Reactive-SC2	78.8	49.2	(1, N=85) = 29.84, p < .001
Black (solely)
Proactive-NY1	18.0	13.8	(1, N=139) = 2.05, p = .15
Reactive-SC1	16.9	39.8	(1, N=89) = 19.56, p < .001
Reactive-NY2	2.4	9.7	(1, N=84) = 5.14, p = .02
Reactive-SC2	15.7	39.8	(1, N=85) = 21.31, p < .001
Hispanic/Latina/o/x
Proactive-NY1	6.5 (n = 9)	7.3	(1, N=139) = 0.14, p = .71
Reactive-SC1	1.1 (n = 1)	5.5	(1, N=89) = 3.28, p = .10 a
Reactive-NY2	3.6 (n = 3)	6.4	(1, N=84) = 1.12, p = .29
Reactive-SC2	1.2 (n = 1)	5.5	(1, N=85) = 3.06, p = .09 a

^aExact test of goodness-of-fit is used to account for expected frequencies of fewer than 5. Frequencies are provided for reference.

Intercept recruitment also yielded a significantly higher proportion of participants with limited health literacy (61.2%) than reactive recruitment (13.5%-27.1%; χ² [3, N=397] =75.69, p<.001). Bonferroni-corrected pairwise tests revealed significant differences only between the intercept-based group and each reactive recruitment group.

Study Completion (H3)

H3 was supported. Study completion was lower for Proactive-NY1 (79.1%) than for all reactive recruitment groups (94.1%-98.8%), χ² (3, N=397) =34.94, p <.001. Bonferroni-corrected pairwise tests again detected differences only between the proactive and each reactive recruitment group.

Explaining Rates of Study Completion (RQ1)

Table 3 presents the results of logistic regression models predicting study completion by recruitment group, experimental condition, demographic composition, and smoking characteristics with Proactive-NY1 as the reference group. The Base Model confirmed H3 in showing that the odds of study completion for all reactive groups were higher (3.6 times so for Reactive-SC2, to 19.5 times so for Reactive-NY2) than for Proactive-NY1. The Full Model found that female gender (2.7 times higher odds) and adequate health literacy (8.5 times higher odds) each predicted higher levels of study completion, while other measured demographic and smoking characteristics did not. After adjusting for all these factors, the odds of study completion for Reactive-NY2 were still 11.2 times that of Proactive-NY1, but with Reactive-SC1 (odds ratio [OR] = 4.0) and SC2 (OR=2.1) no longer differing from Proactive-NY1.

Table 3.

Odds Likelihood Models Predicting Study Completion by Recruitment Group, Experimental Condition, Demographic Composition, and Smoking Characteristics.

Predictors	Base Model			Full Model
		95% CI			95% CI
	OR	LL	UL	OR	LL	UL
Recruitment [vs Proactive-NY1] a
Reactive-SC1	10.59**	2.43	46.29	3.96	0.76	20.73
Reactive-SC2	3.59*	1.30	9.87	2.18	0.64	7.45
Reactive-NY2	19.47**	2.58	147.05	11.21*	1.32	95.26
Condition [vs Text-only] a
Graphic Label	1.28	0.43	3.83	1.26	0.39	4.08
Insert	0.70	0.27	1.83	1.03	0.31	3.45
Insert + Graphic Label	1.63	0.55	4.85	0.78	0.27	2.24
Participant Characteristics
Age	-	-	-	1.00	0.97	1.04
Female [vs Not Female] a	-	-	-	2.68*	1.15	6.25
Race [vs Not White-only] a	-	-	-	0.48	0.18	1.31
Education [vs < High Sch.] a	-	-	-	0.77	0.28	2.14
Income [vs Lower Income] a	-	-	-	0.72	0.27	1.97
Health Lit. [vs < Adequate] a	-	-	-	8.45***	2.81	25.45
Smoking Characteristics
Quit Intent [vs No Intent] a	-	-	-	4.63	0.94	22.78
Nicotine Dependence	-	-	-	1.96	0.79	4.86
Constant	4.12***			1.47
Model Fit Statistics
Model χ2	32.76***			64.46***
Nagelkerke R2	.186			.351
N	377			377

Note. The outcome variable is the ratio of the probability for study completion to the probability for non-completion ([n completions / n enrollments] / [n non-completions / n enrollments]).

^aBracketed items were the reference categories in the analyses.

^*p ≤ .05; **p ≤ .01; ***p ≤ .001.

Task Completion (H4)

H4 was supported. Average evening report completion was significantly lower for Proactive-NY1 (81.8%, SD=23.9) than for all reactive recruitment groups (SC1 [87.2%, SD=14.3], NY2 [89.8%, SD=13.4], SC2 [87.2%, SD=15.7], all t-test ps <.05). In contrast, proportions of report completion among the three reactive recruitment groups were not statistically different.

Explaining Rates of Task Completion (RQ2)

Table 4 presents the results of the multivariable models predicting report completion by recruitment group, experimental condition, time, demographic composition, and smoking characteristics again with Proactive-NY1 as the reference. Consistent with H4, the Base Model showed that the odds of report completion for Reactive-NY2 were 1.99 times that of Proactive-NY1, though odds of completion within the other reactive recruitment groups did not differ significantly from those of Proactive-NY1 (ORs: SC1=1.41, SC2=1.50). The Full Model showed that the difference between the Proactive-NY1 and Reactive-NY2 groups attenuated to non-significance (OR=1.62), while time and participant age both affected completion likelihoods. Participants across recruitment groups were less likely to complete evening reports as the study progressed (OR=0.90) and more likely at higher ages (OR=1.02).

Table 4.

Odds Likelihood Models Predicting Evening Report Completion by Recruitment Group, Experimental Condition, Demographic Composition, Smoking Characteristics, and Study Day.

Predictors	Base Model			Full Model			Full Model + Day Interactions
		95% CI			95% CI			95% CI
	OR	LL	UL	OR	LL	UL	OR	LL	UL
Constant	23.78***	14.26	39.64	4.60**	1.43	14.82	6.91**	2.03	23.56
Recruitment [vs Proactive-NY1] a
Reactive-SC1	1.41	0.86	2.31	1.28	0.73	2.24	0.76	0.33	1.76
Reactive-SC2	1.50	0.91	2.48	1.43	0.81	2.51	0.75	0.32	1.75
Reactive-NY2	1.99**	1.20	3.30	1.62	0.91	2.88	0.78	0.33	1.88
Condition [vs Text-only] a
Graphic Label	0.66	0.40	1.10	0.68	0.41	1.12	0.68	0.41	1.12
Insert	0.72	0.43	1.21	0.73	0.44	1.20	0.72	0.43	1.21
Insert + Graphic Label	1.01	0.60	1.68	1.00	0.60	1.66	1.00	0.60	1.6
Study Day	0.90***	0.88	0.92	0.90***	0.88	0.92	0.86***	0.82	0.90
Participant characteristics
Age	-	-	-	1.02*	1.00	1.03	1.02*	1.00	1.03
Female (vs Not Female) a	-	-	-	0.77	0.52	1.12	0.77	0.52	1.12
Race (vs Not White-only) a	-	-	-	1.35	0.84	2.17	1.35	0.84	2.18
Education (vs < High Sch.) a	-	-	-	1.19	0.67	2.13	1.20	0.67	2.15
Income (vs Lower Income) a	-	-	-	0.97	0.63	1.49	0.97	0.63	1.49
Health Lit. (vs < Adequate) a	-	-	-	1.22	0.77	1.92	1.22	0.77	1.93
Smoking Characteristics
Quit Intent (vs No Intent) a	-	-	-	0.98	0.64	1.49	0.98	0.64	1.49
Nicotine Dependence	-	-	-	1.11	0.99	1.24	1.11	0.99	1.24
Interactions (vs Proactive-NY1 × Day) a
Reactive-SC1 × Day	-	-	-	-	-	-	1.06	0.99	1.14
Reactive-SC2 × Day	-	-	-	-	-	-	1.07*	1.00	1.15
Reactive-NY2 × Day	-	-	-	-	-	-	1.08*	1.01	1.17
Model Fit Statistics
Random Effects χ2	3.29			3.29			3.29
Random Effects ICC	0.37			0.35			0.36
N Participants	355			355			355
N Observations	4615			4615			4615
Marginal R2	.047			.068			.067
Conditional R2	.395			.397			.400

Note. The outcome variable is the daily ratio of the probability for evening report completion to the probability for non-completion ([n completions on day i / n possible completions on day i] / [n non-completions on day i / n possible completions on day i]). See Supplemental Table A for raw counts and proportions of completed reports by day and recruitment group.

^aBracketed items were the reference categories in the analyses.

^*p ≤ .05; **p ≤ .01; ***p ≤ .001.

Exploring the Role of TIME (RQ3)

Raw overall proportions of task completion (Supplemental Appendix A) showed that 89.7% of participants completed evening reports on the first day of their study period versus only 78.8% completed reports on the final day. We examined the potential moderating influence of time on the effect of recruitment on report completion by adding terms representing interactions between time and each recruitment group to the full model. Two of the interaction terms were significant, indicating different rates of decline between groups. We calculated model-predicted probabilities of report completion and plotted the relationship between recruitment group, study day, and evening report completion. Figure 1 depicts these relationships and shows that group differences in report completion increased over time. Completion rates were uniformly high for roughly the first 6 days but quickly diverged over Days 7 to 13, with rates declining more rapidly in the Proactive-NY1 group than in the two COVID-period reactive recruitment groups (NY2, SC2).

Figure 1.

Model-Predicted Probability of Evening Report Completion by Recruitment Group and Day.

Note. Predicted probabilities for completing evening reports each day of the study are shown for each recruitment group. Shaded areas represent 95% confidence intervals for the predicted probabilities of the same color.

Discussion

The findings of this secondary data analysis of a single RCT that employed different recruitment strategies suggest that, compared with reactive recruitment where individuals responded to online advertisements and flyers, proactive recruitment via intercept at locations frequented by members of historically marginalized communities was more inclusive of historically underrepresented populations. However, there were some trade-offs in lower task and study completion that were also associated with participant age, gender, health literacy, and the number of days that had elapsed.

Intercept recruitment resulted in higher proportions of individuals with low income and education levels relative to city-level census estimates. In contrast, reactive recruitment resulted in samples that underrepresented low SES individuals and included fewer individuals with low health literacy. Similarly, only the intercept method recruited proportions comparable to census estimates of individuals who identified only as Black or White, or who were of Hispanic/Latina/o/x descent. These results confirm previous assertions that proactive recruitment via intercepts may help answer calls for more inclusive health research (NASEM, 2022).

Past studies have documented lower protocol adherence and retention among participants from historically marginalized populations (e.g., Bonevski et al., 2014). In the current study, proactively recruited participants failed to complete approximately 18% of their evening reports compared with 10% to 13% for reactively recruited participants. However, being younger, male, or of limited health literacy were the only significant sociodemographic predictors for evening report and study completion, not race/ethnicity, income, education, or smoking characteristics. This suggests that while challenges associated with the inclusion of underrepresented populations can be overcome with proactive recruitment, appropriate study development, and administration practices, additional research may be needed to address factors that may impact participation among younger adults, men, and individuals with limited health literacy.

Past studies have also observed declines in task completion over time, particularly in the final days of a study (Broderick et al., 2003; Comulada et al., 2015; Courvoisier et al., 2012). Our findings indicate the magnitude of decline increased more rapidly among the proactive group, where we observed a decline of 20% over 13 days compared to the 6% to 7% decline we observed among the reactive recruitment groups. The moderating influence of time on the impact of the recruitment method on task completion suggests a variety of factors that may shift more abruptly for intercept-recruited participants. Choosing to respond to an internet ad and scheduling and keeping an appointment at a designated site requires greater social and material resources (e.g., fewer competing resources, transportation, internet literacy, and access), trust (e.g., in the scientific enterprise), and motivation to participate (e.g., active seeking of participation opportunities) than proactive recruitment. However, proactive recruitment permits the opportunity to engage with study researchers during a normal life activity (e.g., purchasing cigarettes) and the option to enroll and receive study materials immediately. The ability to ask questions of the researcher during recruitment may increase levels of trust in the researcher and motivate consideration to participate, as might a bandwagon effect from watching other individuals line up to participate. However, these factors may not be sufficient to offset broader structural or age-related barriers to sustained participation over the course of a 14-day longitudinal trial.

Implications

Recruitment methods that reach historically marginalized populations by “meeting them where they are” can be effective in producing more equitable sample compositions. The findings from this study provide additional evidence that more inclusive recruitment can require additional time and resources (e.g., Nagler et al., 2013) to match the completion rates of other methods through a novel investigation that used consistent completion metrics across methods and periods of recruitment during a single randomized controlled trial. In this study, proactive recruitment via intercepts with a mobile lab at locations frequented by underrepresented populations required additional staff, staff time, and the cost of fuel and general upkeep for the mobile lab, which itself required significant separate funding to purchase. The higher rate of attrition among these participants magnified these costs as well as the cost of study materials for each participant and the recruitment trip needed to offset attrition.

Our findings also suggest a need to place a stronger focus on factors related to health literacy, age, gender, and study duration. Proactive recruitment provided better representation for members of historically marginalized groups and membership in these groups did not predict the higher study and task attrition associated with this form of recruitment. This may at least in part be due to additional strategies held constant across the RCT design that were informed by the potential for higher attrition. Recommendations for addressing challenges to inclusion, adherence, and retention include providing more meaningful incentives, flexible protocols, and regular engagement with participants across the study period (Bonevski et al., 2014; El-Khorazaty et al., 2007). We offered US$150, flexible call schedules, appropriately trained staff who were regularly available to address participant needs, and regular monitoring of participation. Research staff also conducted two to three brief check-in calls with participants in the first week of the study. This underscores the ongoing need to more precisely identify and methodologically address barriers to successful sustained participation, particularly when employing face-to-face intercepts, for its value in increasing representation for historically marginalized groups.

Given the disproportionate health burden that low SES and BIPOC populations suffer from tobacco-related and other diseases, they are critical populations for efforts to promote health equity and social justice and to continue answering the call for greater inclusion in research aimed at informing health interventions and policy. Both researchers and funders of this work must be willing to support the additional time and costs associated with proactive recruitment methods and reduce any barriers funding applicants may face when seeking to advance more inclusive practices (e.g., additional requirements for exceeding funding limits). Those evaluating science that centers on perspectives from historically underrepresented populations must also acknowledge these realities in evaluating the scientific, practical, and policy contributions of this work. Additional research is also needed to unpack the relationship between task completion and participant age, including the identification of strategies to remediate this issue. Existing literatures focus on inclusion and participation among those at the two ends of the age distribution (i.e., children/adolescents and the elderly), highlighting the need to focus attention on a broader life course perspective. Similarly, additional research is needed to uncover potential barriers to successful participation for males and individuals with low health literacy. Literature regarding the latter suggests individuals with inadequate health literacy may be less interested in participation (Kripalani et al., 2019). While the fact that the intercept recruitment method required the least effort from prospective recruits to join the RCT, the role of interest beyond immediate material reward remains an open question.

The current study was intensive, requiring participants to log each cigarette they smoked, respond to 4 to 5 brief surveys each day, and complete an additional survey each evening for 2 weeks. The magnitude of differences in task completion rates over time may be smaller for less burdensome studies. Longer studies, however, may face larger challenges to task and study completion given the influence of time we observed. While our study design incorporated mechanisms to improve protocol adherence and limit attrition, including brief check-in calls with participants in the first week of the study, we made no calls in the second week except for a few extreme cases of missing task data. Additional calls during the second week may have improved task and study completion through the final days of the study. Specific to our use of EMA methods, but applicable beyond clinical trials related to tobacco cessation, regular engagement can also mean including more frequent prompts. A meta-analysis of EMA-based studies suggests including at least six daily prompts (Wen et al., 2017) though Hufford (2007) warns of the risk of overburdening participants which may reduce task completion. This adds further emphasis to the need to pilot protocols and integrate more inclusive forms of protocol co-development with target populations.

Limitations

Our assessments of inclusion relied on comparisons to city-level census data. Given that census estimates represent overall area populations, we cannot say whether these proportions reflect area demographics for cigarette smokers.

The data for this study were not collected for the purpose of comparing recruitment methods but as part of an RCT that had clear and systematic variation in recruitment methods made necessary by the pandemic. While this reliance on secondary data tempers the conclusions we might have been able to draw from an RCT designed exclusively for the purpose of comparing recruitment methods, we believe that it still holds significant value alongside the growing body of literature in this area by addressing other limitations found therein, namely, the fact that we hold study domain, protocols, burden, and incentives constant rather than comparing methods between entirely different RCTs. This also points to the need for funders to more readily support RCTs designed specifically to test methods of recruitment, protocol adherence, and retention with the scientific rigor they deserve.

Given that the proactive intercept method was only administered in New York, site differences may confound comparisons between these site-specific methods. However, we detected no difference between sites after they switched to similar reactive methods during the pandemic which offers some safeguard against this possibility.

The onset of the pandemic caused profound changes to our world well beyond our recruitment strategies. The higher task and study completion rates we observed among the post-onset reactive recruitment groups relative to the pre-COVID proactive group may have to do with unmeasured pandemic-related factors. Yet we detected no differences in completion rates between the reactive recruitment groups regardless of when they occurred, suggesting the pandemic’s onset did not have a profound impact in this regard.

Post-onset recruitment prioritized smokers who intended to quit which could have inflated study and task completion rates in those groups (Woods et al., 2002). However, consistent with Prochaska et al. (2001), quit intention did not predict study or task completion suggesting it probably does not account for any differences between groups.

Conclusion

Numerous sectors of the health research enterprise, including this journal (Graham et al., 2011), have expressed that building equity in the research process is imperative. These efforts include acquiring large enough samples to benefit those most negatively impacted by structural determinants of health. The intercept method is showing promise for providing more inclusive samples in future RCTs. The costs in task and study completion may be non-trivial, yet we encourage researchers to use this approach while thoughtfully assessing study-specific participant burden, sampling estimates and power analyses, compensatory additions to data collection protocols, and the additional time and resources required relative to the potential for greater equity in research. We also hope to further motivate funders to facilitate access to these resources for more inclusive research.