Abstract
Background:
Choosing the right recruitment strategy has implications for the successful conduct of a trial. Our objective was to compare a novel peer recruitment strategy to four other recruitment strategies for a large randomized trial testing a digital tobacco intervention.
Methods:
We compared enrollment rates, demographic and baseline smoking characteristics, and odds of completing the 6-month study by recruitment strategy. Cost of recruitment strategies per retained participant was calculated using staff personnel time and advertisement costs.
Findings:
We enrolled 1,487 participants between August 2017 and March 2019 from: Peer recruitment n=273 (18.4%), Facebook Ads n=505 (34%), Google Ads=200 (13.4%), ResearchMatch n=356 (23.9%) and Smokefree.gov n=153 (10.3%). Mean enrollment rate per active recruitment month: 1) Peer recruitment, n=13.9, 2) Facebook ads, n=25.3, 3) Google ads, n=10.51, 4) Research Match, n=59.3, and 5) Smokefree.gov, n=13.9. Peer recruitment recruited the greatest number of males (n=110, 40.3%), young adults (n=41, 14.7%), participants with a high school degree or less (n=24, 12.5%) and smokers within one’s social network. Compared to peer recruitment (retention rate=57%), participants from Facebook were less likely (OR 0.46, p<0.01, retention rate=40%), and those from ResearchMatch were more likely to complete the study (OR 1.90, p<0.01, retention rate=70%). Peer recruitment was moderate in cost per retained participant ($47.18) and substantially less costly than Facebook ($173.60).
Conclusions:
Though peer recruitment had lower enrollment than other strategies, it may provide greater access to harder to reach populations and possibly others who smoke within one’s social network while being moderately cost-effective.
ClinicalTrials.gov:
Keywords: Smoking cessation, Peer recruitment, Digital Intervention, Tailored, Dissemination
Background
Recruitment of participants to randomized trials is an ongoing challenge. Only about 40% of trials are successful in recruiting their proposed sample (1) and many struggle to recruit diverse populations.(2) Despite the promise of increased reach, this is also true for recruitment to trials evaluating digital interventions, such as web-assisted tobacco interventions.(3) There is now increasing recognition that we need more data on different recruitment strategies.(4) Choosing the right recruitment strategy has implications for the successful conduct of a trial and requires weighing multiple factors, including time to recruit, target population, and cost. Despite the importance, only a few studies have evaluated the success of recruitment strategies in digital smoking cessation trials.(3)
A few studies have evaluated recruitment strategies to digital tobacco interventions, including evaluating social and online advertisements. Graham et al. reported a high enrollment rate (9.1%) of participants who smoke using online advertisements.(5) However, other studies have reported that they were less successful recruiting participants who smoke using online advertisements.(6) Further, the cost to recruit also varied among studies. While Graham et al. reported a low cost per participant using online advertisements ($5-$8 per participant), Hefferner et al reported a higher recruitment cost to their online smoking cessation study ($42.48 per participant).(7) There are also reported methodological challenges of recruiting participants who smoke from a single source, as the population recruited may not be representative of the general population of those who smoke. For example, we found that recruiting participants who smoke from clinical practices complimented online advertisements attracting those less motivated to quit and less experienced with Web-assisted tobacco interventions.(8) Smoking cessation trials will benefit from being able to choose from multiple approaches to recruit participants who smoke.
Our objective was to compare a novel peer recruitment strategy to four other recruitment strategies for a large randomized trial testing a digital tobacco intervention. Peer recruitment, where enrolled participants recruited new participants uses the natural power of social networks, can spread use of health interventions between social contacts and networks (9, 10). Peer recruitment can potentially facilitate access to groups who are less motivated to participate in interventions. Peer recruitment has rapidly become the method of choice for recruiting hard-to-reach subjects using their social networks.(11) Our prior pilot demonstrated the potential of using peer recruitment to recruit people who smoked to an online web intervention.(12) We found that the peer recruitment increased reach to those who smoked and successfully expanded the number and variability of the sample.(12) Despite the promise, peer recruitment is underutilized for recruiting participants into digital interventions.
In this paper, we report a more real-world evaluation of the peer recruitment strategy. We compared peer recruitment with four other recruitment approaches: social media advertisements, online advertisements, ResearchMatch, and posting on smokefree.gov.(13) In a systemic mapping of recruitment strategies used in 105 comparative studies, social media and online advertisements were found to be the most widely used approach.(4) ResearchMatch is a National Institutes of Health funded resource for recruitment which is currently being used in many studies.(14) Smokefree.gov is National Cancer Institute funded online smoking resource that is widely used by people who smoke and gaining more traction as a recruitment source for smoking cessation studies.(15) The objective of this paper is to compare enrollment rates, diversity of participant characteristics, retention, and cost of recruitment strategies for a digital smoking cessation trial. These results should provide additional data for studies to select appropriate recruitment strategies.
Methods
Study design, Setting, and Participants
The present report is a cohort study of people who smoke who enrolled into a large 6-month hybrid effectiveness dissemination trial (Patient-Centered Outcomes Research Institute funded award (CDR-1603-34645) testing a web-assisted tobacco digital intervention (Smoker-to-Smoker, or S2S). Eligible participants were asked to participate in a 6-month trial in which they would receive tailored motivational messages related to smoking cessation via email or text, have access to functions on the website and possibly be asked to recruit their family and friends. The primary outcome measure was self-report 7-day point prevalence obtained during the 6-month follow-up. Details of S2S have been published elsewhere.(13) In brief, the eligibility requirements for the study included: 1) English-speaking, 2) currently smoke (as determined by a self-report question, “Do you currently smoke?”), and 3) aged ≥18 years. The S2S trial enrolled 1,478 participants between August 2017 and March 2019. The study began by using three recruitment strategies (peer recruitment, Facebook and Google ads) with two additional strategies (ResearchMatch and smokefree.gov) added in at a later time to bolster enrollment numbers. Regardless of the recruitment strategy, the enrollment process and data collection methods were the exact same for each participant once they were directed to the Decide2Quit website. All participants (once they were directed to the website) first completed a baseline survey. They were also sent a survey to complete at one-week and one-month post-enrollment. The final follow-up for the study to assess smoking cessation was conducted at six-months. Participants could receive up to $100 in incentives for completion of the surveys ($25 each for the one-week and one-month survey and up-to $50 for the six month survey). Individuals contact information (phone number and email address) were verified prior to distributing incentives to verify validity. This study was approved by the UMass Medical School Institutional Review Board.
Recruitment approach
As noted, we used multiple approaches to recruit participants. We initially started with Facebook ads, Google ads, and peer recruitment. We added two other recruitment approaches, ResearchMatch and Smokefree.gov, as needed to supplement our existing recruitment strategies. We detail each strategy below, including the timeframe of “active recruitment efforts”, or the time the study team was actively devoting staff time and financial resources for each strategy:
Peer recruitment:
During enrollment, participants were encouraged to peer recruit their friends or family members to the website. To do this, we started with the standard online recruitment of seeds. The seeds are the first wave of smokers who are directly recruited via other recruitment strategies. Smokers recruited by the seeds and subsequent waves of smokers are the peer recruits. The seeds and peer recruits were provided the peer recruitment toolset to recruit others. The toolset included a Facebook website plugin, an online training video, and a recruitment tracker we developed in our pilot.(12) The Facebook plugin allowed smokers to browse through their Facebook friends and recruit them by sending private recruitment messages. They were also able to peer recruit through text messages and emails. The online training video taught the recruiter how to use available tools to recruit other smokers from their social network. The recruitment tracker will allow smokers to track successful peer recruitment. We also emailed a weekly motivational email encouraging peer recruitment (see Appendix). Active peer recruitment efforts were conducted between August 2017 and March 2019 for a total of 20 months.
Facebook and Google recruitment involved posting online advertisements targeted to people searching for “quit smoking” related search terms online. When they clicked on these advertisements, they were redirected to Decide2Quit, where they were provided study information and enrollment instructions. Examples of keywords we used were, “quit smoking”, and “smoking cessation, and “stop smoking”, and we limited our search to adults in the United States over the age of 18 years. We used the functions provided in the ad managers of Facebook and Google to target ads for those who smoked. For example, the Facebook ad manager allows advertisers to target users based on their interests derived from their profile’s keywords, pages they like, and groups they visit. Advertisements were then displayed on the Facebook page of the user. Facebook and Google ad active recruitment efforts were conducted between August 2017 and March 2019 for a total of 20 months.
ResearchMatch is a free and secure online tool developed by Vanderbilt University and used by researchers nationwide.(16) Volunteers enroll on the ResearchMatch website, fill out demographic and optional health history questionnaires, and submit their profile. Using the ResearchMatch search engine, research assistants searched for appropriate matches (according to eligibility criteria) amongst the non-identifiable research match volunteer profiles in the system. ResearchMatch volunteers who were eligible to participate in the study received an email with an invitation letter that provided more information about S2S. Recipients had the opportunity to review the invitation letter and decide whether they would like to participate in the study. ResearchMatch active recruitment efforts were conducted between June 2018 and November 2018 for a total of 6 months.
Smokefree.gov is a website created by the National Cancer Institute designed to help people quit smoking (smokefree.gov). It is a part of the Health and Human Services’ efforts to reduce smoking rates in the United States. The website contains quizzes, tips, plans, apps, text messaging programs, research studies and other ways to get ready to quit and remain smoke-free. To advertise on smokefree.gov, research assistants worked with website staff to display information about the study. On the www.smokefree.gov website, under the “join a research study” tab, we provided the name of the study, a link to the study website and a one-paragraph description of the study. The link listed on www.smokefree.gov directed potential participants to the Decide2Quit website. Smokefree.gov recruitment efforts were conducted between May 2018 and March 2019 for a total of 11 months.
Retention approach
We contacted participants via phone and email to complete their 6-month survey (emails had the survey link from the secure data management website, REDCap). Emails were sent out up-to five times and phone calls were made up to six times. Retention rates were calculated as the number of participants who completed the 6-month survey divided by the number of participants who enrolled into the study.
Data collection and Measures
We collected data on the months of active recruitment for each strategy, as all strategies were not used during the entirety of the recruitment period. As previously mentioned, active recruitment was defined as the time we were using resources (staff time and paid advertisements) to actively recruit from each source. Upon enrollment, participants indicated how they heard about the study (Facebook ad, Google ad, ResearchMatch, Smokefree.gov or peer recruitment) in a multiple-choice question. During enrollment, we collected baseline demographic data of age, gender, race, ethnicity and education level. One week following enrollment, we collected data on smoking behaviors including number of cigarettes per day, current smoking status, use of nicotine replacement therapy, participation in tobacco counseling, nicotine dependence using the Fagerstrom test,(17) past use of e-cigarettes, readiness to quit and connections to other people who smoke.
The total recruitment cost calculation included personnel time for developing and managing each strategy in addition to advertisement costs (Facebook and Google ads). Personnel time was determined by detailed staff logs of time spent on each activity during both the development and active recruitment phases of each strategy. Because all advertisements were posted online, development costs included personnel time to create advertisement posts, communicate with marketing departments of respective platforms and posting of the ads (Facebook, Google, ResearchMatch and Smokefree.gov). Personnel cost for all strategies were based on research assistants’ hourly rates, except for peer recruitment, as study personnel carrying out tasks for this strategy included a computer programmer. The peer recruitment strategy required running administration reports and programming email/messaging frequency, thus these costs are reflected in our calculations. We tracked the number of weeks each phase (development and active recruitment) lasted per strategy and summed these totals to determine a total personnel cost. These costs combined with cost to run advertisements, yielded a total recruitment cost.
Statistical Analysis
Mean enrollment rates per month were calculated by dividing the total number of participants enrolled through each strategy by the number of months the strategy was used. We compared baseline demographic characteristics, smoking behaviors and social networks (number of people who smoke in their social networks) across levels of recruitment strategies using χ2 test for categorical variables and t-tests for continuous variables. We used logistic regression to estimate the odds ratios of completing the study (or the 6-month survey) with recruitment strategies as the independent variable. Peer recruitment was used as the reference category in the logistic model. Cost of recruitment strategies was calculated as cost per participant by recruitment strategy. Cost per enrolled participant was calculated as the ratio of the total cost of a recruitment strategy divided by total number of participants enrolled, and cost per retained participant was calculated as the total cost per recruitment strategy (completed the six-month survey) divided by the total number of participants retained per recruitment strategy. Stata version 15 was used for all analyses.
Results
Recruitment overview
We recruited 1,487 participants using 5 recruitment approaches. The sample consisted of 18% (273/1478) of participants recruited through peer recruitment, 34% (505/1478) via Facebook, 24% (356/1478) via ResearchMatch, 14% (200/1478) via Google ads, and 10% (153/1478) via Smokefree.gov. The mean enrollment rate per active recruitment month by strategy is as follows: 1) Research Match, n=59.3, 2) Facebook ads, n=25.3, 3) peer recruitment, n=13.9, 4) Smokefree.gov, n=13.9, and 5) Google ads, n=10.5.
Participant demographics
Table 1 presents demographic differences across recruitment strategies (N=1478). Peer recruitment led to a significantly greater percentage of participants with lower education levels (some high school) as compared to Facebook (12.5% v. 3.6%; p<0.05). Participants recruited from peer recruitment had significantly lower ages compared to all other strategies (all p’s<0.05). The percentage of adults over the age of 55 (12.5%) was significantly less than that of Facebook (71.5%, p<0.05). Gender distribution for participants recruited through peer recruitment was significantly different from Facebook, Google and ResearchMatch distribution (p<0.05) and recruited the greatest percentage of males (40.3%).
Table 1.
Comparison of demographic factors across recruitment strategies
| Participant Characteristics | Research Match | smokefree.gov | Peer recruitment | ||
|---|---|---|---|---|---|
| n = 505 | n = 200 | n = 356 | n = 153 | n = 273 | |
| Age | |||||
| 19-24 | 14 (2.8%)** | 16 (8.0%)* | 24 (6.7%)** | 28 (18.3%)* | 40 (14.7%) |
| 25-34 | 26 (5.1%) | 69 (34.5%) | 89 (25.0%) | 46 (30.1%) | 92 (33.7%) |
| 35-44 | 39 (7.7%) | 46 (23.0%) | 79 (22.2%) | 32 (20.9%) | 77 (28.2%) |
| 45-54 | 65 (12.9%) | 30 (15.0%) | 84 (23.6%) | 26 (17.0%) | 30 (11.0%) |
| 55-64 | 273 (54.1%) | 32 (16.0%) | 63 (17.7%) | 19 (12.4%) | 21 (7.7%) |
| 65 + | 88 (17.4%) | 7 (3.5%) | 17 (4.8%) | 2 (1.3%) | 13 (4.8%) |
| Gender | |||||
| Female | 427 (84.6%)** | 148 (74.0%)** | 263 (73.9%)** | 100 (65.4%) | 163 (59.7%) |
| Male | 78 (15.4%) | 52 (26.0%) | 93 (26.1%) | 53 (34.6%) | 110 (40.3%) |
| Race1 | |||||
| White | 450 (91.3%)** | 140 (74.1%) | 273 (79.1%) | 107 (73.8%) | 197 (78.8%) |
| African American | 33 (6.7%) | 41 (21.7%) | 52 (15.1%) | 24 (16.6%) | 38 (15.2%) |
| Other race# | 10 (2.0%) | 8 (4.2%) | 20 (5.8%) | 14 (9.7%) | 15 (6.0%) |
| Ethnicity1 | |||||
| Not Hispanic/Latino | 452 (96.0%)** | 162 (88.5%) | 313 (91.5%) | 132 (90.4%) | 231 (90.9%) |
| Hispanic | 19 (4.0%) | 21 (11.5%) | 29 (8.5%) | 14 (9.6%) | 23 (9.1%) |
| Education2 | |||||
| Some high school | 13 (3.6%)** | 4 (4.0%)** | 8 (3.2%)** | 3 (3.6%) | 24 (12.5%) |
| High school graduate | 89 (24.7%) | 20 (19.8%) | 47 (18.7%) | 27 (32.5%) | 64 (33.3%) |
| Some college/technical school | 161 (44.7%) | 48 (47.5%) | 110 (43.8%) | 34 (41.0%) | 73 (38.0%) |
| College graduate | 97 (26.9%) | 29 (28.7%) | 86 (34.3%) | 19 (22.9%) | 31 (16.1%) |
Asian, American Indian or Alaska Native, Native Hawaiian or Other Pacific Islander
p-value less than 0.05 (peer recruitment as comparison group)
p-value less than 0.01
Includes those who refused to answer or didn’ know the answer to the question.
Data collected at 1-week survey (n=991)
Smoking characteristics
For the sample of 1,478, as compared to peer recruited participants mean reported cigarette consumption per day, those recruited from Facebook and Google ads had a significantly greater mean consumption (p’s<0.05; table 2). Peer recruited participants reported greater use of Nicotine Replacement Therapy compared to those recruited from ResearchMatch, Facebook, and Google (p’s<0.05). A significantly greater number of peer recruited participants previously participated in tobacco counseling compared to those from Facebook (p<0.01). For the sample of 991 participants from the one-week survey, peer recruited participants were more likely to use e-cigarettes recently and frequently compared to participants from Facebook and ResearchMatch (p’s<0.01). Using the Fagerstrom nicotine dependence scale(13), we found a significant difference in nicotine dependence in participants recruited from Facebook compared to those who were peer recruited (p<0.05). In assessing readiness to quit, as compared to those from peer recruitment, participants recruited from Facebook, Google ads, and ResearchMatch had significant differences in readiness to quit.
Table 2.
Comparison of baseline smoking habits across recruitment strategy.
| Research Match | smokefree.gov | Friend/family | |||
|---|---|---|---|---|---|
| Baseline data | |||||
| n = 505 | n = 200 | n = 356 | n = 153 | n = 273 | |
| Cigarettes smoked per day, M (SD) | 18.8 (9.5) | 16.8 (23.3) | 14.7 (9.6)* | 20.9 (72.3) | 16.2 (11.5) |
| Nicotine Replacement Therapy | |||||
| No | 346 (68.5% | 160 (80%)* | 210 (59%) | 101 (66%) | 158 (57.9%) |
| Yes | 159 (31.5%) | 40 (20%) | 146 (41%) | 52 (34%) | 115 (42.1%) |
| Participated in tobacco counseling | |||||
| No | 466 (92.3%) | 168 (84.0%)* | 304 (85.4%)* | 138 (90.2%) | 228 (83.5%) |
| Yes | 39 (7.7%) | 32 (16%) | 52 (14.6%) | 15 (9.8%) | 45 (16.5%) |
| Readiness to quit | |||||
| Not thinking of quitting | 6 (1.2%)* | 6 (3%)* | 31 (8.7%)* | 4 (2.6%) | 15 (5.5%) |
| Thinking of quitting | 292 (57.8%) | 86 (43%) | 238 (66.8%) | 75 (49%) | 164 (60.1%) |
| Already quit | 43 (8.5%) | 16 (8%) | 18 (5.1%) | 13 (8.5%) | 19 (7%) |
| Set a quit date | 116 (23%) | 51 (25.5%) | 54 (15.2%) | 49 (13.2%) | 57 (20.9%) |
| Quit today | 42 (8.3%) | 41 (20%) | 13 (3.7%) | 12 (7.8%) | 16 (5.9%) |
| One-week data | |||||
| n = 359 | n = 100 | n = 250 | n = 83 | n = 193 | |
| Have you ever tried an e-cigarette? | |||||
| No | 87 (24.0%)* | 27 (27.0%) | 46 (18.3%)* | 15 (18.1%) | 45 (23.1%) |
| Yes | 275 (76.0%) | 73 (73.0%) | 205 (81.7%) | 68 (81.9%) | 150 (76.9%) |
| Used an e-cigarette in the past 30 days | |||||
| Every day | 15 (4.2%)* | 8 (8.0%) | 14 (5.6%)* | 7 (8.4%) | 17 (8.7%) |
| Some days | 69 (19.1%) | 37 (37.0%) | 56 (22.3%) | 18 (21.7%) | 68 (34.9%) |
| Not at all | 277 (76.7%) | 55 (55.0%) | 181 (72.1%) | 58 (69.9%) | 110 (56.4%) |
| Nicotine Dependence Scale | |||||
| Within 5 minutes | 161 (44.5%) | 40 (40.0%) | 76 (30.3%)* | 38 (45.8%) | 71 (36.4%) |
| 6-30 minutes | 142 (39.2%) | 35 (35.0%) | 99 (39.4%) | 26 (31.3%) | 73 (37.4%) |
| 31-60 minutes | 30 (8.3%) | 16 (16.0%) | 30 (12.0%) | 8 (9.6%) | 27 (13.8%) |
| After 60 minutes | 29 (8.0%) | 9 (9.0%) | 46 (18.3%) | 11 (13.3%) | 24 (12.3%) |
p-value less than 0.05 (Peer recruitment as comparison group)
Connections with others who smoke
Complete data were available for 991 participants from the one-week survey. Table 3 presents differences in participants connections to other people who smoke between recruitment strategies. Peer recruited participants reported a significantly higher percentage of people who smoke at home (55.4%) as compared to all other strategies (p<0.05). We asked participants to report the number of people who smoke within six groups of people connected to them: Immediate family, extended family, close friends, friends, acquaintances, and co-workers. Peer recruited participants reported the highest mean number of people who smoke in 3 of the 6 groups (immediate family 2.6±2.6, close friends 3.5±3.7, and co-workers 4.3±9.4) as compared to all other strategies.
Table 3.
Comparison of participants connections to others who smoke across recruitment strategies.
| Research Match | Smokefree.gov | Peer recruitment | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| n = 362 | n = 100 | n = 251 | n = 83 | n = 195 | ||||||
| Does anyone else living in your home smoke cigarettes? | ||||||||||
| n (%) | n (%) | n (%) | n (%) | n (%) | ||||||
| No | 250 (69.1%)* | 65 (65.0%)* | 130 (51.8%) | 48 (57.8%)* | 87 (44.6%) | |||||
| Yes | 112 (30.9%) | 35 (35.0%) | 121 (48.2%) | 35 (42.2%) | 108 (55.4%) | |||||
| M (SD) | Med (IQR) | M (SD) | Med (IQR) | M (SD) | Med (IQR) | M (SD) | Med (IQR) | M (SD) | Med (IQR) | |
| Connections to others who smoke | ||||||||||
| Immediate family | 1.5 (2.0)^ | 1 (2) | 2.1 (2.4) | 1(3) | 1.5 (1.8)^ | 0 (1) | 1.7 (2.1)* | 0 (1) | 2.6 (2.6) | 2 (3) |
| Extended family | 2.9 (5.5) | 1 (4) | 3.9 (5.3) | 2 (6) | 2.7 (3.4)^ | 0 (2) | 3.7 (6.6) | 2 (5) | 3.7 (4.0) | 3 (5) |
| Close friends, with whom you feel at ease and discuss private matters | 2.2 (3.1)^ | 1 (3) | 3.7 (4.9) | 2 (3) | 2.7 (2.7)^ | 2 (3) | 2.4 (2.3)^ | 2 (3) | 3.5 (3.7) | 3 (4) |
| Friends, with whom you feel at ease but DO NOT discuss private matters | 3.3 (6.7) | 2 (5) | 4.3 (6.0) | 2 (6) | 3.9 (8.4) | 2 (5) | 2.4 (3.3)* | 1 (3) | 3.8 (6.0) | 2 (5) |
| Acquaintances, to whom you may say hello but about whom you know little1 | 7.4 (14.9) | 2 (9) | 8.8 (16.0) | 3 (10) | 6.0 (12.0) | 2 (6) | 5.0 (12.9) | 1 (5) | 8.5 (19.4) | 2 (8) |
| Co-workers1 | 3.5 (10.4) | 0 (3) | 3.5 (4.5)* | 1 (5) | 2.7 (7.5) | 0 (3) | 3.9 (7.5) | 1 (4) | 4.3 (9.4) | 2 (5) |
p-value less than 0.05 (peer recruitment as comparison group)
p-value<0.01 (peer recruitment as comparison group)
Maximum value limited to 100
M=Mean; SD=Standard Deviation; Med=Median; IQR=Interquartile Range
Retention by recruitment strategy
The retention rate for peer recruited participants was 57% (156/273; see Table 3). Retention rates for other recruitment strategies were as follows: 71% (254/356) for ResearchMatch, 57% (87/153) for Smokefree.gov, 54% (108/200) for Google ads, and 40% (200/505) for Facebook advertisements. Compared to peer recruitment, participants enrolled through Facebook were less likely to complete the study (OR 0.46; CI 0.34 to 0.62), while those from ResearchMatch were more likely to complete the trial (OR 1.90, CI 1.36 to 2.64; See Figure 1).
Figure 1:
Forest plot showing the odds ratios and 95% confidence intervals for completing the study (6 months survey) by recruitment strategy using Peer recruitment as the reference category.
*Statistically significant at p<0.05. Peer recruitment as comparison group.
Cost of recruitment strategies
Peer recruitment personnel costs were a total of $7360.00 ($360.00 monthly). Other strategy’s personnel costs included: Facebook $8,670.09 total ($433.50 monthly), ResearchMatch $4439.53 total ($739.92 monthly), Google $2,826.49 total ($166.26 monthly) and Smokefree.gov $54.99 total ($7.86 monthly). Two strategies, Facebook and Google ads also required additional costs for advertisements: Facebook ($26,050 total and $1302.50 monthly), and Google ($21,758 total and $1279.90 monthly). As shown in Table 4, peer recruitment cost $26.96 per enrolled participant and $47.18 per retained participant. Facebook was the costliest recruitment strategy at $68.75 per enrolled participant, though enrolled the highest number. Smokefree.gov was the least costly at $0.36 per enrolled participant but enrolled the lowest number.
Table 4.
Cost per participant by recruitment strategy and retention rates.
| Number enrolled | Cost per enrolled participant (total) | Number retained | Retention Rate | Cost per retained participant (total) | |
|---|---|---|---|---|---|
| Facebook Ad2 | 505 | $68.75 | 200 | 40% | $173.60 |
| Google Ad2 | 200 | $22.20 | 108 | 54% | $41.11 |
| ResearchMatch2 | 356 | $12.47 | 254 | 71% | $17.48 |
| Smokefree.gov2 | 153 | $0.36 | 87 | 57% | $0.63 |
| Peer recruitment1 | 273 | $26.96 | 156 | 57% | $47.18 |
Includes the cost of admin reports and programming email frequency. The toolkit and website development took 445 hours at a Computer Programmer average salary $36.80/hour; (Glassdoor); Total cost to develop toolkit was $13,432.
Based on research assistants time and salary ($38,120 annual salary (Glassdoor) $18.33/hour). Includes start-up cost of the ads/platforms.
Discussion
Recruitment into research trials, including digital trials, continues to present challenges for researchers. We compared a novel peer recruitment approach with other recruitment strategies that are more widely used.(4) Peer recruitment accounted for 18% of enrolled participants and increased the variability and diversity of the sample. Peer recruitment was modestly cost-effective, but also increased the number of participants with lower education levels, greater number of connections to others who smoke, and greater use of e-cigarettes, while having a modest retention rate of 57%. The results of each strategy should be weighed and taken into context based on the needs of the present and future studies. Below, we place our results in context and will discuss future work.
Our overall peer recruitment enrollment rate (18% of the sample) was less than our pilot study (75%),(13). However, the pilot was focused on testing whether peer recruitment works, and we had used only two strategies to recruit participants who smoke (Facebook and peer recruitment). The Facebook ads were used to recruit the initial seeds who were then incentivized ($5 for each recruitment (to recruit the subsequent waves). In contrast, peer recruitment was one of the five strategies we used to recruit smokers for a trial in the current study. Further, in the current study, participants were not incentivized to peer recruit. Our peer recruitment enrollment rate compares favorably with a longitudinal trial that examined men’s health using peer recruitment to enroll 16.6% of their sample.(18) However, this study differed from our approach in that it relied on paper or electronic referrals to be delivered by enrolled participants to others in their network and included incentives for every successfully enrolled peer recruited participant. Our trial provided options for recruiting participants via email, texting and Facebook messaging and did not provide incentives for recruiting other participants. Although some researchers have recommended the use of incentives to promote participation(19), having participants peer recruit without incentives may be more realistic for real world adoption of peer recruitment.
Participant demographics differed across recruitment strategies, with peer recruitment resulting in more proportionate numbers of men (~40%) and women (~60%) as compared to other strategies. Recruiting men to online smoking cessation trials is a challenge. Prior online recruitment, especially via social media or online advertisements, have resulted in recruiting a higher proportion of women.(20) It also resulted in enrollment of a greater percentage of participants with lower education levels. This population has been difficult to recruit in prior smoking cessation interventions (21, 22), suggesting peer recruitment may be a useful strategy to recruit hard-to-reach populations. As many as 46% of participants recruited by peers had a high school degree or less, compared to rates of 24% and 34% reported by other trials using online recruitment methods.(5, 8) Data show that those with a high school education smoke cigarettes for a duration of more than twice as many years as people with at least a bachelor’s degree,(23) thus peer recruitment is a possible strategy to increase participation of this at-risk population.
While our pilot evaluation of peer recruitment resulted in a higher proportion of African Americans who smoked (especially males), we did not see similar findings in this study. The proportion of African Americans who were peer recruited (15.2%) was similar to other strategies (range 15.1 to 21.7%), with the exception of Facebook which was lower and recruited a considerably high proportion of White participants. Aside from Facebook, the percent of African Americans recruited using other strategies was higher than those reported in other trials using online recruitment [rates of 5.5% (5) and 8% (24)] using online strategies. African Americans who smoke are a challenging group to recruit to digital trials and understanding methods to increase their participation is of considerable importance. One reason for lower recruitment in this trial could be the lack of incentives provided to participants to encourage peer recruitment. As previously noted, in our pilot we provided an incentive of up-to $15 for each peer recruited participant. Following guidance from our funders and our panel of stakeholders to keep the current trial as real-world as possible, we did not provide any incentives to participants. Incentives have been shown to overcome barriers and, influence participation.(25) In lieu of incentives, other non-monetary incentives should be explored to increase participation of minority groups. Another reason may be our lack of targeting ads, through content and who ads were displayed, toward racially and ethnically diverse populations. As noted above, Facebook recruited a higher percentage of non-Hispanic and White participants. Other trials have shown similar low rates of recruitment of racially and ethnically diverse participants using Facebook advertisements.(3, 26) Though national data suggests the use of Facebook is similar amongst white (70%), Black (70%) and Hispanic (69%) United States adults (27), prior research and our data suggest Facebook may not be the best platform to reach diverse populations.
The peer recruitment strategy resulted in a younger population of participants which may explain their higher usage of e-cigarettes.(28) While the harmful effects of e-cigarettes relative to regular cigarettes are controversial (29, 30), peer recruitment may be a viable option for researchers who want to target young people who smoke and those using e-cigarettes. Peer recruited participants also reported the highest use of nicotine replacement therapy (NRT). Studies have shown that the use of NRT is lower among less educated smokers.(31–33) Thus, the higher use of NRT among peer recruits’ contrasts with our above results, which shows that a higher proportion of peer recruited participants were less educated. Additional investigations are needed to explain this contrasting finding.
Peer recruited participants reported a higher average number of people who smoked in their social networks, and more than half reported that they lived with a person who smoked. Prior studies have shown that people who smoke tend to be connected to each other.(10, 34, 35) The size of the network suggests that peer recruitment has the potential to continue propagating the intervention to a large number of those who smoke. The social network of a person who smokes has been shown to have considerable influence on them. Christakis et al. showed that smokers often quit in unison with their networks.(10) Trials have also shown that a person is more likely to have a smoking relapse if they have more people within their network who smoke. (34) Thus, long-term studies may particularly benefit from including peer recruitment into their recruitment strategies to not only aid in recruitment, but also in promoting and longitudinal maintenance of cessation.
Retention of participants to digital smoking cessation intervention is challenging.(36–40) We were able to retain 57% of our peer recruited participants to complete the six-month follow-up. This rate is similar to our prior trial in which we recruited participants to an digital smoking cessation interventions via referrals from medical provider.(41) Our participants recruited through Facebook had a lower retention rate compared to all other strategies, consistent with other trials that recruited people who smoked from Facebook (42, 43). The high retention rate (71%) from ResearchMatch is possibly attributed to their motivation to pursue changes to their health, as indicated by them voluntarily signing up on ResearchMatch.(14)
Peer recruitment was the second least costly strategy per retained participant in the trial. Smokefree.gov was the most cost-effective, though resulted in the lowest number of participants. ResearchMatch was also very cost-effective. However, it required the most personnel time to sift through potentially eligible participants, send emails and conduct follow-up phone calls. Facebook ads resulted in the highest number of enrolled participants but was the costliest resource. Both Google and Facebook require extensive advertising costs. Ramo et al. estimated the cost to recruit participants into a smoking cessation trials using Facebook ads was $10 per enrolled participant (42), compared to our costs of $68.75. However, it is unclear whether the $10 figure incorporated personnel time for ad development and for running ads into their analyses. In a prior study, we found that the cost to recruit a participant to our smoking cessation trial was $74. Using previously developed technology to facilitate peer recruitment only required staff personnel time to perform administrative tasks.
There were several notable limitations. Cost estimates are based on a number of variables, including study staff estimates of time spent on recruitment activities per strategy. These staff estimations of time were performed after the end of the recruitment period; thus, staff may have been susceptible to recall bias and time can only be considered as a conservative estimation. Secondly, information on why participants declined to participate by recruitment strategy was not collected, though would be useful knowledge to ascertain to address reported barriers in future trials. Third, participants were unable to report hearing about the study from multiple recruitment strategies, thus we may have only captured one out of multiple ways they heard about the study. Fourth, different recruitment strategies were used for varying amounts of time to meet the needs of the study’s recruitment numbers. Though we accounted for this in our cost analyses, the time variance may affect overall recruitment numbers. Fifth, as this was a dissemination trial, many of our measures involved self-report, including the verification of smoking upon enrollment. Lastly, we do not have data on responses to specific ads used within each strategy, such as which ads were viewed more than others and what the demographics of participants viewing said ads consisted of. We also were not able to capture demographic data on those who “clicked” on the ads to be able to compare them to those who enrolled in the trial. Though we used a patient panel to aid in developing our recruitment materials, we did not develop materials to specifically target African Americans who smoke. Future trials could benefit from using these targeted methods as a means of increasing reach to this population and may also track data by measuring “clicks” generated from each ad to examine responses, though this may not directly lead to enrollment.
Conclusion
Although peer recruitment had a modest enrollment rate (13.9) in comparison to Facebook (25.3) and ResearchMatch (59.3), this should be evaluated in terms of the other potential benefits of peer recruitment. Our findings suggest that peer recruitment may provide greater access to harder to reach populations and possibly others who smoke within one’s social network. It was also modestly cost-effective per retained participant in the trial. Using social media approaches may be prudent for researchers with large budgets who are interested in enrolling high numbers of participants, though are subject to high attrition rates. ResearchMatch may require more staff personnel time, though may be able to recruit and enroll participants who are more likely to complete the study. Although posting on social media yielded fewer participants, it may be a cost-effective way to supplement other recruitment approaches. Researchers may consider these findings as they determine the selection of recruitment strategies, taking in consideration the research budget, target population demographics, smoking characteristics and social networks.
Highlights.
Recruitment remains an issue for digital smoking cessation interventions
Technology peer recruitment via Facebook, email and text may increase reach
Peer recruits were lower educated, younger, and more gender-proportionate
Peer recruits had greater connections to other smokers in 4 of 6 social networks
Peer recruitment has real-world dissemination potential in future trials
Acknowledgements
We would like to thank the S2S program staff, IT team as well as the patient panel members for their contributions to the project.
Funding
This work was supported through a Patient-Centered Outcomes Research Institute (PCORI) Program Award (CDR-1603-34645), NHLBI (1K12HL138049-01), NCI (1P50 CA244693-01) and NCI PRACCTIS Grant (CA172009). The content is solely the responsibility of the authors and does not necessarily represent the official views of PCORI or the National Institutes of Health.
Abbreviations:
- S2S
Smoker-to-smoker
Appendix. Email to participants in peer recruitment group
Dear <Participant Name>,
Thank you once again for registering on our website. We hope that you are finding our site helpful in reaching your goal of quitting smoking.
We are sending this email as a friendly reminder that you have the opportunity to refer your friends and family to our website.
There are a number of reasons to refer your friends or family, the most important of which is you will be helping them to quit smoking.
Quit smoking systems like our work but are not used by many smokers. Your referral will help us reach many more smokers so that we can support them on their journey to quit smoking.
You can use the referral tools we have developed by logging on to www.decide2quit.org.
AUTHOR DECLARATION
We wish to draw the attention of the Editor to the following facts which may be considered as potential conflicts of interest and to significant financial contributions to this work. [LIST POTENTIAL CONFLICTS OF INTEREST HERE]
[OR]
We wish to confirm that there are no known conflicts of interest associated with this publication and there has been no significant financial support for this work that could have influenced its outcome.
We confirm that the manuscript has been read and approved by all named authors and that there are no other persons who satisfied the criteria for authorship but are not listed. We further confirm that the order of authors listed in the manuscript has been approved by all of us.
We confirm that neither the entire paper nor any of its content has been submitted, published, or accepted by another journal. The paper will not be submitted elsewhere if accepted for publication in the Journal.
We confirm that we have given due consideration to the protection of intellectual property associated with this work and that there are no impediments to publication, including the timing of publication, with respect to intellectual property. In so doing we confirm that we have followed the regulations of our institutions concerning intellectual property.
We confirm that any aspect of the work covered in this manuscript that has involved either experimental animals or human patients has been conducted with the ethical approval of all relevant bodies and that such approvals are acknowledged within the manuscript.
We understand that the Corresponding Author is the sole contact for the Editorial process (including Editorial Manager and direct communications with the office). He/she is responsible for communicating with the other authors about progress, submissions of revisions and final approval of proofs.
Signed by all authors as follows:
[LIST AUTHORS AND DATED SIGNATURES ALONGSIDE]
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Declaration of Competing Interest
None
References
- 1.Walters SJ, Bonacho Dos Anjos Henriques-Cadby I, Bortolami O, Flight L, Hind D, Jacques RM, et al. Recruitment and retention of participants in randomised controlled trials: a review of trials funded and published by the United Kingdom Health Technology Assessment Programme. BMJ open. 2017;7(3):e015276–e. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Danaher BG, Seeley JR. Methodological issues in research on web-based behavioral interventions. Ann Behav Med. 2009;38. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Watson NL, Mull KE, Heffner JL, McClure JB, Bricker JB. Participant Recruitment and Retention in Remote eHealth Intervention Trials: Methods and Lessons Learned From a Large Randomized Controlled Trial of Two Web-Based Smoking Interventions. J Med Internet Res. 2018;20(8):e10351. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Frampton GK, Shepherd J, Pickett K, Griffiths G, Wyatt JC. Digital tools for the recruitment and retention of participants in randomised controlled trials: a systematic map. Trials. 2020;21(1):478-. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Graham AL, Milner P, Saul JE, Pfaff L. Online advertising as a public health and recruitment tool: comparison of different media campaigns to increase demand for smoking cessation interventions. J Med Internet Res. 2008;10(5):e50. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Koo M, Skinner H. Challenges of internet recruitment: a case study with disappointing results. J Med Internet Res. 2005;7(1):e6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Heffner JL, Wyszynski CM, Comstock B, Mercer LD, Bricker J. Overcoming recruitment challenges of web-based interventions for tobacco use: the case of web-based acceptance and commitment therapy for smoking cessation. Addict Behav. 2013;38(10):2473–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Sadasivam RS, Kinney RL, Delaughter K, Rao SR, Williams JH, Coley HL, et al. Who participates in Web-assisted tobacco interventions? The QUIT-PRIMO and National Dental Practice-Based Research Network Hi-Quit studies. J Med Internet Res. 2013;15(5):e77. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Huang GC, Unger JB, Soto D, Fujimoto K, Pentz MA, Jordan-Marsh M, et al. Peer influences: The impact of online and offline friendship networks on adolescent smoking and alcohol use. Journal of Adolescent Health. 2013. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Christakis NA, Fowler JH. The collective dynamics of smoking in a large social network. New England journal of medicine. 2008;358(21):2249–58. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Sadasivam RS, Borglund EM, Adams R, Marlin BM, Houston TK. Impact of a Collective Intelligence Tailored Messaging System on Smoking Cessation: The Perspect Randomized Experiment. J Med Internet Res. 2016;18(11):e285. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Sadasivam RS, Cutrona SL, Luger TM, Volz E, Kinney R, Rao SR, et al. Share2Quit: online social network peer marketing of tobacco cessation systems. Nicotine & Tobacco Research. 2017;19(3):314–23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Faro JM, Orvek EA, Blok AC, Nagawa CS, McDonald AJ, Seward G, et al. Dissemination and Effectiveness of the Peer Marketing and Messaging of a Web-Assisted Tobacco Intervention: Protocol for a Hybrid Effectiveness Trial. JMIR research protocols. 2019;8(7):e14814. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Pulley JM, Jerome RN, Bernard GR, Olson EJ, Tan J, Wilkins CH, et al. Connecting the public with clinical trial options: the researchmatch trials today tool. Journal of Clinical and Translational Science. 2018;2(4):253–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Fraser D, Kobinsky K, Smith S, Kramer J, Theobald W, Baker T. Five population-based interventions for smoking cessation: a MOST trial. Translational behavioral medicine. 2014;4(4):382–90. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Vanderbilt Medical Center, researchmatch.org 2020. [Available from: https://www.researchmatch.org/.
- 17.Heatherton TF, Kozlowski LT, Frecker RC, FAGERSTROM KO. The Fagerström test for nicotine dependence: a revision of the Fagerstrom Tolerance Questionnaire. British journal of addiction. 1991;86(9):1119–27. [DOI] [PubMed] [Google Scholar]
- 18.Moore DM, Cui Z, Lachowsky N, Raymond HF, Roth E, Rich A, et al. HIV Community Viral Load and Factors Associated With Elevated Viremia Among a Community-Based Sample of Men Who Have Sex With Men in Vancouver, Canada. J Acquir Immune Defic Syndr. 2016;72(1):87–95. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Volpp KG, John LK, Troxel AB, Norton L, Fassbender J, Loewenstein G. Financial Incentive-Based Approaches for Weight Loss: A Randomized Trial. JAMA. 2008;300(22):2631–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Okoli CTC, Torchalla I, Oliffe JL, Bottorff JL. Men’s smoking cessation interventions: a brief review. Journal of Men’s Health. 2011;8(2):100–8. [Google Scholar]
- 21.Gilman SE, Martin LT, Abrams DB, Kawachi I, Kubzansky L, Loucks EB, et al. Educational attainment and cigarette smoking: a causal association? Int J Epidemiol. 2008;37(3):615–24. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Zhuang Y-L, Gamst AC, Cummins SE, Wolfson T, Zhu S-H. Comparison of smoking cessation between education groups: findings from 2 US National Surveys over 2 decades. American journal of public health. 2015;105(2):373–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Siahpush M, Singh GK, Jones PR, Timsina LR. Racial/ethnic and socioeconomic variations in duration of smoking: results from 2003, 2006 and 2007 Tobacco Use Supplement of the Current Population Survey. Journal of Public Health. 2009;32(2):210–8. [DOI] [PubMed] [Google Scholar]
- 24.Frandsen M, Walters J, Ferguson SG. Exploring the Viability of Using Online Social Media Advertising as a Recruitment Method for Smoking Cessation Clinical Trials. Nicotine & Tobacco Research. 2014;16(2):247–51. [DOI] [PubMed] [Google Scholar]
- 25.Heckathorn D Respondent-driven sampling II. Deriving valid population estimates from chain-referral samples of hidden populations. . Social Problems. 2002;49:11–34. [Google Scholar]
- 26.Derrick JL, Eliseo-Arras RK, Hanny C, Britton M, Haddad S. Comparison of internet and mailing methods to recruit couples into research on unaided smoking cessation. Addict Behav. 2017;75:12–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.PEW Research Center. Share of U.S. adults using social media, including Facebook, is mostly unchanged since 2018 2019. [ [Google Scholar]
- 28.Centers for Disease Control and Prevention. Electronic Cigarettes: Office on Smoking and Health; 2020. [updated February 24, 2020. Available from: https://www.cdc.gov/tobacco/basic_information/e-cigarettes/index.htm.
- 29.Bareham D, Ahmadi K, Elie M, Jones AW. E-cigarettes: controversies within the controversy. The Lancet Respiratory Medicine. 2016;4(11):868–9. [DOI] [PubMed] [Google Scholar]
- 30.Sanford Z, Goebel MDLJ. E-cigarettes: an up to date review and discussion of the controversy. 2014. [PubMed] [Google Scholar]
- 31.Dahne J, Wahlquist AE, Smith TT, Carpenter MJ. The differential impact of nicotine replacement therapy sampling on cessation outcomes across established tobacco disparities groups. Preventive Medicine. 2020;136:106096. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Lei L, Liu F. Medicaid coverage and use of nicotine replacement treatment. Economics & Human Biology. 2021;40:100938. [DOI] [PubMed] [Google Scholar]
- 33.Siahpush M, McNeill A, Borland R, Fong GT. Socioeconomic variations in nicotine dependence, self-efficacy, and intention to quit across four countries: findings from the International Tobacco Control (ITC) Four Country Survey. Tobacco Control. 2006;15(suppl 3):iii71. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Blok DJ, de Vlas SJ, van Empelen P, van Lenthe FJ. The role of smoking in social networks on smoking cessation and relapse among adults: A longitudinal study. Preventive Medicine. 2017;99:105–10. [DOI] [PubMed] [Google Scholar]
- 35.Hitchman SC, Fong GT, Zanna MP, Thrasher JF, Laux FL. The relation between number of smoking friends, and quit intentions, attempts, and success: findings from the International Tobacco Control (ITC) Four Country Survey. Psychol Addict Behav. 2014;28(4):1144–52. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Houston TK, Sadasivam RS, Allison JJ, Ash AS, Ray MN, English TM, et al. Evaluating the QUIT-PRIMO clinical practice ePortal to increase smoker engagement with online cessation interventions: a national hybrid type 2 implementation study. Implement Sci. 2015;10:154. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Danaher BG, Seeley JR. Methodological issues in research on web-based behavioral interventions. Annals of behavioral medicine : a publication of the Society of Behavioral Medicine. 2009;38(1):28–39. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Nelson DB, Partin MR, Fu SS, Joseph AM, An LC. Why assigning ongoing tobacco use is not necessarily a conservative approach to handling missing tobacco cessation outcomes. Nicotine Tob Res. 2009;11(1):77–83. [DOI] [PubMed] [Google Scholar]
- 39.Smolkowski K, Danaher BG, Seeley JR, Kosty DB, Severson HH. Modeling missing binary outcome data in a successful web-based smokeless tobacco cessation program. Addiction. 2010;105(6):1005–15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Saul JE, Amato MS, Cha S, Graham AL. Engagement and attrition in Internet smoking cessation interventions: Insights from a cross-sectional survey of “one-hit-wonders”. Internet interventions. 2016;5:23–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Houston TK, Sadasivam RS, Allison JJ, Ash AS, Ray MN, English TM, et al. Evaluating the QUIT-PRIMO clinical practice ePortal to increase smoker engagement with online cessation interventions: a national hybrid type 2 implementation study. Implementation Science. 2015;10(1):154. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Ramo DE, Rodriguez TM, Chavez K, Sommer MJ, Prochaska JJ. Facebook Recruitment of Young Adult Smokers for a Cessation Trial: Methods, Metrics, and Lessons Learned. Internet interventions. 2014;1(2):58–64. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Ramo DE, Prochaska JJ. Broad reach and targeted recruitment using Facebook for an online survey of young adult substance use. Journal of medical Internet research. 2012;14(1):e28. [DOI] [PMC free article] [PubMed] [Google Scholar]