Abstract
Web-based behavioral interventions for substance use are being developed at a rapid pace, yet there is a dearth of information regarding the most effective methods for recruiting participants into web-based intervention trials. In this paper, we describe our successful recruitment of participants into a pilot trial of web-based Acceptance and Commitment Therapy (ACT) for smoking cessation and compare traditional and web-based methods of recruitment in terms of their effects on baseline participant characteristics, association with study retention and treatment outcome, yield, and cost-effectiveness. Over a 10-week period starting June 15, 2010, we recruited 222 smokers for a web-based smoking cessation study using a variety of recruitment methods. The largest portion of randomized participants were recruited through Google AdWords (36%), followed by medical Internet media (23%), standard media (14%), word of mouth (12%), broadcast emails (11%), and social media (6%). Recruitment source was not related to baseline participant characteristics, 3-month data retention, or 30-day point prevalence smoking abstinence at the 3-month outcome assessment. Cost per randomized participant ranged from $5.27/participant for word of mouth to $172.76/participant for social media, with a mean cost of $42.48/participant. Our diversified approach to recruitment, including both traditional and web-based methods, enabled timely enrollment of participants into the study. Because there was no evidence of a substantive difference in baseline characteristics, retention, or outcomes based on recruitment channel, the yield and cost-effectiveness of recruitment methods may be the more critical considerations in developing a feasible recruitment plan for a web-based smoking cessation intervention study.
Keywords: smoking cessation, tobacco dependence, eHealth, recruitment
1. Introduction
Delivery of behavioral interventions via the Internet has great potential to improve the reach and accessibility of evidence-based treatment for substance use disorders, including nicotine dependence (Civljak, Sheikh, Stead, & Car, 2010; Bewick et al., 2008). Smoking cessation is the focus of a rapidly growing number of web-based interventions (Civljak et al., 2010) that are now available to the estimated 13 million U.S. adults per year who search online for assistance with quitting smoking (Fox, 2006).
The burgeoning field of web-based smoking cessation research is advancing the science of eHealth interventions for tobacco use and providing a more complete picture of their promise, yet a major methodological challenge impedes scientific progress: recruitment (Danaher & Seeley, 2009). In fact, the difficulty of recruiting smokers into clinical trials of all types of cessation interventions—including web-based—has been well-documented, as have the generally low rates of utilization of any type of cessation assistance (e.g., McDonald, 1999; McClure et al., 2006). Despite success in demonstrating feasible recruitment for web-based smoking cessation studies (e.g., Brendryen, Drozd, & Kraft, 2008; Graham et al., 2011), there are many important unanswered questions: (1) How do we effectively reach smokers and engage them in these studies?; (2) What effect do these recruitment methods have on study outcome?; (3) Are some methods more cost-effective than others?
We are aware of only two prior studies (Gordon, Akers, Severson, Danaher, & Boles, 2006; Graham, Milner, Saul, & Pfaff, 2008) that have partially addressed these questions. Both studies concluded that participant characteristics differed depending on whether they were recruited using traditional approaches (e.g., standard media press releases) or web-based approaches (e.g., Google AdWords). However, with the exception of one consistent finding that web-based recruitment methods reached younger tobacco users, the two studies' comparisons of web-based and traditional recruitment methods provided conflicting results. Specifically, Graham et al. (2008) found that smokers recruited through online advertising had very mildly higher (albeit statistically significant) fractions of men, lower educational levels, and higher nicotine dependence. In contrast, Gordon et al. (2006) found the participants recruited online to be more likely to be unmarried, have attended college, and have shorter histories of tobacco use. Both studies concluded that cost-effectiveness differed considerably by recruitment channel (Gordon et al., 2006; Graham et al., 2008). Neither study examined how recruitment channel might impact follow-up data retention and treatment outcomes.
In this report, we used recruitment data from our pilot study (Bricker, Wyszynski, Comstock, & Heffner, submitted) of a web-based Acceptance and Commitment Therapy program for smoking cessation (WebQuit.org) to answer the following questions: (1) Do recruitment methods differ in their impact on participant demographics and smoking characteristics?; (2) Do follow-up data retention and treatment outcomes differ by recruitment method?; (3) What is the estimated yield and cost-effectiveness of the different recruitment methods?
2. Materials and Methods
2.1. Traditional recruitment methods
2.1.1. Standard media
Press releases about the study resulted in earned media through local television, radio, newspaper, and website news stories. We also released an abbreviated version of the press release and pre-recorded 15- and 30-second public service announcements to local and national news media outlets and newswire services. Newswire services and media outlets targeting underrepresented populations were also included (e.g., BlackPR.com).
2.1.2. Broadcast emails
Broadcast emails of the edited study press release, with a link to the study website, were sent to Pacific Northwest area health insurance agencies, Pacific Northwest and Wisconsin state health organizations, health-related listservs, and the human resource departments of major local and national employers.
2.1.3. Word of mouth
The above recruitment methods generated word of mouth referrals from friends, family, and physicians of the study participants. Because it was impractical to trace these referrals back to the original recruitment source, we treated word of mouth as a separate recruitment category.
2.2. Web-based recruitment methods
2.2.1. Medical Internet media
Press releases developed for standard media outlets were also distributed to medical media websites such as WebMD.com.
2.2.2. Google AdWords
Study ads were posted online through Google AdWords using the keywords, “quit smoking,” “smoking,” and “stop smoking.” A sample ad, which included a link to the study website, read: “New quit-smoking study. Fred Hutchinson Cancer Research Center has a new program to help you quit.”
2.2.3. Social media
Smokers were also recruited through Facebook and Twitter. The study Facebook page included a brief overview of the study, daily messages, and a link to the study website. Daily messages were posted on the Facebook page and on Twitter, such as, “Do you know anyone who is trying to quit smoking? Learn more about a new study designed to help them quit.” Facebook Ads identical to the Facebook daily messages and Google ads were initially tried, but discontinued after two weeks due to a low volume of website visitors coming from this source (only 73 unique visitors, 71 of whom briefly viewed the site and left).
2.3 Eligibility and enrollment
Individuals were eligible for this study if they: (1) were age 18 or older, (2) smoked at least five cigarettes daily for at least the past 12 months, (3) wanted to quit in the next 30 days, (4) were willing to be randomly assigned to either group, (5) resided in the U.S., (6) had at least weekly access to a high-speed Internet connection, (7) were willing and able to read in English, (8) were not participating in other smoking cessation interventions, and (9) had never used the Smokefree.gov website.
Study enrollment occurred entirely online through the study website. Participants who met eligibility criteria (n = 222) were automatically randomized to treatment and emailed a unique URL that directed them to either the WebQuit.org (ACT) or Smokefree.gov program. Three months after randomization, participants completed a follow-up survey measuring smoking outcomes. The study was reviewed and approved by the Institutional Review Board of the Fred Hutchinson Cancer Research Center.
2.4 Measures
As part of the initial eligibility screening survey, individuals were asked how they learned about the study. Fixed response options including the methods of recruitment used for the study were provided, along with an “other” option to write in responses. Six major recruitment channels were identified—standard media, broadcast email, word of mouth, medical Internet media, social media, and Google AdWords. Twenty-two “other” responses were reviewed and coded into one of these recruitment channel categories by two independent reviewers with 100% agreement.
The baseline survey assessed standard participant demographics and pertinent smoking characteristics. The primary smoking cessation outcome was self-reported thirty-day point prevalence abstinence, meaning no smoking at all in the 30 days preceding the 3-month follow-up assessment.
2.5 Statistical analysis
Analysis of variance (continuous variables) and chi-square tests (categorical variables) were used to test differences in baseline demographics by recruitment method—traditional vs. web-based. Logistic regression models were used to investigate associations between recruitment method and retention and 30-day quit status at 3-month follow-up. The model for retention was adjusted for group randomization and the quit status models were adjusted for treatment group assignment and post-randomization enrollment in a cessation program outside of the study. For 30-day quit status, we used only the evaluable study population who completed the three-month follow-up, consistent with the primary outcome measure for this pilot study. We also conducted a secondary set of analyses to examine differences in baseline demographics, retention, and 30-day quit status for each of the six recruitment channels. Cost effectiveness was calculated by dividing the total cost for each of the six recruitment channels by the number of randomized participants recruited through that channel.
3. Results
3.1 Participant demographics according to recruitment channel
As seen in Table 1, baseline demographics and smoking behaviors were similar for traditional and web-based recruitment methods, with no statistically significant differences among them (all p-values > .05). Similar results were obtained in the comparisons among the six recruitment channels (see Table A.1 in Appendix A for summary results of these secondary analyses).
Table 1.
Baseline characteristics and 3-month outcomes for web-based versus traditional recruitment.
| Web-based Recruitment (n=142) | Traditional Recruitment (n=80) | p-value* | |
|---|---|---|---|
| Baseline characteristics | |||
| Age, mean (SD) | 45.6 (13.5) | 44.0 (13.0) | 0.402 |
| Male, n (%) | 52 (37%) | 31 (39%) | 0.885 |
| Caucasian, n (%) | 131 (92%) | 73 (91%) | 0.802 |
| Hispanic, n (%) | 6 (4%) | 4 (5%) | 0.750 |
| Married, n (%) | 62 (44%) | 35 (44%) | 0.999 |
| Working, n (%) | 81 (57%) | 55 (69%) | 0.114 |
| At least some college, n (%) | 113 (80%) | 61 (76%) | 0.612 |
| First cigarette within 5 min of waking, n (%) | 58 (41%) | 32 (40%) | 0.999 |
| Smokes more than half pack/day, n (%) | 112 (79%) | 61 (76%) | 0.736 |
| Smoked 10 or more years, n (%) | 113 (80%) | 64 (80%) | 0.999 |
| Quit attempts in last 12 months, mean (SD) | 1.4 (2.1) | 1.5 (2.8) | 0.803 |
| Three-month outcomes | |||
| Data retention, n (%) | 72 (51%) | 43 (54%) | 0.665 |
| 30-day PPA, n (%) | 14 (19%) | 5 (12%) | 0.614 |
Note: Web-based recruitment category includes medical Internet, social media, and Google AdWords.
Traditional recruitment category includes standard media, broadcast email, and word of mouth.
T-tests(continuous variables) and chi-square tests (categorical variables) were used to test differences in baseline characteristics by recruitment category. The likelihood ratio test was used for the overall test of differences in data retention and 30-day point prevalence abstinence (PPA) outcomes by recruitment category.
3.2 Three-month data retention and 30-day point prevalence smoking abstinence according to recruitment channel
Table 1 shows the relationship between recruitment method and two key outcomes: data retention and 30-day point prevalence abstinence at the 3-month follow-up assessment. Traditional and web-based methods of recruitment did not differ on data retention or smoking abstinence (Table 1). In secondary analyses comparing the six recruitment channels, tests of the overall difference in retention and abstinence by recruitment channel (using standard media as the reference category) also showed no significant differences (Table A.1).
3.3 Yield and cost-effectiveness according to recruitment channel
Total cost of recruitment was $9,429.83, consisting of $3,562 in personnel cost, $1,250 in press releases, $3,320.53 for Google AdWords, $1,250 for Facebook advertisements, and $47.30 in copy charges. For personnel costs, we distributed the $822 for the principal investigator evenly across the six recruitment sources, and the $1,200 for the media officer and the $1,540 the research assistant was evenly distributed between social media and broadcast emails. The $1,250 for the press releases was evenly distributed between standard media and medical Internet media. Copy expenses were a cost for standard media recruitment.
Table 2 shows the yield and cost-effectiveness of each of the six recruitment sources. The highest-yielding methods of recruitment in terms of the percentage of enrolled participants from each source were Google AdWords, medical Internet media, and standard media. These were also the three most expensive recruitment methods. The most cost-effective of the three top-yielding methods was medical Internet media. As could be expected, the recruitment method with the least cost overall, and the greatest cost-effectiveness, was word-of-mouth. Social media was the least cost-effective. Averaging across recruitment methods, overall cost per enrolled participant was $42.48.
Table 2.
Total cost and cost per randomized subject by recruitment channel.
| Source | N (%) of randomized participants | Total cost ($) | Cost ($)/randomized |
|---|---|---|---|
| Standard media | 30 (14%) | 1409.30 | 46.98 |
| Broadcast email | 24 (11%) | 650.33 | 27.10 |
| Word of mouth | 26 (12%) | 137.00 | 5.27 |
| Medical Internet | 52 (23%) | 1362.00 | 26.19 |
| Social media | 11 (5%) | 1900.33 | 172.76 |
| Google AdWords | 79 (36%) | 3970.53 | 50.26 |
4. Discussion
Our finding that baseline demographic and smoking characteristics did not differ significantly by recruitment method are modestly discrepant with the conclusions of two prior studies on this topic, which found that use of web-based recruitment methods impacts participant characteristics (Gordon et al., 2006; Graham et al., 2008). The discrepancies between our results and those of these two prior studies may be attributable to a number of differences in study design, such as the target population (e.g., smokeless tobacco users in Gordon et al., 2006). Our study also included a broader range of web-based recruitment channels. Finally, the prior studies had very large sample sizes (N = 2,500 in Gordon et al., 2006; N = 130,000 in Graham et al., 2008), thereby yielding high power that detected modest differences. Consequently, although our results were discrepant with the two prior studies in terms of statistical significance of the differences, they concur with those studies in that they do not provide evidence of large differences among the recruitment methods. The current study builds on the Gordon et al. (2006) and Graham et al. (2008) studies by showing no relationship between recruitment method and follow-up data retention or treatment outcome. Taken together, the three studies suggest that use of web-based recruitment methods for a web-based smoking cessation trial do not substantially alter the generalizability of study findings.
The finding that Google AdWords was among of the highest-yielding methods is consistent with a prior study of a web-based intervention for smokeless tobacco users (Gordon et al., 2006). We also found that medical Internet media demonstrated relatively strong yield and cost-effectiveness, perhaps because it cost less than paid search word advertising and still targeted a population seeking health-related information on the Internet. At the same time, using a service such as Google AdWords is time efficient and requires minimal personnel resources. Our main caution about Google AdWords is that the recruitment cost per randomized participant varies depending on current competition for high value ad keywords. Nonetheless, our findings indicate that use of non-traditional channels such as Google AdWords and medical Internet media can be an efficient and cost-effective way to recruit participants into web-based smoking cessation studies. These outlets may also be useful ways to encourage smokers to seek other forms of treatment, such as tobacco quitlines (Graham et al., 2008).
Of the six recruitment channels that we used for this study, social media had the lowest cost-effectiveness. This finding suggests that what is needed next is development of more sophisticated methods of recruiting via social media in order to take advantage of the benefits of online social networks.
This study has key limitations. First, although we did not calculate statistical power because the study is a secondary analysis of pilot data, modest sample sizes for each recruitment channel limited statistical power, particularly for comparisons of rates of smoking abstinence. Additional limitations include reliance on self-report of recruitment source, which is subject to recall bias, and use of educated guessing to calculate the cost of word-of-mouth referrals, which was necessary given that this is a secondary recruitment method with no empirically-validated methods for identifying the original source. In spite of these limitations, our findings suggest that both traditional and web-based methods can be used to facilitate timely recruitment into web-based smoking cessation interventions without substantial impact on the generalizability of study findings. Yield and cost-effectiveness of recruitment methods are therefore the more critical considerations in developing a recruitment plan.
Highlights
• Use of both traditional and web-based recruitment channels facilitated timely enrollment into a web-based smoking cessation trial
• Recruitment channel was not associated with baseline participant characteristics, study retention, or treatment outcome
• Yield and cost-effectiveness differed by recruitment channel and may be the more critical considerations in developing a recruitment plan
Acknowledgements
We gratefully thank the study participants, the excellent contributions of our study staff at the Fred Hutchinson Cancer Research Center, and the helpful comments of our scientific colleague Dr. Jennifer McClure. The WebQuit study was funded by the Fred Hutchinson Cancer Research Center. Dr. Heffner's work on the project was supported by a grant from the National Institute on Drug Abuse (#K23DA026517).
Funding: This study was funded by the Fred Hutchinson Cancer Research Center.
Role of funding source The Fred Hutchinson Cancer Research Center played no role in the design of the study; the collection, analysis, or interpretation of the data; the writing of this report; and the decision to submit this manuscript for publication.
Table A.1.
Baseline characteristics and 3-month outcomes by recruitment channel.
| Overall (n=222) |
Standard Media (n=30) |
Broadcast (n=24) |
Word of Mouth (n=26) |
Medical Internet (n=52) |
Social Media (n=11) |
Google AdWords (n=79) |
p- value* |
|
|---|---|---|---|---|---|---|---|---|
| Baseline characteristics | ||||||||
| Age, mean (SD) | 45.0 (13.3) | 45.2 (11.4) | 46.0 (10.0) | 40.9 (16.7) | 46.8 (13.6) | 39.5 (16.5) | 45.7 (13.0) | 0.343 |
| Male, n (%) | 83 (38%) | 12 (40%) | 9 (38%) | 10 (38%) | 17 (33%) | 6 (55%) | 29 (37%) | 0.853 |
| Caucasian, n (%) | 204 (92%) | 28 (93%) | 22 (92%) | 23 (88%) | 49 (94%) | 11 (100%) | 71 (90%) | 0.809 |
| Hispanic, n (%) | 10 (5%) | 2 (7%) | 1 (4%) | 1 (4%) | 1 (2%) | 1 (9%) | 4 (5%) | 0.881 |
| Married, n (%) | 97 (44%) | 15 (50%) | 13 (54%) | 7 (27%) | 21 (40%) | 5 (45%) | 36 (46%) | 0.430 |
| Working, n (%) | 136 (61%) | 21 (70%) | 18 (75%) | 16 (62%) | 33 (63%) | 4 (36%) | 44 (56%) | 0.229 |
| At least some college, n (%) |
174 (78%) | 19 (63%) | 19 (79%) | 23 (88%) | 44 (85%) | 11 (100%) | 58 (73%) | 0.052 |
| First cigarette within 5 min of waking, n (%) |
90 (41%) | 14 (47%) | 9 (38%) | 9 (35%) | 21 (40%) | 4 (36%) | 33 (42%) | 0.957 |
| Smokes more than half pack/day, n (%) |
173 (78%) | 23 (77%) | 18 (75%) | 20 (77%) | 43 (83%) | 7 (64%) | 62 (78%) | 0.825 |
| Smoked 10 or more years, n (%) |
177 (80%) | 28 (93%) | 20 (83%) | 16 (62%) | 39 (75%) | 8 (73%) | 66 (84%) | 0.057 |
| Quit attempts in last 12 months, mean (SD) |
1.5 (2.4) | 0.8 (1.3) | 1.8 (3.3) | 2.1 (3.4) | 1.3 (1.5) | 1.9 (3.5) | 1.5 (2.1) | 0.331 |
| Three-month outcomes | ||||||||
| Data retention, n (%) | 115 (52%) | 16 (53%) | 11 (46%) | 16 (62%) | 29 (56%) | 7 (64%) | 36 (46%) | 0.602 |
| 30-day PPA, n (%) | 19 (17%) | 2 (13%) | 2 (18%) | 1 (6%) | 6 (21%) | 3 (43%) | 5 (14%) | 0.843 |
Analysis of variance (continuous variables) and chi-square tests (categorical variables) were used to test differences in baseline characteristics by recruitment medium. The likelihood ratio test was used for the overall test of differences in data retention and 30-day point prevalence abstinence (PPA) outcomes by recruitment method.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Contributors Dr. Heffner conducted the literature search and wrote the first draft of the manuscript along with Mr. Wyszynski, and she wrote all subsequent drafts of the manuscript. Mr. Comstock and Ms. Mercer conducted the statistical analyses. Dr. Bricker designed the study, planned the secondary analysis, and edited all drafts of the manuscript. All authors contributed to and approved the final version of the manuscript.
Conflict of interest statement Dr. Heffner served as a consultant for Pfizer from 2010–2011. None of the other authors have competing interests to disclose.
References
- Bewick BM, Trusler K, Barkham M, Hill AJ, Cahill J, Mulhern B. The effectiveness of web-based interventions designed to decrease alcohol consumption—a systematic review. Preventive Medicine. 2008;47:17–26. doi: 10.1016/j.ypmed.2008.01.005. [DOI] [PubMed] [Google Scholar]
- Brendryen H, Drozd F, Kraft P. A digital smoking cessation program delivered through Internet and cell phone without nicotine replacement (happy ending): randomized controlled trial. Journal of Medical Internet Research. 2008;10:e51. doi: 10.2196/jmir.1005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bricker J, Wyszynski C, Comstock B, Heffner JL. Pilot randomized controlled trial of web- based acceptance and commitment therapy for smoking cessation. Manuscript submitted for publication. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Civljak M, Sheikh A, Stead LF, Car J. Internet-based interventions for smoking cessation. Cochrane Database of Systematic Reviews. 2010;9:CD007078. doi: 10.1002/14651858.CD007078.pub3. [DOI] [PubMed] [Google Scholar]
- Danaher BG, Seeley JR. Methodological issues in research on web-based behavioral interventions. Annals of Behavioral Medicine. 2009;38:28–39. doi: 10.1007/s12160-009-9129-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Fox S. Online health search 2006. 2006 Retrieved December 10, 2012, from http://www.pewinternet.org/Reports/2006/Online-Health-Search-2006/01-Summary-of-Findings/Summary-of-Findings.aspx.
- Gordon JS, Akers L, Severson HH, Danaher BG, Boles SM. Successful participant recruitment strategies for an online smokeless tobacco cessation program. Nicotine & Tobacco Research. 2006;8:S35–S41. doi: 10.1080/14622200601039014. [DOI] [PubMed] [Google Scholar]
- Graham AL, Cobb NK, Papandonatos GD, Moreno JL, Kang H, Tinkelman DG, et al. A randomized trial of Internet and telephone treatment for smoking cessation. Archives of Internal Medicine. 2011;171:46–53. doi: 10.1001/archinternmed.2010.451. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Graham AL, Milner P, Saul JE, Pfaff L. Online advertising as a public health and recruitment tool. Comparison of different media campaigns to increase demand for smoking cessation interventions. Journal of Medical Internet Research. 2008;10:e50. doi: 10.2196/jmir.1001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- McClure JB, Greene SM, Wiese C, Johnson KE, Alexander G, Strecher V. Interest in an online smoking cessation program and effective recruitment strategies: results from Project Quit. Journal of Medical Internet Research. 2006;8:e14. doi: 10.2196/jmir.8.3.e14. [DOI] [PMC free article] [PubMed] [Google Scholar]
- McDonald PW. Population-based recruitment for quit-smoking programs: an analytic review of communication variables. Preventive Medicine. 1999;28:545–557. doi: 10.1006/pmed.1998.0479. [DOI] [PubMed] [Google Scholar]