Abstract
Objectives
To compare 4 on-line and off-line recruiting methods.
Methods
Young adult smokers (n=3353) were recruited to a trial comparing smoking cessation services with an on-line health risk assessment (HRA), on-line ads, off-line materials, and quit line screening.
Results
On-line ads (n=1426; $41.35) and off-line materials recruited the most smokers (n=1341; $56.23) for the lowest cost. Quit line screening was more expensive (n=189; $133.61), but enrollees used cessation services the most (34%-82%). On-line HRA was least successful and most costly (n=397; $630.85) but had the highest follow-up (45%-55%).
Conclusions
On-line ads and off-line materials were most effective and cost-effective methods.
Keywords: recruitment, smokers, smoking cessation, Internet
INTRODUCTION
Recruitment to behavioral research trials continues to be a challenge. Methods such as print and broadcast advertising, direct mail, and telephone random-digit dialing can be expensive1 (eg, $36 to $597 per enrollee)2 and all methods risk losing potential participants during screening and consenting, resulting in low enrollment rates (eg, 3%-4%3).4 This analysis compares the success rates and costs of 4 strategies for recruiting young adults smokers (aged 18-30 years) to a smoking cessation trial employing on-line and off-line methods.
The Internet has a large potential for recruiting research participants. Most U.S. adults (77%) are on-line, including 90% of adults 18-29.5 Internet advertising has been shown to be more successful at recruiting research participants than off-line methods (eg, television and radio ads, direct mailings, billboards),6 and automated systems in Internet recruiting are potentially less costly than off-line methods especially those that require contact with study personnel.4
However, there can be shortcomings with Internet recruitment. First, this method has recruited samples with different demographic characteristics than off-line methods,7,8 suggesting that the Internet reaches a different population of adults. Some of these differences may have declined in recent years now that most adults under age 65 are on-line.9 Second, the Internet has achieved very low enrollment rates (0.24% to 7%)4,8,10,11 although low enrollment rates are not unique to on-line recruiting; off-line enrollment methods recruit only 2%-4% of eligible participants.3,12 Low enrollment in general has been attributed to preferences of participants for the treatment arm of a study over a control group13 and unwillingness to submit to randomization.14 Finally, samples recruited over the Internet may have selection biases. For instance, one study recruiting smokers through on-line methods obtained samples that were more likely to be female, married, young, living with others, and have higher readiness to quit than non-participants15 while another recruited smokers more likely to be non-daily smokers, exhibit high motivation to quit, and use alcohol and marijuana than reported in other surveys.8 Additionally, a sample of Korean Americans were more likely to be recent immigrants, college educated, male and in poor health compared to the larger Korean American population.11 Studies of volunteerism indicate that volunteers in general are diverse,16 but tend to be female,17-19 young,19 single,17 and high in empathy and agreeableness.20
On-line and off-line recruitment methods may recruit participants who vary in their motivation to complete study requirements. The direct, often face-to-face, contact with study personnel when recruitment occurs off-line in clinics or over the telephone may achieve more public commitment than the automated, impersonal contact over the Internet. Public commitment in general elevates compliance,21,22 which in the context of a research trial should result in more participants following study protocols both in terms of using experimental interventions and completing posttests.23
Four different strategies of recruiting smokers and referring them to an on-line enrollment system were used in a randomized trial evaluating an on-line smoking cessation intervention. The study population was young adult smokers aged 18-30 years, the adult cohort with the highest smoking prevalence. Strategies included on-line and off-line methods, some that explicitly advertised the study and others that referred smokers who had completed an on-line health risk assessment (HRA) or called a telephone quit line. The recruitment yield and costs of each recruitment method were compared. It was likely that these recruiting methods reached different portions of the young adult smoker population so differences in demographics and smoking status were examined, as well. Finally, the motivation of smokers recruited by each method to use cessation services and complete posttests were compared. The results are useful for guiding selection of recruitment strategies in future studies.
METHODS
From September 2007 to October 2009, 3353 young adults (18-30 years old) who were current smokers and interested in quitting were recruited to a trial comparing three smoking cessation services, using 4 different recruitment methods. Initially, an on-line HRA was created and deployed on college campuses in partnership with the BACCHUS Network of peer education chapters. Smokers interested in joining the study enrolled on-line through a registration page attached to the on-line HRA. After 12 months, it was evident that the on-line HRA was not yielding enough smokers to obtain the sample required for the trial, so 3 strategies that directly promoted the trial were deployed – off-line promotions, on-line advertisements and telephone quit line screening. These 3 additional strategies referred smokers to a direct-to-study enrollment website. Both the on-line HRA and the direct-to-study website were maintained by the research group on secure web servers. The 4 recruitment strategies ran for different periods of time due the sequential deployment and time needed to prepare and launch them (eg, quit line screening was slow to launch because it required securing approval from several state quit lines).
Trial Design
The trial design was a randomized 3-group pretest-posttest design. An on-line smoking cessation website, Real e Quit (n=1,092 smokers), was evaluated for effectiveness compared to a telephone quit line delivered by one of 2 providers (n=1160) and a self-help booklet (ie, Clearing the Air from the U.S. National Cancer Institute; n=1,101). Real e Quit contained an expert system, Quit Coach,24-26 a computer program that provided tailored smoking cessation counseling. Participants were sent on-line reminders to use the assigned intervention and re-contacted at 12 weeks and 26 weeks post-randomization for follow up surveys. A free 2-week supply of nicotine patches was made available. All recruitment and trial procedures were approved by the Western Institutional Review Board and collaborating research organizations’ institutional review boards (IRBs).
Recruiting Methods
On-line Health Risk Assessment (HRA)
The on-line HRA was used as a health education tool by the BACCHUS Network on college campuses. It consisted of approximately 28 questions (depending on skip patterns) on 7 common health risk behaviors for young adults: alcohol, tobacco, marijuana, nutrition, physical activity, mental health, and sexual health. After completing the HRA, users were given a detailed personal feedback report on ways to reduce these risk behaviors.
Users who indicated that they were smokers (ie, smoked cigarettes every day or some days, or that they smoked at least once in the past 30 days), aged 18-30, and lived in the continental United States received a web screen with an invitation to join the cessation trial before receiving their personalized feedback report. If they clicked on a link saying they were interested in participating, the consent and enrollment forms were displayed, after which the baseline survey was administered and participants were randomized to study condition by on-line routines.
The HRA was promoted to campus health and student life professionals in partnership with the BACCHUS Network, a network of student chapters on over 900 campuses in the United States that works to improve student health through peer education. The HRA was promoted through presentations at BACCHUS Network conferences, web-based trainings, direct mailings, and articles in BACCHUS Network newsletters. BACCHUS Network chapters self-selected to register to use the on-line HRA. Participating chapters were given access to the administrative tools and the names of their school were populated in a drop-down menu where students could identify the community college, college, or university they attended. All responses to the on-line HRA were stored in an anonymous database. Registered chapters could access data reports and files with responses from students from their campus only after 125 students had completed it, to maintain student anonymity. They could add links to local resources in the feedback report.
The HRA was promoted by various methods as a health education tool by BACCHUS Network chapters on 165 campuses across the United States from September 2007 to October 2009 (26 months). Collateral materials were distributed to the chapters for promoting the on-line HRA. Chapters received incentives for use of the on-line HRA - $50 when 250 participants completed the on-line HRA and $100 of health education materials from BACCHUS Network when 500 participants completed the on-line HRA. Campus personnel did not promote the smoking cessation trial nor mention it in promotions of the on-line HRA, per requirements set forth by the IRBs.
Recruiting Methods Using On-line Direct-to-Study Enrollment System
On-line direct-to-study system
The on-line direct-to-study enrollment system was created to allow smokers who were interested in quitting to directly enter the trial without first completing the on-line HRA. Participants were consented and enrolled on-line (using the same consent and enrollment forms as the on-line HRA) and completed a pretest survey on-line that was composed of both the health risk questions in the HRA and the pretest questions. After completing the survey, participants were randomized to one of the 3 trial arms.
Off-line promotional materials
Participants were recruited to the direct-to-study enrollment system, using 3 promotional strategies including off-line promotions, on-line advertisements and telephone quit line screening. Off-line items such as posters, business cards and newsletter inserts were created with promotional messages and the URL for the direct-to-study enrollment system. They were provided to 280 health departments, clinics, employers, and union training centers, who displayed them as they wished from October 2008 to October 2009 (12 months). Similar off-line promotional materials were posted or distributed around 5 college or university towns, in locations nearby but off campus, by individuals hired for this purpose. The 5 towns were selected because they were located in states with high smoking rates.
On-line advertisements
On-line advertisements were placed on the Google Adwords system from January 2009 to October 2009 (9 months). Advertisements were linked to relevant search keywords such as smoking cessation, quit smoking, help quitting smoking, and nicotine patches and on websites, blogs, and other on-line sites that were paid by Google Adwords to display its advertisements. Advertisements had short promotional messages and provided a link to the direct-to-study enrollment system.
Quit line screening
Finally, the project contracted with a telephone quit line to screen and refer eligible smokers from June 2009 to September 2009 (4 months). When calling for services from 5 state quit lines, age-eligible young adult smokers were invited to participate in the trial by the telephone counselors. Contact information for interested smokers was provided to the research staff who emailed them a link to the direct-to-study enrollment system.
Yield Rate Analysis
The number of participants enrolled in the trial was recorded for all recruitment methods. In the on-line HRA, the number of users who completed the HRA, were issued study invitations because they smoked, and enrolled in the trial was recorded. On-line landing pages with separate URLs were created for each method of referring participants to the direct-to-study enrollment system to record where each participant who visited the system originated. Yield rate was calculated for all recruitment methods except off-line promotions by using as the denominator the number of users who completed the on-line HRA and indicated they smoked, number of clicks on the on-line ads, and the number of smokers who called the quit line, were age-eligible, and were offered the trial. It was not possible to calculate a yield rate for off-line promotions of the direct-to-study system because the number of eligible smokers who were exposed to those advertisements was unknown.
Sample Analysis Methods
The smoking status and demographic characteristics of the sample, use of cessation services, and follow-up (ie, posttest) rates were compared among the 4 recruitment methods, using chi-square, Wilcoxon-Mann-Whitney (z-score) tests, and analysis of covariance. Demographics and Internet use were included as covariates for comparisons on smoking status, use of cessation services, and follow-up rates, if they demonstrated significant univariate correlations with these variables. Treatment group was also included as a covariate in the comparison on use of cessation services and follow-up rates. Additionally, demographic information and smoking status was available from the on-line HRA for all smokers identified, so a comparison was performed between smokers who enrolled and did not enroll in the trial on these measures. All comparisons were performed at P=0.05 (2-tailed).
Cost Analysis Methods
Costs associated with the recruitment procedures were tracked and summarized by project staff, with assistance from the quit line service provider. These included costs associated with investigator and staff time with all 4 methods; travel, meeting expenses, press releases, and preparing, printing and postage for materials to promote on-line HRA, telephone and web conferences to train BACCHUS Network chapters to use the on-line HRA; preparing, printing, and postage for materials for off-line promotion; Google Adword charges; and costs charged by the telephone quit line provider. Our objective in evaluating the cost implications of the various recruitment methods was to compare the relative costs and effects (ie, numbers of enrollees) of each method. Although we did not view this comparison as a cost-effectiveness analysis, strictly speaking, we did adopt the incremental nature of such an analysis as we framed our results in terms of efficiency. In other words, our approach was to sum the total costs of each recruitment method and use that value to generate a cost per enrollee, both in total and per month (to standardize somewhat for the varying length of time each method was in use). These amounts were not discounted.
The costs associated with the on-line HRA included expenses for launching it, training BACCHUS Network chapters on how to use it, and promoting it on college campuses to create an audience for it. Such expenses were not incurred by the project in the other 3 recruiting methods because Google Adwords, state tobacco control programs, and telephone quit lines already existed and had reached substantial audiences. Thus, we also modeled the cost of recruiting smokers with the on-line HRA, removing the costs associated with launching, training, and promoting it, to estimate the costs if the project had been able to rely on an already-existing on-line HRA.
RESULTS
Yield Rates
Overall, the on-line HRA yielded 397 smokers (16 per month) who enrolled in the trial. Within the direct-to-study system, off-line promotion yielded 1341 smokers (112 per month), on-line advertisements, 1426 smokers (158 per month), and telephone quit line screening, 189 smokers (47 per month). If used over an entire year (ie, 12 months), the on-line HRA would have recruited 192 smokers, the off-line promotion, 1344 smokers, the on-line advertisements, 1896 smokers, and the telephone quit line, 564 smokers, as used in this trial.
Yield rates varied by recruitment method. The on-line HRA identified 5677 smokers out of 25,961 users (22%) who completed the on-line HRA, and 8% of smokers enrolled in the trial. For the on-line advertisements in the direct-to-study system, the 1426 participants who enrolled represented 10% of clicks on an on-line advertisement. A total of 836 smokers calling the quit line were offered the trial, 436 expressed an interest in participating and received the URL for the direct-to-study website; and 23% of smokers enrolled (43% of smokers expressing an interest).
Differences in Smoking Status by Recruitment Methods
The HRA recruited a sample of smokers quite different from those recruited using the other strategies (Table 1). The HRA sample contained lighter smokers, with more smoking on some days (not every day), on fewer days in the past month, and fewer cigarettes per day compared to other recruitment methods. This group also had smokers who appeared to be the least nicotine dependent, as indicated by waiting longer to smoke the first cigarette of the day and finding it less difficult not to smoke when friends smoked, compared to smokers in direct-to-study samples. Nearly all smokers recruited through direct-to-study methods smoked every day but those from on-line advertisements and quit line screening were the heaviest smokers.
Table 1.
Differences in Tobacco Use for Participants by Recruitment Method
| Online HRA | Off-line Promotions | Online Advertisements | Quit-Line Screening | P | |
|---|---|---|---|---|---|
| N | 397 | 1341 | 1426 | 189 | |
| Smoke cigarettes: | |||||
| Smoked in the past | 14%a | 1%b | <1%c | 1%b | <0.0011,3,4,5 |
| Some days | 25% | 3% | 1% | 6% | |
| Every day | 61% | 96% | 98% | 93% | |
| Number of days smoked in past 30 days (mean) | 15.6a | 17.3a | 23.1b | 17.5a | 0.049 |
| Number of cigarettes smoked per day (mean) | 9.4a | 17.9b | 19.2c | 18.1bc | < 0.0011,2,3,4,5,6 |
| Smoke first cigarette after waking up: | |||||
| Less than 5 min | 12%a | 38%b | 43%c | 40%d | <0.0011,4,,5,6 |
| 5 – 30 min | 28% | 43% | 42% | 35% | |
| 31 – 60 min | 19% | 12% | 11% | 16% | |
| 60 min or longer | 41% | 7% | 4% | 9% | |
| Find it difficult to keep from smoking when friend(s) is smoking: | |||||
| No – never difficult | 5%a | 1%b | 1%c | 0%bc | <0.0013 |
| No – usually not difficult | 11% | 5% | 4% | 5% | |
| Don't know | 4% | 5% | 6% | 6% | |
| Yes – somewhat difficult | 32% | 29% | 27% | 27% | |
| Yes – very difficult | 48% | 60% | 62% | 62% | |
| Stage of readiness to quit: | |||||
| Want to keep smoking | <1%a | <1%b | <1%c | 2%c | < 0.0011,2,3,4,5,6 |
| Open to possibility of quitting | 13% | 8% | 8% | 4% | |
| Like to quit but don't have immediate plans | 26% | 14% | 10% | 6% | |
| Planning to quit but haven't decided when | 50% | 54% | 49% | 54% | |
| Set a quit date | 9% | 20% | 29% | 30% | |
| Have quit smoking | 1% | 3% | 3% | 4% | |
| Made quit attempt in past year and succeeded in not smoking for 24 hours | 78%a | 57%b | 57%c | 59%bc | <0.0011,3,4,5 |
| Longest time went without smoking in past year: | |||||
| Less than 24 hours | 10%a | 34%b | 39%c | 30%b | <0.0011,3,4,5,6 |
| 1 day | 9% | 14% | 16% | 16% | |
| 2 days | 7% | 14% | 14% | 10% | |
| 3-6 days | 15% | 15% | 12% | 16% | |
| 7 days to 1 month | 23% | 12% | 11% | 11% | |
| 1-3 months | 16% | 8% | 5% | 12% | |
| More than 3 months | 20% | 3% | 3% | 5% | |
| Use smokeless tobacco: | |||||
| Never | 74%a | 75%ab | 81%c | 69%ad | 0.0021,2,3,4 |
| Used in past | 17% | 18% | 14% | 17% | |
| Some days | 6% | 5% | 4% | 11% | |
| Every day | 3% | 2% | 1% | 3% |
Note. Statistically significant contrasts (P<.05) between groups indicated by different letters; P-value reported for tests adjusted for age (1), gender (2), race/ethnicity (3), education (4), enrolled in college (5), employed (6), and Internet use (7).
Quitting behavior by the smokers enrolled in the trial also varied by recruitment method (Table 1). More smokers recruited through the 3 direct-to-study methods appeared to be taking action to quit smoking by setting a quit date, while more smokers from the HRA indicated they would like to quit but did not have immediate plans to do so. Still, more HRA smokers had made an attempt to quit in the past year and had not smoked for longer periods than smokers enrolled through the direct-to-study methods. Finally, more of the smokers recruited from the quit line were also using smokeless tobacco, while fewer smokers enrolled from off-line promotions and on-line advertisements were also using smokeless tobacco products.
Differences in Demographics by Recruitment Method
Recruitment methods also enrolled smokers that differed on demographic characteristics (Table 2). Smokers recruited through the HRA were far more likely to have completed some college education, while those recruited from off-line promotions and on-line advertisements were more likely to have a high school education or less. In addition, smokers recruited using the HRA were the youngest and most ethnically diverse. They were also most likely to be employed while smokers enrolled through off-line promotions were least likely to be working. Off-line promotions also recruited the fewest female smokers.
Table 2.
Differences in Demographics by Recruitment Method
| Online HRA | Off-line Promotions | Online Advertisements | Quit-Line Screening | P | |
|---|---|---|---|---|---|
| N | 397 | 1341 | 1426 | 189 | |
| Age (Mean in years) | 22.5a | 25.1b | 25.3b | 25.1b | <0.001 |
| Sex: Female | 67%a | 64%ab | 66%a | 55%c | 0.022 |
| Race: Non-Hispanic White | 78%a | 84%b | 85%c | 80%abc | 0.008 |
| Education: | |||||
| High school or less | 23%a | 47%b | 47%c | 36%b | <0.001 |
| Some college or vocational education | |||||
| College graduate or postgraduate | 61% | 40% | 40% | 49% | |
| 16% | 13% | 13% | 15% | ||
| Enrolled in college | 83%a | 26%b | 23%c | 28%c | <0.001 |
| Employed outside home | 68%a | 61%b | 62%b | 53%c | 0.002 |
Note. Statistically significant contrasts (P<.05) between groups indicated by different letters.
Comparison of Smokers Enrolled versus Not Enrolled Through HRA
Differences also emerged in the type of smoker likely to enroll in the trial when invited from the on-line HRA. More female (9%) and regular smokers (13% of everyday smokers) accepted the trial invitation than male (6%, χ2 =14.22, P=.0002) and light smokers (5% of non-daily smokers, χ2 =90.62, P<.0001). Those who enrolled were older (M=22.5) than those who did not (M=21.5, Wilcoxon-Mann-Whitney z=5.33, P<.0001).
Differences in Use of Cessation Services and Follow-up Rates by Recruitment Method
Smokers recruited from quit line screening were most likely to use the assigned cessation service (Table 3) while smokers recruited through the on-line HRA used the assigned cessation service the least. By contrast, smokers recruited by the on-line HRA were most likely to complete the follow-up posttest. Those enrolled by the off-line promotions and quit line screening were least likely to complete the posttests (Table 3).
Table 3.
Differences in Use of Cessation Services and Follow-up Rates by Recruitment Method
| Online HRA | Off-line Promotions | Online Advertisements | Quit-Line Screening | P | |
|---|---|---|---|---|---|
| N | 397 | 1341 | 1426 | 189 | |
| Use of Study Cessation Services | |||||
| Cessation Web Site (used expert counseling system)* | 13%a | 30%b | 27%bc | 38%c | 0.00191 |
| Telephone Quit Line (accepted first counseling call)* | 10%a | 22%b | 22%c | 82%d | <0.0015,6 |
| Self-help Booklet (downloaded and read or looked into booklet)* | 17%a | 23%b | 24%bc | 34%c | 0.0372,4 |
| Use of any cessation service** | 13%a | 25%b | 24%b | 49%c | <0.0014 |
| Follow-up Rates | |||||
| Completed 12-week posttest** | 55%a | 43%a | 41%b | 49%ab | 0.0304,5,7 |
| Completed 26-week posttest** | 45%a | 34%a | 31%b | 40%a | 0.0154,5,7 |
Note. Statistically significant contrasts (P<.05) between groups indicated by different letters.
Comparison adjusted for demographics – age (1), gender (2), race/ethnicity (3), education (4), enrolled in college (5), employed (6), and Internet use (7).
Comparison adjusted for demographics – for age (1), gender (2), race/ethnicity (3), education (4), enrolled in college (5), employed (6), and Internet use (7) – and experimental group assignment (Web site, quit line, or booklet).
Cost Comparison of Recruitment Methods
Table 4 summarizes the results of the cost comparison. On either a total or per-month cost basis, on-line ads dominated all other methods, including off-line promotions: Google ads were less expensive and more effective (in terms of generating enrollees) than any other recruitment method used. The on-line HRA was by far the most expensive recruitment method in terms of both total cost and cost per enrollee; however, it was similar in cost per month per enrolled smoker by the quit line screening. Removing costs for promoting the on-line HRA on the college campuses reduced the cost per enrolled smoker to $410.28 ($16.41 monthly cost per enrolled smoker). By comparison, on-line ads recruited more than 3 times as many smokers, in less than half the time, and at only 6% the cost of the on-line HRA, despite the difference in implementation periods. Off-line promotions were just slightly more expensive than on-line ads and both on-line ads and off-line promotions had similar low cost per month per enrolled smoker. Use of telephone quit line screening lasted only 4 months, but its cost per enrollee was higher than on-line ads and off-line promotions. Its total cost per enrolled smoker was substantially lower than the on-line HRA although its cost per month per enrolled smoker was slightly higher than the on-line HRA.
Table 4.
Total Cost and Cost per Enrollee by Recruitment Method
| Online HRA | Off-line Promotions | Online Advertisements | Quit-Line Screening | Total | |
|---|---|---|---|---|---|
| Total costs | $250,446.08 | $75,399.52 | $58,972.03 | $25,253.30 | $410,070.93 |
| Smokers enrolled in trial | 397 | 1,341 | 1,426 | 189 | 3,355 |
| Cost per enrolled smoker | $630.85 | $56.23 | $41.35 | $133.61 | $122.23 |
| Number of months recruited smokers | 25 | 12 | 9 | 4 | 25 |
| Months of use | Sept 2007-Oct 2009 | Oct 2008-Oct 2009 | Jan 2009-Oct 2009 | Jun 2009-Oct 2009 | |
| Smokers enrolled per month | 15.9 | 111.8 | 158.4 | 47.2 | 134.2 |
| Monthly cost per enrolled smoker | $25.23 | $4.68 | $4.59 | $33.40 | $4.89 |
Discussion
Using a combination of 4 recruitment strategies, over 3300 young adult smokers were enrolled in a randomized trial comparing smoking cessation services. The 3 enrollment methods for which we could calculate the proportion of smokers who enrolled (on-line HRA, on-line advertisements, and telephone quit line screening) all had higher yield rates (8% to 23%) than reported in the literature (below 7% in past studies3,4,8,10-12). Still, yield rates were relatively low and the large majority of smokers reached with invitations or advertisements for the trial chose not to participate. On-line advertisements using Google Adwords recruited the most young adult smokers to the trial, both in total and rate per month. Still, the low yield rate showed that on-line advertising methods need to reach a very large number of potential study participants to be successful. Fortunately, Google's popularity (and market dominance) as a search engine and its provision of on-line advertisements to many other websites likely placed advertisements for the trial in front of a very large number of young adult smokers nationwide (95% of American adults under 30 used the Internet in 2010),9 contributing to its lowest cost per enrollee. On-line advertising is likely to reach a large majority of most age cohorts in America (78% or more of adults under 65 were on-line in 2010), with the exception of adults age 65 or older (only 42% of whom were on-line in 2010).9
Other projects have also benefited from the extensive reach of Google's on-line advertising system4,8,11 and at similar costs. In one study, young adults smokers were recruited for $42.77 each for just a one-time survey,8 not a randomized trial with cessation treatment and follow-up. A second study paid $66.50 per enrollee to recruit Korean Americans smokers to a cessation trial;11 the specialized and smaller population probably increased costs compared to those experienced in the current trial that recruited from a more general population, albeit younger. However, a third study on a smokeless tobacco cessation program reported an even lower cost per enrollee ($6.70) with Google ads.1 Notably, the $41 per enrollee with Google advertising in the current trial was on par with, or lower than, costs incurred by this third study for enrolling smokeless tobacco users through other means (ie, direct mail to previous study participants [$36] or purchased lists [$597], earned media [$92], or newspaper display ads [$115]). Other on-line systems with large user populations such as Facebook might be equally successful. Smaller, more closed on-line systems, such as campus or business email lists, may be too small to recruit even modest sized samples with on-line advertising.
It is interesting that off-line promotions yielded almost as many young adult smokers as on-line advertising, especially since smokers needed to remember the URL for the enrollment website and visit later. Mobile Internet platforms on smart phones and tablet computers are proliferating and may elevate the success of future off-line promotions by providing direct access to on-line enrollment sites wherever smokers are located. Especially helpful might be the advent of QR (Quick Response) codes. These specialized bar codes can be printed on promotional materials and when read by the smart phones’ and tablet computers’ cameras will take users directly to a study's enrollment website on their smart phone or tablet computer. We did not track exactly how the printed promotional materials were used by state and local tobacco programs, employers, clinics, and other groups, so we do not know the most effective methods for leveraging off-line promotional materials for this purpose. Still, this is a relatively cheap and effective recruiting method worth considering for future trials. It is notable that the costs incurred by the organizations that posted or distributed the off-line promotions were not included in this analysis because they performed this work free of charge to the project in the normal course of their tobacco control activities, so the actual cost of off-line promotions is likely higher in circumstances where project budgets must support these posting and distribution activities.
Telephone quit line screening has potential as a recruiting method, too. Even though it enrolled fewer smokers per month at a much higher cost than on-line advertising and off-line promotions, a relatively large proportion of smokers identified by this method decided to enroll and used the assigned cessation service. Many smokers contacting quit lines are already motivated to quit and this increased motivation should make them more willing to participate. The success of quit line screening may be especially notable given that it was operational only during summer months when smokers may be less likely to volunteer due to competing demands from vacations and other summer activities and more likely to quit at New Years or other times such as the Christian season of Lent. Telephone quit line screening could be equally or more successful recruiting older adult smokers, since a larger proportion of quit line callers are over age 3027 and available for recruitment.
The on-line HRA recruited the fewest smokers, perhaps because it reached primarily college students. Many college students who smoke do so infrequently, experience less nicotine addiction, believe that they can quit when they want to, and expect to quit before they graduate.28,29 Hence, they may feel that they have little need for smoking cessation services. Also, the HRA addressed many other risk behaviors in addition to smoking, so students may have completed it to obtain advice about improving behaviors other than smoking and were less interested in the trial. Still, this enrollment rate was very low and far lower than the rate of the other high-cost referral method – the telephone quit line – suggesting that the HRA reached relatively unmotivated smokers, not because it was a referral method but because smokers in college may be unwilling to enroll in a trial providing services to support quitting.
Alternative recruitment methods yielded different samples of smokers. The on-line HRA recruited lighter smokers who experienced less nicotine dependence; this was not surprising, given the younger age and college enrollment status of this sample. However, smokers who accepted the invitation in the HRA tended to be female, more regular smokers, and older than smokers who declined. Whether this is true of smokers recruited through the other methods is unknown. On-line HRA smokers were also less ready to make a quit attempt although more had made a quit attempt in the past year. The quit line screening method yielded more unemployed and male smokers. The larger number of males may explain the larger proportion of participants reporting current smokeless tobacco use (dual users) in this group. This is important since dual users often have a more difficult time quitting and the proportion of dual users in a treatment group can affect the cessation outcomes.
The various recruitment methods also produced samples that responded differently to the research protocols. Smokers from quit line screening were probably very interested in obtaining cessation services and thus used the assigned cessation service the most, across all 3 types. Smokers from the on-line HRA used the assigned cessation service the least, perhaps because they were lighter smokers, but follow-up was most successful among these smokers. More of them were enrolled in college and, as students, spend most of their time connected to the Internet and thus possibly more responsive to email invitations. They may also have more experience participating in research and be generally accustomed to filling out on-line surveys as part of their academic routine. Our team improved compliance with research protocols in a subsequent smoking cessation trial by adding a telephone contact and screening by study personnel before enrolling and consenting interested smokers, possibly because it obtained more commitment from participants to fulfill study requirements and eliminated less motivated participants.
The findings presented here are limited by being conducted in a single study, with unique characteristics, eg, testing 3 different cessation services, recruiting only adults under age 30, and being conducted in the United States. Younger adults may be more technology savvy than older smokers and respond better to on-line recruitment. Recruiting with the on-line HRA, on-line advertisements, and off-line promotions occurred during the 2007-08 economic downturn and the 2008 federal cigarette tax increase, both of which may have motivated more smokers to quit and thus elevated interest in enrolling in a trial that provided 2 weeks of free nicotine patches. There may be seasonality effects in the on-line advertising and telephone quit line, but the on-line HRA and off-line promotions occurred for at least 12-months, reducing seasonality differences between them. Still, recruitment was performed nationwide, and the sample was geographically well-represented, which increased generalizability.
Some costs associated with recruitment methods are variable. Up-front costs of providing promotional materials, creating advertisements, and establishing screening protocols are lower per enrollee when amortized over longer recruitment periods. Time periods using each type of recruiting methods varied so some costs may not be accurately amortized. Also, bid limits for on-line advertisements can be increased to obtain more impressions, potentially reducing costs if click-through rates increase. Advertising on-line for smoking cessation services and activities is relatively expensive and we did not have a budget that allowed us to spend enough to be listed high on the search results page. Recruiting from a larger number of quit lines could have yielded more smokers. This also might reduce per smoker costs, because a large proportion of the cost of using quit line screening was the time and effort involved in interfacing with the quit line and study staff time for processing referred smokers. Cost per enrolled smoker would be lower if we had used an existing on-line HRA, rather than creating a new one and devoting a substantial amount of funds to promoting its use on college campuses. Still, the recruiting costs for the HRA remained high even when promotional costs were removed from the model. It also recruited far fewer smokers per month than the other 3 methods, suggesting that an existing HRA on college campuses might not be a good option for a smoking cessation trial.
Recruitment to tobacco cessation research trials continues to be challenging. Selection of recruitment methods needs to balance yield rates, cost, participant risk behaviors, and other characteristics, such as willingness to use the experimental treatments and complete follow-up assessments. On-line advertising and off-line printed promotional materials may be the most cost-effective recruitment methods for enrolling young adults in a smoking cessation trial, but quit line screening, while more expensive, may enroll smokers with more motivation to use the experimental cessation services. Follow-up rates remain a challenge in smokers recruited through all methods. The Internet has expanded opportunities for recruiting participants cost-effectively, but off-line promotions can be cost-effective, too. Other recruiting methods that are more costly can still be advantageous by improving fulfillment of trial requirements by study participants.
Contributor Information
David B. Buller, Klein Buendel, Inc., 1667 Cole Boulevard, Suite 225, Golden, CO 80401, 303.565.4340, 303.565.4320 (fax), dbuller@kleinbuendel.com..
Richard Meenan, Center for Health Research, Kaiser Permanente Northwest, 3800 N. Interstate Ave. Portland, OR 97227, 503.335.2484, 503.335.2424 (fax), richard.meenan@kpchr.org..
Herb Severson, Oregon Research Institute, 1715 Franklin Blvd., Eugene, OR 97403, 541.484.2123, 541.484.1108 (fax), herb@ori.org..
Abigail Halperin, Department of Family Medicine, Box 354982 4311 11th Ave NE, Suite 210, University of Washington, Seattle, WA 98105, 206.616.4482, 206.325.6338 (fax), abigail@u.washington.edu..
Erika Edwards, Cherry Creek Schools, 14270 E. Briarwood Ave., Centennial, CO 80112, 720.886.7169, eedwards10@cherrycreekschools.org..
Brooke Magnusson, Alere Wellbeing, Inc., 999 Third Avenue, Suite 2100, Seattle, WA 98104-1139, 206.876.2209, brooke.magnusson@alere.com..
References
- 1.Gordon JS, Akers L, Severson HH, et al. Successful participant recruitment strategies for an online smokeless tobacco cessation program: A brief report. Nicotine Tob Res. 2006;8(Supplement 1):S35–S41. doi: 10.1080/14622200601039014. [DOI] [PubMed] [Google Scholar]
- 2.Keyzer JF, Melnikow J, Kuppermann M, et al. Recruitment strategies for minority participation: challenges and cost lessons from the POWER interview. Ethn Dis. 2005;15(3):395–406. [PubMed] [Google Scholar]
- 3.McClure JB, Greene SM, Wiese C, et al. Interest in an online smoking cessation program and effective recruitment strategies: results from Project Quit. J Med Internet Res. 2006;8(3):e14. doi: 10.2196/jmir.8.3.e14. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Graham AL, Bock BC, Cobb NK, et al. Characteristics of smokers reached and recruited to an internet smoking cessation trial: a case of denominators. Nicotine Tob Res. 2006;8(Suppl 1):S43–S48. doi: 10.1080/14622200601042521. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Princeton Survey Research Associates International, Pew Research Center's Internet & American Life Project Social Side of the Internet Survey 2010. Pew Research Center's Internet & American Life Project 2010.
- 6.Lieberman DZ. Evaluation of the stability and validity of participant samples recruited over the internet. Cyberpsychology and Behavior. 2008;11(6):743–5. doi: 10.1089/cpb.2007.0254. [DOI] [PubMed] [Google Scholar]
- 7.Graham AL, Milner P, Saul JE, Pfaff L. Online advertising as a public health and recruitment tool: Comparison of different media campaigns to increase demand for smoking cessation interventions. Journal of Medical Internet Research. 2008;10(5):e50. doi: 10.2196/jmir.1001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Ramo DE, Hall SM, Prochaska JJ. Reaching young adult smokers through the internet: Comparison of three recruitment mechanisms. Nicotine Tob Res. 2010;12(7):768–75. doi: 10.1093/ntr/ntq086. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Pew Internet and American Life Project Updated: Change in internet access by age group, 2000-2010. Pew Internet and American Life Project 2010.
- 10.Koo M, Skinner H. Challenges of internet recruitment: A case study with disappointing results. J Med Internet Res. 2005;7(1):e6. doi: 10.2196/jmir.7.1.e6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.McDonnell DD, Lee H, Kazinets G, Moskowitz JM. Online Recruitment of targeted populations: Lessons leared from a smoking cessation study among Korean Americans. SMQ. 2010;16(3):2–22. [Google Scholar]
- 12.Collyar DE. The Value of Clinical Trials from a Patient Perspective. Breast J. 2000;6(5):310–4. doi: 10.1046/j.1524-4741.2000.20060.x. [DOI] [PubMed] [Google Scholar]
- 13.Llewellyn-Thomas HA, McGreal MJ, Thiel EC, et al. Patients’ willingness to enter clinical trials: measuring the association with perceived benefit and preference for decision participation. Soc Sci Med. 1991;32(1):35–42. doi: 10.1016/0277-9536(91)90124-u. [DOI] [PubMed] [Google Scholar]
- 14.Harrison SL, Saunders V, Nowak M. Baseline survey of sun-protection knowledge, practices and policy in early childhood settings in Queensland, Australia. Health Educ Res. 2007;22(2):261–71. doi: 10.1093/her/cyl068. [DOI] [PubMed] [Google Scholar]
- 15.Velicer WF, Keller S, Friedman RH, et al. Comparing participants and nonparticipants recruited for an effectiveness study of nicotine replacement therapy. Ann Behav Med. 2005;29(3):181–91. doi: 10.1207/s15324796abm2903_4. [DOI] [PubMed] [Google Scholar]
- 16.Bussell H, Forbes D. Understanding the volunteer market: the what, where, who and why of volunteering. International Journal of Nonprofit and Voluntary Sector Marketing. 2002;7(3):244–57. [Google Scholar]
- 17.Lee S, Saito T, Takahashi M, Kai I. Volunteer participation among adults in Japan: an analysis of the determinants of participation and reasons for nonparticipation. Arch Gerontol Geriatr. 2008;47(2):173–87. doi: 10.1016/j.archger.2007.08.004. [DOI] [PubMed] [Google Scholar]
- 18.Fitch R. Characteristics and motivations of college strudents volunteering for community service. Journal of College Student Personnel. 1987;28(5):424–31. [Google Scholar]
- 19.Cemalcilar Z. Understanding individual characteristics of adolescents who volunteer. Personality and Individual Differences. 2009;46(4):432–6. [Google Scholar]
- 20.Paterson H, Reniers R, Vollm B. Personality types and mental health experiences of those who volunteer for helplines. Br J Guid Counc. 2009;37(4):459–71. [Google Scholar]
- 21.Cialdini R. Influence: Science and practice. Harper Collier; New York: 1988. [Google Scholar]
- 22.Grant AM, Dutton JE, Rosso S. Giving commitment: employee support programs and the prosocial sensemaking process. Academy Management Journal. 2008;51(5):898–918. [Google Scholar]
- 23.Centers for Disease Control and Prevention (CDC) Prevalence of current cigarette smoking among adults and changes in prevalence of current and some day smoking--United States, 1996-2001. MMWR Morb Mortal Wkly Rep. 2003;52(14):303–7. [PubMed] [Google Scholar]
- 24.Borland R, Balmford J, Hunt D. The effectiveness of personally tailored computer-generated advice letters for smoking cessation. Addiction. 2004;99(3):369–77. doi: 10.1111/j.1360-0443.2003.00623.x. [DOI] [PubMed] [Google Scholar]
- 25.Balmford J, Borland R, Benda P. Patterns of use of an automated interactive personalized coaching program for smoking cessation. J Med Internet Res. 2008;10(5):e54. doi: 10.2196/jmir.1016. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Balmford J, Borland R, Li L, Ferretter I. Usage of an Internet smoking cessation resource: the Australian QuitCoach. Drug Alcohol Rev. 2009;28(1):66–72. doi: 10.1111/j.1465-3362.2008.00009.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Zhu SH, Anderson CM, Johnson CE, et al. A centralised telephone service for tobacco cessation: the California experience. Tob Control. 2000;9(Suppl 2):II48–II55. doi: 10.1136/tc.9.suppl_2.ii48. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Halperin AC, Smith SS, Heiligenstein E, et al. Cigarette smoking and associated health risks among students at five universities. Nicotine Tob Res. 2010;12(2):96–104. doi: 10.1093/ntr/ntp182. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Thompson B, Coronado G, Chen L, et al. Prevalence and characteristics of smokers at 30 Pacific Northwest colleges and universities. Nicotine Tob Res. 2007;9(3):429–38. doi: 10.1080/14622200701188844. [DOI] [PubMed] [Google Scholar]