Craigslist versus print newspaper advertising for recruiting research participants for alcohol studies: Cost and participant characteristics

Abstract

Introduction

Technology has transformed our lifestyles in dramatic and significant ways, including new and less expensive options for recruiting study participants. This study examines cost and participant differences between two recruitment sources, Craigslist (CL), and print newspapers (PNs). This paper also reviewed and compared studies involving clinical trials published since 2010 that recruited participants using CL alone or in combination with other methods.

Method

Secondary data analyses from a parent study involving a randomized controlled trial of a mail-based intervention to promote self-change with problem drinkers.

Results

Significant differences were found between CL and PN participants on most demographic and pretreatment drinking variables. While all participants had AUDIT scores suggestive of an alcohol problem and reported drinking at high-risk levels, CL participants had less severe drinking problem histories, were considerably younger, and had a higher socioeconomic status than PN participants. The total advertising costs for the 65 CL ads ($275) were significantly less than the 69 PN ads ($33, 311). The recruiting cost per eligible participant was vastly less expensive using CL ($1.46) compared to print newspaper ads ($116.88).

Conclusions

Using CL is a viable recruitment method for soliciting participants, particularly those that are younger, for alcohol intervention studies. It is also less expensive than newspaper ads. When CL participants were recruited, they reported being slightly more confident to change their drinking than PN participants. Limitations of using CL are discussed, including that some initial ad responders gave inconsistent answers to similar questions and a few tried to enter the study more than once.

1. Introduction

Technology, which has become an integral part of our everyday lives, has radically changed the way we live and work, including how participants are recruited for research studies. Of the 321 million people in the U.S., 49% used Facebook in 2015 (Statista, 2015, June 8). Last year 87% of those living in North America used the Internet compared to 44% in 2000, a 98% increase in 14 years (Internet World stats, n.d.). While more Internet users were younger 10 years ago, recent research suggests a growing number of older adults are accessing the Internet, especially those that are more affluent and better educated (Older Adults and Technology Reports, n.d.).

Craigslist.org (Craigslist), a website for classified advertisements and discussion forums, is the 9th most popular Internet website in the United States (Alexa, 2015 May). Founded in 1995 in the San Francisco bay area, it was incorporated as a private for-profit company in 1999 (Craigslist Corporate History, 2008, April 21). Today it takes postings from individuals in more than 70 countries and in 12 different languages. It has also become a popular website for recruiting participants for research studies. It allows the placing of a single advertisement in any one geographic location (e.g., cities or metropolitan areas) for several days to a week, often at no cost. Based on factsheets from Craigslist (CL, 2015, May 5), it is estimated that 60 million people in the U.S. access CL monthly. Further, each month more than 80 million classified ads are posted. These statistics indicate that CL ads target an extremely large and broad audience.

Before the advent of the Internet, most researchers used some type of print (e.g., newspapers, magazines, flyers) or broadcast ads (e.g., radio, TV), or a snowball technique to recruit participants. Today, multiple web, Internet, and social media sites (e.g., email, Google Adwords, CL, Facebook) are being used to recruit participants into clinical trials. Because the use of the web and social media is a relatively recent recruitment method for clinical trials, we found, as did Frandsen, Walters, and Ferguson (2014), that there are “few studies that directly compared a sample obtained from a social media to one collected using traditional recruitment strategies” (p. 250). While studies have used CL to recruit different groups (e.g., smokers: Bansal-Travers, O'Connor, Fix, and Cummings, 2011; Mohebati et al., 2012; Ramo, Hall, and Prochaska, 2010; alcohol: Siegel, DiLoreto, Johnson, Fortunato, and DeJong, 2011; obesity: Worthen, 2013; HIV risk and substance use with MSM: Grov, 2011; Grov, Rendina, and Parsons, 2014), little is known about the differences between individuals recruited through CL versus other recruitment sources. It should be noted that there is no specific category called “research studies” for posting CL ads. Researchers that have used CL have reported placing ads in different sections (e.g., “jobs” or “volunteers”).

To better summarize existing studies on CL as a study recruitment source, we reviewed and compared studies involving clinical trials published since 2010 that recruited participants using only CL or in combination with other methods. Studies not designed as clinical interventions were excluded (e.g., examined relatives' perceptions of preschoolers' body sizes: Eli, Howell, Fisher, and Nowicka, 2014; beta tests to confirm technological feasibility: Ybarra, Prescott, and Holtrop, 2014). Nine studies met these criteria and are described in Table 1. For the five studies that used recruitment sources besides CL, data for those sources are included. Several studies compared multiple recruitment sources but did not include CL as a recruitment method; therefore, they are not listed in Table 1 [e.g., Frandsen et al. (2014), Facebook vs. print media; Heffner, Wyszynski, Comstock, Mercer, & Bricker, 2013, Facebook vs. word-of-mouth, print media]. Lastly, one study that used multiple recruitment sources including CL was excluded as only four participants were recruited from there (Morgan, Jorm, and Mackinnon, 2013).

Table 1.

Characteristics of studies that used Craigslist to recruit participants, and if relevant compared Craigslist with other Internet and traditional participant recruitment sources.

First author; year	Targeted clinical population/disorder; # of participants	Participants paid after study entrya	Traditional recruiting sources	Craigslist and other Internet media recruiting sources	Cost effectiveness evaluation	Significant differences (Not applicable = NA)	Most successful sources (%)	Other information
Current study; 2015	Problem drinkers; N= 473	$20.00	Newspapers (N)	Craigslist (C), jobs section (JS) in metropolitan cities in 48 different states	N = $116.88; C = $1.46	C younger, less severe drinking problems, higher socioeconomic status than N participants	N = 60.25%; C = 39.75%
Grov; 2014	Men who have sex with men; N = 148	No	Gay bars/clubs (BC); private sex parties (SP)b	Craigslist (C) in men-seeking-men (MSM) section in NYC	No	SP older, higher % of STI histories; C least likely to self-identify as gay		Goal: recruit 50 from each source (BC, SP, & C). Software error lost 2 BC participants
Worthen; 2014	Obesity; N = 38	$25.00 target gift card	None	Craigslist (C) in volunteer community section (VCS)	No cost for ads	NA	NA
Morgan; 2013	Depression; N = 1699	No	None	Google advertising (GA); Facebook (FB); online forums (depression; OF); other websites (W); online community noticeboards (Craigslist; ON); emails (E)	GA = $11.55 ADc FB = $19.89 ADc	NA	GA = 44.44%; W = % not provided	Due to technical oversight, recruitment source missing for some participants; cost per participant provided only for GA and FB
Grov; 2012	Men who have sex with men; N = 601	Bars/clubs and bathhouses — given 2 $1.00 scratch-off lottery tickets; Craigslist — entered raffle awarding 4 $50 Amazon gift cards	Gay bars/clubs (BC); bathhouses (B)	Craigslist (C) in men-seeking-men (MSM) section in NYC	No	BC youngest; B most ethnically and racially diverse; C least likely to self-identify as gay	C = 34.61%; B = 33.11%	Goal: recruit 200 from each source (BC, B, and C).
Mohebati; 2012	Smokers; N = 84	$40.00 for 1st 2 visits and $190 at completion of study for travel and parking expenses	Word of mouth (WM)	Craigslist (C) in community volunteers section (VCS) in Manhattan	No	NA	C = 97.62%; WM = 2.38%	Study was designed to use only Craigslist; however, 2.38% of participants recruited via WM
Bansal-Travers; 2011	Smokers; N = 193	$20.00	Newspapers (N)	Craigslist (C) no section reported	No	NA	NA	Sample sizes per recruitment source not provided
Siegel; 2011	Adolescents ages 16–18; ≥1 alcoholic drink in past 30 days; N = 241	$10.00 Amazon gift cards	None	Craigslist (C), in VCS	No cost for ads	NA	NA
Ramo; 2010	Smokers ages 18–25; N = 201	Chance to win raffle prize of $25 or $400	None	Craigslist (C) in VCS in 23 metropolitan cities; Adbrite advertisingd (social networking & lifestyle-based websites; AA); Survey Sampling Internationale(email campaign; SSI)	C = $0.66; AA = $42.77; SSI = $19.24	AA younger; SSI slightly heavier smokers; C more ethnically diverse and more likely to have completed college or some postgraduate education	AA = 45.27%; SSI = 33.33%

Note. Traditional recruiting sources refer to any form of broadcast or print media (e.g., radio, television, newspaper, mailings, and flyers) and word of mouth. Internet recruiting sources refer to any form of advertising via the World Wide Web (e.g., Google Adwords, Craigslist, and Facebook), electronic mail (e-mail), and online forums.

^aPayment for follow-up not included here.

^bGrov, Rendina, and Parsons, 2014 indicated that research staff contacted event promoters and space managers of private sex parties (SP), and provided them with study recruitment cards and asked them to include information about the study in e-newsletters to their members. These recruitment efforts were considered traditional, because (a) recruitment cards are a form of print media, and (b) the research staff did not directly send out e-newsletters to SP members.

^cAD = Australian Dollars.

^dPaid advertising on a variety of Web sites through Adbrite, an Internet marketing company.

^ePurchased completed surveys through Survey Sampling International, an online survey sampling company.

While the total cost of advertising is important, the effectiveness of a recruitment method is ultimately related to the number of participants who are enrolled in a study. Consequently, the best way to evaluate and compare the cost-effectiveness of different recruitment methods is to examine the cost per eligible participant by recruitment source. As can be seen in Table 1, the cost per enrolled participant is somewhat variable across studies. Another observation when comparing different types of web-based/Internet recruitment sites is that while some have great potential for reaching large audiences, proportionally the final sample can be much smaller. The following two clinical trials, which only used Facebook, are good examples. Brief et al. (2013) examined almost 11,000 returning combat veterans who had visited Facebook focusing on reducing their alcohol use and PTSD symptoms, and of those 3,500 were assessed for eligibility with about 1,340 determined to be initially eligible yielding a final study sample size of 617. Ramo, Rodriguez, Chavez, Sommer, and Prochaska (2014) using 36 different Facebook ads to target young adult smokers generated 3,198,373 impressions and 5895 unique clicks. The unique clicks only resulted in 10% (586) potentially eligible participants of whom only 39% (230) provided consent to participate in the study. While the reported advertising costs averaged $8.80 per eligible consented participant, only 34% (79/230) of the final study sample entered the actual study.

The present study used a secondary data analysis to evaluate the effectiveness and cost-effectiveness of CL versus print newspaper (PN) ads to recruit participants for a randomized controlled trial designed to promote self-change from alcohol problems (Sobell, Sobell, Gioia, Montgomery, and Marker, 2010, August). The analysis had two objectives: (a) to compare demographic and alcohol history variables for participants recruited using Craigslist.org with those recruited using a traditional strategy, PNs; and (b) to evaluate the cost-effectiveness of using CL versus PNs for recruiting research participants. In this paper, we also review the opportunities and challenges offered by CL for researchers.

2. Methods

2.1. Study participants

For purposes of the present analysis, participants are allocated to one of two groups (CL: Craigslist.org; PN: print newspaper) based on where they the saw the study ad (Sobell et al., 2010, August). Participants were recruited over a 20-month period using the following inclusion criteria: (a) must have seen the study ad (i.e., not have been told about it) and provided the date they saw the ad; (b) had to identify the source of the ad they saw (e.g., name of newspaper or CL, and the city where it was posted); (c) had to call and be screened within 30 days of seeing the ad; (d) had to be ≥21 years of age, the legal drinking age in the U.S.; (e) English speaking; (f) had a permanent address (e.g., P.O. boxes and shelter type facilities were excluded as all study materials were sent and received via courier); (g) one participant per household; and (h) in the 90 days prior to seeing the ad reported (i) binge drinking [i.e., ≥5 standard drinks (1 U.S. standard drink = 0.6 oz. of absolute ethanol or 14 g of absolute ethanol) on at least 5 days], or (ii) drinking on average ≥15 standard drinks per week for men or ≥8 standard drinks per week for women]. These drinking criteria are consistent with the risk drinking criteria used by the National Institute on Alcohol Abuse and Alcoholism (2007). In addition, callers who called more than once after being told they did not meet the eligibility criteria were deemed ineligible and were only counted one time. The study involved no face-to-face contact with participants.

The study was conducted at Nova Southeastern University (NSU) in Fort Lauderdale (FL) and was approved by NSU's Institutional Review Board. The 473 participants in the final study cohort had volunteered to participate in a randomized clinical trial (RCT) designed to promote self-change from alcohol problems. Henceforth, the RCT will be referred to as the parent study. Because the parent study has been described elsewhere (Sobell et al., 2010, August), only procedures relevant to the present manuscript will be described.

2.2. Advertisements

Study ads were placed in PNs and on CL in the 48 contiguous states in the U.S. and the District of Columbia. At the time of the study, the most recent U.S. Census Bureau data were used to select cities or geographic areas in each state with large populations. The same ads were used to recruit participants for PNs and CL. The ads as they appeared in print are shown in Fig. 1.

Fig. 1.

Craigslist and print newspaper advertisements used to recruit participants.

2.2.1. Craigslist (CL)

Ads for CL were posted by going to the main web page (Craigslist. org) and identifying an area of a state (e.g., South Florida), or county (e.g., Miami/Dade), or a specific city. For this study, the “jobs” section of Craigslist was used because other researchers had told us that this is where most research study ads are posted. Under the “jobs” section the subcategory “ETC” was selected. To post an ad it was necessary to click the link in the upper left corner of the webpage that says, “post classified” [see page 3 in Worthen (2013) for a sample Craigslist.org homepage showing these options]. Depending on the area selected we were able to post CL ads in the jobs section for one to two weeks, and then they were removed by CL. Ads were placed at least once in 51 major metropolitan areas across the U.S. On occasion, ads in some areas were run more than once. While Craigslist typically does not charge a fee for ads related to research studies, some major metropolitan cities (e.g., Dallas, Chicago) require a $25.00 fee for job-postings including research studies.

For those who called in response to a CL ad and for whom data were available at the initial telephone screen, about half (52%; 440/846) were recruited from six major cities [Chicago, n = 151; Portland, n = 74; Dallas, n = 62; Los Angeles, n = 61; Seattle, n = 51; Philadelphia, n = 41). Some CL ads resulted in few responders (e.g., ads in 6 cities resulted in only one responder per city). These differences could have reflected the population in the area served by the particular CL, how many times a CL ad was placed for a particular city, how many days the ad was run, and whether the ad appeared near the front or the end of the CL page. The latter two variables were not within our control.

2.2.2. Print newspapers (PNs)

The ad used in PNs was the same as for CL, except it was usually small (e.g., 3″ × 4″) because of cost constraints. As with the CL ads, large cities in the 48 contiguous states and the District of Columbia were targeted for PN ads. The cost of placing ads had a great deal of variability. For budget considerations, the highest priced ad accommodated was $1080, and ads were not placed if they exceeded that amount. As with CL ads, some PN ads were run more than once in some cities. While PN ads were requested to be placed in the main/front section of a newspaper, sometimes they appeared in other sections (e.g., health, fitness). What affects the cost of many newspaper ads is the number of days the ad appears and the section where it appears. Due to such costs, most of the 69 PN ads were run for only one day.

2.3. Study procedures

The study procedures were the same for participants whether they were recruited from CL or PN ads. All callers were prescreened by phone for initial eligibility. Those who were initially eligible were sent a consent form by mail and asked to read, sign and return it in a prepaid envelope addressed to the project. Once the signed consent was received, potential participants were sent their assessment materials by mail to complete and to return to the project using a prepaid envelope. After returning their completed assessment forms, all 473 participants who were eligible and entered the study were sent a $20.00 check. Participants who completed the 3-month follow-up interview were sent a second $20.00 check. All participants received the same intervention materials to assist them in changing their drinking. A slightly modified version of the materials for use outside of the study's requirements is available for download (http://www.nova.edu/gsc/forms/change_alcohol_use.pdf).

2.3.1. Prescreening questions

Based on other studies that have used web-based ads (Klingemann and Sobell, 2007; Mirtenbaum et al., 2013; Sobell, Sobell, and Agrawal, 2009), large numbers of callers were anticipated. Initial callers were prescreened using key questions that could quickly rule out ineligible callers (e.g., age, stable mailing address). Another reason for the phone prescreening was because the placement of the CL ads (i.e., job section) might attract people who search for paid studies to make money and thus might be inclined to give invalid answers hoping to be in the study.

Because prior research has shown that some participants provide invalid answers to CL and Facebook ads (Brief et al., 2013; Mohebati et al., 2012; Ramo et al., 2010), we addressed this as in past studies (Ramo et al., 2010; Sobell et al., 2002). Potential participants were asked several questions on two different occasions, and those who gave inconsistent answers at the prescreen phone call and assessment (e.g., Prescreen: yes/no to the question “ever been in treatment/self-help group for an alcohol problem?” compared with Assessment: yes/no answers for different alcohol treatment programs-inpatient, outpatient, AA) were excluded.

2.3.1.1. Quick Drinking Screen

An important criterion for entry into alcohol studies is the amount of alcohol consumed. During the phone prescreening the QDS (Dum et al., 2009; Roy et al., 2008; Sobell et al., 2003), a 4-item self-report measure of alcohol consumption that uses quantity and frequency questions, was used to assess self-reported drinking. The QDS is psychometrically sound and collects reliable aggregate drinking data for major drinking variables when compared to the Timeline Followback (TLFB). In the current study, the QDS was used to collect prescreening drinking data for the three months prior to seeing the study ad.

2.4. Assessment measures

Participants who were eligible based on their initial prescreening answers were asked to complete an assessment questionnaire and several forms as described below.

2.4.1. Brief psychosocial questionnaire

This form included basic demographic (e.g., gender, age, marital and employment status, years of education), and alcohol history questions (e.g., alcohol-related arrests, alcohol withdrawals and other related consequences, past alcohol treatment).

2.4.2. Alcohol Use Disorders Identification Test (AUDIT)

The World Health Organization developed the AUDIT, a 10-item psychometrically sound scale. Scores can range from 0 to 40, with ≥8 suggestive of an alcohol problem. It is used to initially screen individuals whose scores might suggest a problem with alcohol (Reinert and Allen, 2007).

2.4.3. Motivation to change drinking

Two scales assessing motivation to change drinking have been used in past brief interventions (reviewed in Sobell and Sobell, 2011). The first asks participants “On a scale from 0% to 100%, what number best reflects how CONFIDENT you are RIGHT NOW that you can change your drinking?” (0 = not at all confident, 100 = totally confident). The second scale, a Readiness Ruler, asks participants to “select a number from 1 to 10 that “best reflects how READY you are at the present time to change your drinking?” (1 = definitely not ready to change, 10 = definitely ready to change; Rollnick, Mason, and Butler, 1999). For both scales, higher numbers indicate greater confidence or readiness to change, respectively.

2.4.4. Subjective evaluation of drinking problem

Studies of brief interventions and promoting self-change (reviewed in Klingemann and Sobell, 2007; Sobell and Sobell, 2011) have asked participants to provide a self-appraisal of the severity of their alcohol problem prior to entering the study. In the present study, this time frame was 90 days before seeing the ad. The scale was a 5-point Likert scale (1 = not a problem, 2 = very minor, 3 = minor, 4 = major, 5 = very major). Studies have shown that higher severity ratings have been associated with poorer treatment outcomes (reviewed in Sobell and Sobell, 2011).

2.4.5. Timeline followback (TLFB)

The TLFB is a psychometrically sound measure for assessing daily drinking for intervals ranging from 1- to 12-months (Agrawal, Sobell, and Sobell, 2008; Sobell and Sobell, 1992). Examining alcohol consumption over shorter TLFB intervals (e.g., 1 to 3 months) has been shown to sufficiently capture problem drinkers' annual use when compared to longer TLFB intervals for both pre- (Vakili, Sobell, Sobell, Simco, and Agrawal, 2008) and post-intervention (Gioia, Sobell, Sobell, and Simco, 2012). The present study used 90 days for both the pre- (prior to seeing the ad) and post-intervention (after receiving the materials by courier) intervals.

2.5. Follow-up

Participants were scheduled for a follow-up interview 3-months after they received their intervention materials. The follow-up questions were similar to those asked at the baseline assessment. As in past studies (Sobell et al., 2002), to insure a high follow-up rate with participants, at the start of the study we told them that we would send them a postcard reminder one month before their 3-month follow-up interview. The postcard also indicated that if a participant's contact information had changed to call the study's 800 number. The follow-up materials were sent by the mail using the USPS. Like other alcohol studies (e.g., Miller and Del Boca, 1994; Sobell et al., 2002), to minimize the number of participants lost to follow-up, telephone calls were made to those who had not returned their follow-up materials and they were asked if they would be willing to take a few minutes to answer some questions about their recent alcohol use. The QDS, discussed earlier, was used to collect drinking data in such cases.

2.6. Data analysis

Descriptive differences on demographic and drinking variables for participants recruited by CL and PN ads were evaluated using 2-tailed independent sample t-tests for parametric variables, and 2-tailed Fisher's exact tests for nonparametric variables. Bonferroni adjustments (Holland and Copenhaver, 1988) were made for the 11 alcohol-related variables, thereby setting the Type I error rate at α = 0.005 (0.05/11).

3. Results

3.1. Participants

Demographic and alcohol use variables for the full sample and by recruitment source (CL; PN) are presented in Table 2. Of the 21 variables, 7 are demographic (e.g., education, age, gender), 2 assess motivation to change (i.e., readiness, confidence), and 12 reflect lifetime (e.g., years alcohol has been a problem) and recent (i.e., 90 days prior to seeing the study ad) alcohol use.

Table 2.

Demographic and alcohol variables for 473 participants and by recruitment source.

Variable	Craigslist (n = 188)	Newspapers (n = 285)	Fisher Exact testa	t-Test (p value)b	All participants (N = 473)
% male (n)	43.09 (81)	52.28 (149)	0.060		48.63
% married or common-law (n)	29.26 (55)	27.72 (79)	0.755		28.33
% White/Caucasian (n)	72.87 (137)	56.14 (160)	0.000c		62.79
% full-time or self-employed (n)	50.53 (95)	37.89 (108)	0.008		42.92
% white collar job (n)	53.49 (92)d	32.10 (87)e	0.000c		40.4f
Mean (SD) age, years	36.39 (11.73)	48.35 (11.02)		-11.26 (0.000)c	43.60 (12.73)
Mean (SD) education, years	14.73 (2.35)	13.60 (2.20)		5.32 (0.000)c	14.05 (2.33)
Mean (SD) readiness score to change drinking at the present time	7.88 (1.76)	8.01 (1.98)g		-0.74 (0.460)	7.95 (1.90)h
Mean (SD) confidence score to change drinking at the present time	75.73 (18.00)	71.36 (22.59)g		2.23 (0.026)	73.10 (20.97)h
% participants self-rated their alcohol problem as major to very major (n)i	50.53 (95)	65.14 (185)g	0.002		59.32h
Mean (SD) alcohol arrestsj	0.76 (1.62)	2.00 (5.94)		-2.80 (0.005)m	1.51 (4.76)
Mean (SD) alcohol hospitalizationsj	0.47 (1.49)	1.70 (4.41)		-3.70 (0.000)m	1.21 (3.59)
% previous alcohol treatment (n)j	29.26 (55)	48.42 (138)	0.000m		40.80
% ever had major alcohol withdrawal symptoms upon stopping drinking (n)j	23.94 (45)	38.60 (110)	0.001m		32.77
Mean (SD) years alcohol problemj	9.28 (8.27)	16.91 (12.03)		-7.59 (0.000)k	13.88 (11.32)
Mean (SD) AUDIT scorej	20.94 (7.57)	23.57 (7.63)		-3.69 (0.000)k	22.53 (7.71)
Mean (SD) alcohol consequencesi,j	4.04 (2.14)	4.25 (2.10)		-1.07 (0.287)	4.16 (2.11)
Mean (SD) % days abstinenti,j	36.40 (26.96)	25.45 (24.34)		4.59 (0.000)k	29.80 (25.95)
Mean (SD) % days drinking ≥5 Standard Drinks (SDs)i,j,l	28.32 (25.74)	43.93 (32.23)		-5.57 (0.000)k	37.73 (30.76)
Mean (SD) SDs per drinking dayi,j,l	4.91 (2.32)	5.91 (2.87)		-4.01 (0.000)k	5.52 (2.70)
Mean (SD) SDs per weeki,j,l	21.95 (15.31)	30.56 (18.22)		-5.35 (0.000)k	27.14 (17.62)

Note: Unless indicated otherwise, variables are for lifetime. Readiness to change scores range from 1 = not at all ready, to 10 = definitely ready. Confidence to change scores range from 0 = not at all confident, to 100 = totally confident. Subjective alcohol problem severity was rated on a 5-point scale (Very Major, Major, Minor, Very Minor, Not a Problem). AUDIT = Alcohol Use Disorders Identification Test (scores range from 0 to 40; a score of ≥8 is suggestive of an alcohol problem; Allen, Fertig, Litten, Sillanaukee, & Anton, 1997).

^aFisher Exact tests used for categorical variables.

^bt-Test for independent samples (2-tailed) used for quantitative variables; df = 471 unless N < 473.

^cSignificant at p = 0.000 (rounded).

^dn = 172.

^en = 271.

^fn = 443.

^gn = 284.

^hn = 472.

ⁱ90 days prior to seeing the ad.

^jBonferroni adjustments were made for these last 11 alcohol variables thought to be related a priori.

^kSignificant at p ≤ 0.005 level with Bonferroni adjustment to maintain a family wise error rate at α = 0.005 (0.05/11).

^lOne standard drink = 14 g of absolute ethanol.

Significant differences occurred between the CL and PN participants for 17 of the 21 variables in Table 2. Gender, marital status, readiness to change drinking, and consequences in the 90 days prior to seeing the ad did not differ between the groups. Although the percentage of males was slightly less for CL (43%) compared to PN (52%) participants, the difference was not statistically significant. Over 25% of all participants were married, and participants reported an average of 4 alcohol-related consequences in the 90 days prior to seeing the ad. Significantly (p < .001) more CL participants than PN participants identified as White/Caucasian (73% vs. 56%), CL participants were on average 12 years younger (36 vs. 48), about half were working fulltime/self-employed and held white-collar jobs compared to PN participants (roughly one-third). In terms of education, while the difference between the two groups was significant, both groups had a mean education level reflecting some undergraduate training. The two groups differed significantly (CL: 76%, PN = 71%; p = .026) on only one of the two motivational variables, confidence to change. At the start of the study, both groups had high mean readiness to change drinking scores (8, with 10 = very ready).

Participants recruited from CL vs. PN ads differed significantly on 11 of the 12 alcohol variables in Table 2. The differences suggest that those recruited by PN ads had a more severe and longer history of alcohol problems. For example, the PN participants had more and heavier drinking days than CL participants and reported more consequences including alcohol withdrawal symptoms. However, although the PN participants had more severe problems, both groups had a mean AUDIT score of ≥20, well above a score of ≥8, which is suggestive of an alcohol problem (Reinert and Allen, 2007), and above a score of ≥16 reflective of a “high level of alcohol problems” (Babor, Higgins-Biddle, Saunders, and Monteiro, 2001). In addition, participants in both groups reported drinking at high-risk levels prior to the intervention.

3.2. Recruitment methods

Fig. 2 presents a CONSORT diagram for participant flow into the study by recruitment source. During the 20-month recruitment period, 2249 people called in response to all study ads. Of the 2249 callers, 918 (40.82%) met the initial eligibility criteria with the remaining callers being excluded because: (a) 900 (40.02%) did not meet one or more of the study inclusion criteria at the first call (e.g., did not complete a telephone screening within 30 days of seeing the study ad; under 21 years of age; did not meet minimum drinking requirements, did not see the ad themselves–was told by someone else about the ad); and (b) for 431 (19.16%) further contact was not possible or they were not interested after the initial phone call. Of the 900 ineligible participants, 10.11% (91) were ineligible because their assessment form answers did not match those they gave at the initial phone screen, and 1.29% (29/2249) of initial callers after being told they were ineligible called back a second or third time trying to re-enter the study. Repeat callers were identified through identical phone numbers and email addresses. These latter callers were counted only once as a caller.

Fig. 2.

Consort diagram by advertisement source.

As shown in Fig. 2, another 257 (28.00%) potential participants were excluded because they did not return their consent form, and 188 (20.48%) who signed and returned their consent form never returned their assessment materials. Thus, slightly more than half (473/918, 51.52%) of those initially eligible met all study criteria and were entered into the parent study.

3.3. Costs

The 65 CL ads cost $275 (54 were free; 11 cost $25 each) compared to $33,311 for the 69 PN ads (ranging from $144 to $1080). Using only the costs for ads, the cost to recruit one eligible CL and PN participant (total cost/number of eligible participants) was $1.46 ($275/188) and $116.88 ($33,311/285), respectively. However, a more comprehensive cost analysis might consider other factors that could lower the cost differential somewhat between the two ad sources such as more attempts needed to contact callers with more severe problems or more time spent in conducting screenings. For example, in this study screening was discontinued with the first ineligible answer (e.g., did not see the ad themselves). However, this would depend on how many questions it took to screen ineligible participants (e.g., some CL callers screened out quickly as they did not drink or were underage). The present study did not keep records of prescreening and screening procedures. However, such costs may not be relevant as major technological advances (e.g., Internet recruitment at SurveyMonkey®) can reach large numbers of people using automated screening procedures.

4. Discussion

The objectives of the present study were to evaluate the cost-effectiveness of using Craigslist (CL) versus print newspaper (PN) advertisements to solicit participants for a study promoting self-change, and to evaluate differences between participants recruited through these two sources. In terms of cost-effectiveness, soliciting participants for a study promoting self-change from alcohol problems was considerably less expensive using CL ads than PN ads ($1.46 vs. $116.88, respectively).

Participants from the two recruitment sources also were found to differ significantly on most demographic and alcohol variables. The PN participants reported having a longer drinking problem history (17 years vs. 9 years), and evaluated their drinking problem as more severe (i.e., major to very major) than CL participants (major to very major, 65 and 51%, respectively). On the other hand, CL compared to PN participants reported a higher socio-economic status (i.e., employment and education), and were younger on average by over a decade.

With respect to differences found between participants recruited from CL and PN ads, a Pew Center Research survey (Internet user demographics, n.d.) found that among adults who used the Internet in 2014 more were educated and younger and had a higher SES. The Pew survey also examined adults who read newspapers and found that more frequent Internet users tended to be younger and better educated than less frequent users which is consistent with the present study's CL participants.

At the time study participants saw the ad, on average they reported having a high mean readiness to change their drinking score, and most reported being very confident in their ability to change their drinking. Such findings are consistent with most studies that have used ads to recruit individuals with alcohol problems into brief interventions or tudies promoting self-change (see Klingemann and Sobell, 2007; Sobell and Sobell, 2011).

While significantly more of the 473 final study participants responded to PN (30%) ads than CL (20%) ads, the cost of recruiting each PN participant was 81 times greater for PN than for recruiting CL participants ($1.46 vs. $116.88, respectively). Finally, while the total number of ads for CL and PN was very similar (65 vs. 69, respectively), more calls were received from the CL ads. The larger number of inquiries from the CL ads may have reflected in part that many of the ads appeared in the Jobs “ETC” section, but it could also be due to CL ads being available for several days while most PN ads only ran for one day.

The results of this study suggest that advertising on Craigslist is an efficient and highly cost-effective method for recruiting persons with alcohol problems to participate in clinical research studies. However, there are some things that should be considered before deciding upon which type of recruitment method to use. As with other Internet recruitment sources, the CL ads in the present study attracted a large number of callers who did not meet initial study criteria. One reason may be that the ad provided little information about screening criteria except stating that eligible participants would be paid $40 and receive free materials. Because CL does not have a specific “research” section, study ads have been placed in several different sections. For example, the present study and others (Eli et al., 2014; Sobell et al., 2010, August; Ybarra et al., 2014) have posted study ads in the “Jobs” section of Craigslist, while other studies have placed ads in the “Community [Volunteers]” section (Mohebati et al., 2012; Ramo et al., 2010; Siegel et al., 2011). Another important issue is that in this study participants recruited through CL were younger and had less severe problems than those recruited through PNs. Thus, studies seeking participants with more severe alcohol problems would be advised to explore other recruitment streams including print ads. In addition, differences in characteristics such as socioeconomic level between CL and other sources should be explored being selecting a recruitment source. Future research is needed to identify the most appropriate or effective placement of CL study ads, including how to recruit for specific clinical problems (e.g., depression, anxiety, hoarding, substance use). Suggestions and costs for posting and reposting ads are discussed in a recent article about using CL to recruit participants for clinical studies (Worthen, 2013).

A recent article has criticized CL as a recruitment source because it is limited to (a) ads in major metropolitan areas (vs. rural areas), (b) individuals who have access to the Internet, and (c) persons who know about CL ads (Worthen, 2013). Many of these points are weakened by a recent Pew survey (“Internet User Demographics, n.d.) showing that almost 90% of adults accessed the Internet in 2014. Although CL ads are location specific, they can be placed in multiple locations or cities in the U.S. at little or no cost.

For research studies that use online surveys or put interventions on smartphones or on the Internet, geographical restrictions are irrelevant as participants can respond from any locale. In fact, a very recent review of Facebook studies, which reported that they have “attracted more than 10 million participants” (p. 544; Kosinski, Matz, Gosling, Popov, and Stillwell, 2015), suggests that for many studies geographical boundaries are disappearing. This widespread access also provides for greater external validity of findings (e.g., recruiting participants from many locations when there are no geographic restrictions.

Despite the steady increase in the number of published clinical trials using the Web and Internet to recruit and enroll participants in research studies, emerging technology has raised two concerns regarding study data integrity and sample validity: (a) How accurate are the self-reports of participants' recruited from these sites? and (b) How can dishonest responding or wrongful entry into a study be prevented? In three other studies using Internet recruiting some percentages of respondents have given inconsistent or invalid answers, (2%, Brief et al., 2013; 3%, Mohebati et al., 2012; 2%, Ramo et al., 2010). In the present study, 10% of the 900 ineligible participants provided inconsistent answers and 1.3% called more than once after being deemed ineligible. To address inconsistent participant answers it is recommended that online or web-based studies include a data integrity check as one way to minimize participants' inaccurate self-reports. For a further discussion of these issues, readers are referred to a recent paper that (a) describes the challenges experienced in two web-based studies, and (b) presents several strategies to address concerns with new recruitment sources (Kramer et al., 2014). These strategies include suggestions for how to limit misrepresentation by respondents (e.g., use software to record dates and times of entries; disable computer programs written to auto populate surveys; use paired items to identify inconsistent responders).

One important finding that emerged in this study is that when both CL and PN participants responded to the study ad they indicated they were highly “ready” to change their drinking. Regarding PN ads, this finding is consistent with previous promoting self-change studies that have recruited participants using PN ads (Sobell et al., 2002). The CL findings are consistent with two other studies by other investigators using CL ads to recruit smokers that found that over half reported themselves as willing to quit smoking in the next six months (Gordon, Akers, Severson, Danaher, and Boles, 2006; Ramo et al., 2010). This suggests that the Internet can be used to recruit participants who are ready to change for interventions designed to change substance use. Further research needs to evaluate what other factors might be associated with Internet recruitment. For example, participants recruited by the Internet may be more likely to agree to participate in studies that do not require face-to-face interactions and for which participation can be flexibly scheduled.

Given the above considerations, both Internet and newspaper ads successfully recruited participants for a research study. Each route has its own advantages and disadvantages. For example, it appears that more individuals with less severe alcohol problems responded to CL ads whereas PN ads attracted individuals with more severe problems. This conclusion, however, relates to the present study, and it is not known whether different CL ads would solicit more severe cases. Craigslist ads were placed in the jobs section because other studies have successfully recruited research participants from this section. Placement in this section may have attracted a greater number of respondents looking for money vs. participating in a research study. Another consideration is how fast a study needs to recruit participants. For this study, 60% of participants were recruited from PNs, which was possible because advertising costs were paid by a grant. For investigators lacking research support, CL ads would be more cost-effective. Obviously, several factors must be considered when deciding which recruitment methods to use and why.

In summary, this and other studies have shown that Craigslist.org and other major web-based and Internet sites are a viable and inexpensive way to recruit participants for health and mental health studies. This is especially the case because available survey development programs have now made it possible to completely or nearly completely automate all study recruitment and follow-up procedures. In our review of studies using CL and other major web-based sites, it appears CL is the least costly way to recruit participants. This is because most postings using CL do not charge for advertising. The only consistent demographic difference between traditional and web-based sites, including CL, is that the Internet has been attracting younger potential participants. This, however, is changing, with all but 13% of the U.S. population accessing the Internet last year. However, as with all research the best methods are those that will address the research needs. Thus, for example, while CL would seem a good approach for soliciting younger individuals who have alcohol problems that are not severe, other methods might be better for attracting older and more severely dependent populations of problem drinkers. A caveat, however, is that the present study did not seek out severely dependent individuals, and targeted ads might be successful in recruiting such a sample.

Highlights.

• 1st study to compare differences for Craigslist (CL) and print newspapers (PN) ads

• Significant differences between CL and PN participants on most variables

• Significantly different costs per participant for CL ($1.46) vs. PN ($117) ads

• Web and social media sites effective and easy ways to recruit younger participants

• With some limitations, CL is an efficient and inexpensive recruitment tool.