Recruiting men who have sex with men on Craigslist.org for face-to-face assessments: Implications for research

Abstract

We adapted time-space sampling to enroll men who have sex with men (MSM) off Craigslist.org for face-to-face interviews. Men responding to our ads (n=322) were instructed to either complete an online pre-screening survey (to determine preliminary eligibility) or call our office directly. Of those taking further initiative to enroll, 29% (n=41) called directly and 71% (n=101) opted to first complete the online survey. Participants scheduled via online pre-screening were more likely to present for their face-to-face assessment than men deemed eligible directly via phone screening (72.3% vs. 47.1%). Online pre-screening was a useful tool to offer potential participants when recruiting on Craigslist and improved study enrollment.

INTRODUCTION

Online recruitment has become a popular approach to reach large numbers of people for research on HIV prevention and sexual health [1–3]. Many men who have sex with men (MSM) use the Internet for social and sexual networking on both sex-themed “hook-up” websites (e.g., Manhunt.net, Adam4adam.com) and social networking sites (e.g., facebook.com) [4, 5]. Studies suggest between 40%–53% of MSM seek sex partners online, thereby making the Internet a prolific venue for research and prevention [4, 6]. Despite the many advantages of using the Internet for collecting data, engaging hard-to-reach individuals, and conducting web-based interventions [6], only a few studies have reported on using the Internet to recruit MSM for face-to-face assessments [1–3].

Data suggest that it is possible to engage MSM online for subsequent in-person interviews, though with high attrition between first contact and actual assessments. Parsons et al. [1] compared field-based recruitment to Internet-based recruitment. In this study, 948 men completed a brief screening online, from which 220 were eligible and provided contact information, 115 were reached via phone, 28 were scheduled for an assessment, and 15 (54%, 15/28) completed a baseline interview. Elford et al. [2] conducted a study of Internet use and HIV risk among MSM men in London. Out of the 1,845 men who completed online surveys, 93 completed in-person interviews. Similarly, Fernández et al. [3] were successful in using Internet chat rooms to recruit Latino MSM for an in-person intervention—of the 735 men they “chatted” with online, 176 (24%) showed for an assessment.

Much research on Internet-based MSM has recruited on membership-based websites such as Manhunt.net or Gay.com [5]. Techniques typically used to recruit on these sites include banner advertising, registering a profile for the study and then “lurking” on the website, posting messages in chat rooms, emailing members directly, and coordinating with the website moderator to send “e-blasts” to users [5, 6]. Craigslist—an online bulletin board—is a medium where users post single ads through the site, but subsequent correspondence occurs out of the website via private email. The site is organized by city and neighborhood such that users can search for and post ads near their geographic location. Ads are displayed in reverse chronological order (newest on top). Up to 100 ad headlines are displayed on a page, with older posts in queue on subsequent pages. In New York City (NYC), the men-seeking-men section is extremely active—one study noted that between two to four thousand ads were posted every day [7].

The structure of Craigslist makes its virtual environment different from many traditional membership-based websites. Craigslist is free to use, requires no membership/subscription, and listings are available to the public. It does not maintain internal correspondence mechanisms (e.g., instant messaging/chatting). Aside from automated filters to prevent duplicate ads or the inclusion of a telephone number or URL, content is not moderated by a central administrator. Instead, users moderate the site. Any user can flag an ad for removal.

A common method for recruiting on Craigslist is to respond to individuals’ postings (via email) and encourage them to participate in a study [8]. However, this procedure excludes individuals who use Craigslist exclusively as browsers (i.e., men who browse through ads and respond to postings, but never actually post an ad themselves). Adapting recruitment approaches designed to maximize the representativeness of samples in physical spaces where MSM congregate, might be useful to overcome some sampling bias in recruiting Craigslist users.

Our goal was to adapt time-space sampling [9, 10], a probability-based strategy that utilizes a two-stage process to systematically sample individuals at randomly selected venues and times, to reach sexually active MSM in the NYC metropolitan area via Craigslist.org for in-person interviews. Time-space sampling has been adapted by the CDC for web-based HIV behavioral surveillance [11] and for recruitment in Internet chat rooms [3]. In this study, we report on screening and enrollment outcomes to reach our target sample. We compared eligibility and participant loss rates based on whether potential participants chose to call our office directly or self-selected to complete the online pre-screener after responding to our study ad. Given that it is largely unknown whether Internet-based samples are representative of the target population and whether they are comparable to samples recruited through other methods [11], this descriptive study was intended to provide data on online recruitment using an adaption of time-space sampling for in-person interviews in order to inform decisions about sampling and enrollment approaches that target MSM in online environments.

METHOD

Participants and Procedure

Data are taken from Project Score, a pilot study investigating the places where sexually-active MSM meet their sex partners. To be eligible, participants had to be biologically male, at least 18 years of age, report at least two new male sex partners within the last 30 days, able to complete an assessment in English, and have a working phone number. Those eligible were invited to participate in a face-to-face interview at our research office. One objective of Project Score was to adapt time-space sampling to recruit n = 50 men off Craigslist. Craigslist.org was selected because of (1) its unique nature as a free and publically available online bulletin board being used by MSM to facilitate sexual encounters, (2) the large magnitude of MSM who use this site every day [7], and (3) the limited research currently available on MSM who use this site.

We adopted time-space sampling in order to reduce bias caused by only posting ads at times that were convenient for the research staff (e.g., during typical business hours), which may be outside of times when Craigslist use is at its peak (i.e., evenings and weekends). Following guidelines for time-space sampling [9, 10], the research team first identified the most socially-viable times in which to recruit on Craigslist—determined as having an adequate magnitude of the target population at the venue. Based on our previous experience doing research on Craigslist we determined that the hours between 7am and 2am were the most viable in which to encounter the target population. We divided these times into one hour increments (e.g., 7am–7:59am, 8am–8:59am, 9am–9:59am). Then the research team used a random digit generator to select a (1) day of the week, (2) borough/neighborhood within NYC, and (3) time-increment to post. The number of postings in a given week was limited to two in order to ensure adequate resources were available to see participants during eventual face-to-face assessments. We weighted borough selection to match NYC census data (e.g., 31% of the NYC population resided in Brooklyn while only 6% resided in Staten Island).

At the date and time selected, a member of the research staff posted an ad for the study in the men-seeking-men (M4M) section on Craigslist, which is the only section exclusively dedicated for men to seek sex with other men. When the time came to post a new ad, the old ad was pulled off the website; however, note that our older ads were “pushed down” by newer posts from the Craigslist community such that within hours of posting our ad, it was queued behind hundreds of other postings (greatly reducing visibility to users).

Automated filters on Craigslist prevent users from posting a duplicate ad within seven days (even if said ad was pulled offline, for example, five days ago). As such, and in sequential order, we alternated between one of three “headlines” in ads: (1) “Answer a few questions about your sex life,” (2) “Participate in a Sex Research Study,” or (3) “Let’s talk about sex.” These headlines were selected by the research team as way to generate interest and curiosity by potential participants. The body text of the ads were identical and included the study logo, study description (i.e., face-to-face assessments at our Manhattan office, $40 incentive at baseline), and instructed men to respond to the ad via email. Automated filters prevented us from including our telephone number or URL in the text of the ad.

Research staff replied to email inquiries promptly. Our messages indicated that they could (a) call the project directly and screen via phone or (b) complete a brief online preliminary screener (URL included) that would ask for their contact information if they appeared eligible (and study staff would re-contact them via phone). Ultimately, all participants had to complete a full screening via phone before they could be scheduled for a face-to-face assessment. The Brooklyn College Institutional Review Board approved study procedures.

Measures and Analytic Plan

The brief online screening questionnaire assessed minimum eligibility criteria including participants’ age, biological sex, and the number of new male partners they had in the last 30 days. It also assessed sexual identity (gay, bisexual, heterosexual, other) and race or ethnicity, but these were not factored into eligibility. We conducted phone assessments to screen for full eligibility criteria (outlined previously) in addition to participants self-reported HIV status, race and ethnicity, and the NYC borough in which they lived.

We tracked recruitment efforts in multiple ways. We recorded the day, time, and NYC borough in which ads were posted in addition to the number of email responses we received from each posting. We recorded the number of men who telephoned us directly versus those who opted to first complete the brief screening survey online. We compared eligibility and participant loss rates based on whether potential participants chose to call our office directly or self-selected to complete the online screener first.

RESULTS

Between August 2010 and March 2011, our research team posted to Craigslist 36 times. Per NYC population-weighted randomization, one-third (n = 12) of ads were posted to Queens, 30.6% (n = 11) to Brooklyn, 13.9% (n = 5) to Manhattan, 13.9% (n = 5) to the Bronx, and 8.3% (n = 3) to Staten Island. Three of our ads were flagged and removed by the Craigslist community before we received any email responses (an ad posted in Manhattan, Staten Island, and the Bronx). An additional five ads were flagged for removal by the Craigslist community, but after we already received responses (M = 2.6 emails)—four ads were posted in Queens and one in the Bronx. Of the 28 ads that were never flagged, we received an average of 11 emails per ad (range 0 to 22). Across all postings, there were no significant differences in the mean number of emails received by (1) the borough where ads were posted, (2) the time of day that ads were posted (morning vs. afternoon vs. evening), or (3) whether ads were posted on a weekday or weekend. The headline “Participate in a Sex Research Study” resulted in significantly more email responses on average than the other two headlines (M = 14.6, versus 7.9 for “Answer a few questions about your sex life,” and 7.3 for “Let’s talk about sex” F (2, 30) = 5.79, p = .007).

In total, we received 322 unique email replies to our postings. Project staff responded to every email directing participants to either call our center directly or complete a brief online screener. Figure 1 displays the paths individuals self-selected during this recruitment process. We had no further contact with 55% (n = 177) of men. Forty-one men (12.7%) called us directly and 101 (31.4%) self-selected to complete the brief online pre-screener. There were three additional men (0.9%) we spoke with via phone, but were unable to determine if they self-routed through the online survey first. None of these three participants were eligible for the study.

Figure 1.

Comparing retention of participants who opted to telephone us directly versus those who first completed a survey online

Eighty-three percent (34/41) of those who telephoned us directly were eligible and scheduled for a face-to-face assessment—47.1% (16/34) showed for their assessment. Of the seven men who telephoned us directly and were not eligible, six reported an insufficient number of new male partners in the last 30 days.

In comparison, one-hundred men completed the online pre-screening survey and 83% of these men met preliminary eligibility criteria (i.e., biologically male, aged 18+, reported at least two new male sex partners in the last 30 days). All men who did not meet preliminary criteria were excluded solely because they reported an insufficient number of new male partners in the last 30 days. Eighty-eight percent (73/83) of those deemed preliminary eligible on the online screener provided contact information and project staff were able to reach 70% (51/73) of these men via phone for verification screening. Ninety-two percent of the men we reached via phone (47/51) met full eligibility criteria and were scheduled for a face-to-face assessment—72.3% (34/47) showed for their assessment. Among those who we ultimately reached via phone, participants deemed eligible via the online pre-screening route were significantly more likely to show for their face-to-face assessment than men deemed eligible directly via phone screening (72.3% vs. 47.1%, χ² (1) = 5.34, p = .02).

We reviewed data gathered during the full telephone screening and compared men who called us directly (n = 41) to those who were deemed preliminary eligible via online pre-screener (n = 51). We found no significant differences with regard to the proportion who were White/non-White (46.2% White overall), the percentage of gay men (relative to bisexual men, 71.4% overall), or the percentage of men who were HIV positive (12% overall). Similarly, those who opted to go to the online survey first did not significantly differ from those who called us directly in mean number of new male sex partners in the last 30 days (M = 5.5, SD = 6.0, range 0 to 33). There was a marginally significant association between age and whether men called the office directly or self-routed into the online screener (M_{called directly} = 40.2, SD = 13.6, M_{online survey} = 35.5, SD = 11.1, t (90) = 1.85, p = .067). Thirty four percent (n = 17) of those who showed for their face-to-face assessment lived in the borough of Manhattan, 30% (n = 15) were from Brooklyn, 16% (n = 8) from Queens, 10% (n = 5) from the Bronx, 2% (n = 1) from Staten Island, and 2% (n = 1) from outside of NYC. Three participants did not report their borough.

DISCUSSION

We adapted time-space sampling for use in the men-seeking-men section of Craigslist.org in order to recruit sexually active MSM to participate in face-to-face assessments. Our goal was to determine if men who elected to first complete an online pre-screener were more likely to show for a face-to-face assessment, compared to men who elected to call our office and screen via phone directly. Although the online pre-screener route ultimately involved more steps, we believe it was a less socially invasive approach for participants—it is easier to complete a confidential online survey than to speak with a stranger on the phone. We believe this is why a larger proportion opted for the online pre-screener. More than double the number of men self-selected to complete the online pre-screener and, from this pool of potential participants, a significantly higher proportion of those who our staff were able to schedule via phone ultimately showed for their face-to-face assessment. These findings suggest online pre-screening is a useful tool to offer potential participants when recruiting in a virtual environment. We believe these findings provide further support for a low threshold “foot-in-the-door” approach [12]. This approach suggests that participants who first complete a brief research study component are more likely than others to participate in a more intensive research component. In turn, this may improve overall external validity.

Our findings, however, should be understood in light of several limitations. These data do not generalize to all MSM, as this sample was limited to those who use the men-seeking-men section of Craigslist in NYC. Craigslist also maintains other listings (e.g., casual encounters) that are not exclusively dedicated to men-seeking-men; however, MSM may also use this forum. Although we have data on the number of potential participants who responded via email to our ad on Craigslist, we do not know how many men viewed our ad. Craigslist has automated filters preventing inclusion of a phone number or URL as text; however, it is possible to embed this information as part of a photo (i.e., the project logo) and this is something future research should consider to improve the response rate.

Craigslist is moderated by its users and our posts were not necessarily germane to the bulletin board (i.e., sexual encounters). Some of our ads were flagged for removal by the Craigslist community, particularly those posted in the borough of Queens. Further, although we adapted time-space sampling to maximize exposure of our ads, the physical location of our office (in Manhattan) may have precluded some participants in other boroughs from participating.

We noted high attrition in our efforts to reach MSM online for face-to-face assessments; however, this appears to be common across studies and with a wide range of variability. Parsons et al. [1] noted that only 54% (15/28) of men who were scheduled for a face-to-face assessment actually showed. Similarly, Fernandez et al. [3] engaged in online chatting with 735 men, of which only 24% showed. Whereas 62% of men scheduled in our study ultimately showed (50/81). However, given that recruitment often takes place in steps, there are many places where attrition can occur. For example, more than half of the men who responded to our ads via email never took further initiative to join the study. We do not have data on these men, but suspect that upon learning more (i.e., via our email reply which disclosed the length and nature of the assessment), they decided they were no longer interested. Project staff reached out to these men multiple times to reschedule. More often than not, these calls went directly to voicemail and our emails were not returned. We do not have sufficient tracking data on men who did not show to determine if there were differential reasons for no-show based on whether men self-selected to complete the online survey first or called us directly.

Participants were recruited using an adapted method of time-space sampling. An advantage of time-space sampling is its systematic approach for capturing location-based populations; however, it has the potential to oversample those who frequent venues of recruitment. Traditionally, time-space sampling involves actively approaching participants (e.g., researchers responding to men’s ads); however, we took a more passive approach (i.e., posting ads for the study). This allowed us to reach a broader population of the Craigslist community, by including men who browse through ads but may have not posted one themselves [7]. Future research should investigate if active versus passive approaches online result in different response rates and sample characteristics.

Our randomization schema was determined in part based on resources available to see participants during face-to-face assessments (we limited randomization to two postings a week). As a result, our postings to Craigslist were randomized over an extended period of time. Studies with greater resources would be able to scale up the frequency of posting/randomization to suit their needs. Nevertheless, geographic location should also be considered. A benefit to researchers is that they can use Craigslist to target ads in specific cities and areas within cities (e.g., we targeted ads based on NYC borough). However, it may be less feasible to recruit MSM in non-urban areas on Craigslist, where Craigslist communities may be less active. Researchers seeking to work in non-urban areas should first identify if the number of postings in their targeted city seems adequate—number of postings is a useful indicator of overall activity. Similarly, it may not be feasible for researchers to conduct nationwide studies using Craigslist, which would require staff to manually and singly post ads in hundreds of locales.

Conclusion

Despite these limitations, this study provides an important contribution for researchers seeking to recruit MSM online to take part in face-to-face assessments [1–3]. We found that it may be important to provide potential participants with enrollment options such as calling directly or completing a brief online pre-screener—with most selecting the latter. Although online pre-screening ultimately involved more steps in getting a participant enrolled in the study, it was a useful tool to offer potential participants when recruiting in a virtual environment. We do not suggest researchers use online pre-screening solely, as this may deter certain sets of participants who prefer other enrollment options. For example, it appeared that older men were more likely to call us directly; however, this association was marginally significant.

The headline “Participate in a Sex Research Study” resulted in significantly more email responses than our other two headlines. Although the body text of our ads (once clicked) indicated that this was a research study, this headline was the only one to specify “research.” It may be that a more direct approach of indicating “research” resulted in more clicks on our ad. Taken together, this finding suggests it is important for researchers to investigate and report on the success of various recruitment messages. Finally, more research would be needed to determine the reasons why the online pre-screener route was selected more often, and why, from this pool of potential participants, a significantly higher proportion of those who our staff were able to schedule via phone ultimately showed for their face-to-face assessment.