Recruitment Strategies of Methamphetamine-Using Men Who Have Sex with Men into an Online Survey

Abstract

Recruiting hidden populations into online research remains challenging. In this manuscript, we report lessons learned from our efforts to recruit methamphetamine-using men who have sex with men. Between July and October 2012, we implemented a four-phase recruitment strategy to enroll a total of 343 methamphetamine-using MSM into an online survey about recent substance use, sexual behavior, and various psychosocial measures. The four phases were implemented sequentially. During phase one, we placed advertisements on mobile applications, and during phase two, we placed advertisements on traditional websites formatted for browsers. During phase three, we used e-mail to initiate snowball recruitment, and during phase four, we used social media for snowball recruitment. Advertisements on mobile devices and websites formatted for browsers proved to be expensive options and resulted in few eligible participants. Our attempts to initiate a snowball through e-mail also proved unsuccessful. The majority (n=320) of observations in our final dataset came from our use of social media. However, participant fraud was a concern, requiring us to implement a strong participant verification protocol. For maximum recruitment and cost-effectiveness, researchers should use social media for recruitment provided they employ strong participant verification protocols.

INTRODUCTION

Recruiting participants into online research is challenging. Online recruitment efforts are frequently expensive and result in convenience samples drawn from a sampling frame that can range from an unspecified number of Internet users to a more defined, but limited number of members of a particular online community. Persons excluded from sample frames because they lack Internet access or persons with access who lack an active profile on a particular website being used for recruitment raise concerns about sampling bias (Sinclair et al., 2012). Low response rates (Im et al., 2006; Jenkins, 2012; Miller et al., 2010; Tuten, 2010; Vehovar and Manfreda, 2008) and professional participants who misrepresent themselves to get into a study for which they are ineligible (Jenkins, 2012) compound concerns about the representativeness of samples recruited using online methods.

Researchers have responded to online recruitment challenges by recommending strategies to increase response rates and the representativeness of online samples. To increase response rates, researchers have recommended tailoring messages to both the audience and the medium so that participants can quickly identify the trustworthiness of the research team and researchers can quickly begin to establish rapport (Hershberger et al., 2011; Riggle et al., 2005; Temple and Brown, 2011). Information about the research team, purpose of the study, and expectations of participation in the study can be placed on a website or social media page, e.g., Facebook, rich site summary (RSS) feeds and, if using social media, “liking” pages and cross-posting information from other organizations can communicate to a potential participant some of the study-related values of the research team.

Researchers have also recommended employing multiple recruitment methods (Temple and Brown, 2011), including a combination of offline and online methods (Cook et al., 2009; Gordon et al., 2006; Hershberger et al., 2011; McClure et al., 2006). Such methods include e-mailing members of a closed network who have agreed to be contacted (Temple and Brown, 2011), posting recruitment information on blogs or message boards (Riggle et al., 2005), purchasing website banner advertisements (Bull et al., 2008; Buller et al., 2012) or pop-up windows on websites frequented by members of the target population (Graham et al., 2006), and issuing electronic coupons to key informants who have agreed to assist in implementing online respondent-driven sampling (Evans et al., 2011). When employing offline recruitment methods, quick response (QR) codes can supplement online efforts by providing members of the target population with a link to the study website or social media page (Buller et al., 2012). The challenge with employing multiple recruitment methods is that the research team frequently loses the ability to have a defined sampling frame and to assume homogeneity within the sample. Some methods also increase the risk of attracting fraudulent participants, increasing the need for a protocol to verify legitimate participants (Bull et al., 2008; Konstan, Rosser, Ross, Stanton, & Edwards, 2005). Despite the inability to quantify the sampling frame, risks to homogeneity, and participant fraud, for some hard-to-reach populations the multi-method approach is necessary (Jenkins, 2012).

In this manuscript we report lessons learned from our efforts to recruit methamphetamine-using MSM into a national online survey about substance use and sexual behavior. By sharing our experiences, we hope to contribute to the academic conversation about recruitment and to offer advice to researchers recruiting hard-to-reach populations into an online study. When recruiting, we had to balance our desire to recruit from websites accessed by members of our population with the realities of a limited budget and a desire to not directly compensate online sex-seeking websites that actively promote or condone unprotected anal sex, a risk factor for Human Immunodeficiency Virus (HIV) transmission (Centers for Disease Control and Prevention, 2012).

METHOD

Study Design

In the parTy Study, we collected formative data to inform the content and design of a mobile-health intervention for methamphetamine-using MSM. Between July-October 2012, a total of 343 methamphetamine-using MSM responded to questions about recent substance use, sexual behavior, and various psychosocial measures. Eligibility criteria included self-identifying as male, age 18 or older, living in the United States or one its territories, having sex with a man in the last 30 days, and having used methamphetamine in the last 30 days. Participants were compensated $25 for completing the survey. The median completion time was 21 minutes. The institutional review boards at the University of Minnesota and The University of Texas Health Science Center at Houston approved all study procedures.

Recruitment

The recruitment of methamphetamine-using MSM into this study occurred in four phases. During phase one, we contracted with an advertising network experienced in targeting gay men to run banner advertisements on nine mobile websites or applications used by MSM to meet other men. We asked the advertising network to run our advertisements during evening hours because we thought this would be the time when the majority of men would frequent the targeted mobile websites or applications. When someone clicked on our advertisement, they were directed to a university webpage and asked to provide their e-mail address. Persons who provided their e-mail address were sent the eligibility screener, consent, and if eligible, the survey. We were charged for each click on our advertisement regardless of whether the person who clicked on it chose to provide us with their e-mail address.

Our phase two recruitment protocol was similar to our phase one protocol. We contracted with the same advertising network. Instead of targeting mobile websites or applications, we targeted eight websites formatted for browsers that were used by MSM to meet other men. Since website browsers allow for more variety in the dimensions of advertisements, we provided the advertising network with advertisements of various dimensions. Because persons who clicked on the banner advertisements formatted for browsers were probably using a desktop, laptop, or tablet, we did not ask them to provide their e-mail. Instead, clicks on an advertisement directed to the eligibility screener, consent form, and survey. As in phase one, we were charged for each click on our advertisement.

Phases three and four relied on snowball recruitment. During phase three, we e-mailed 14 colleagues we knew at other universities and community-based organizations who worked with substance-using MSM. The e-mail asked our colleagues to forward an e-mail containing information about the study and a link to the eligibility screener, consent form, and survey to their professional contacts and e-mail distribution lists. During phase four, we took a more active role, creating a Facebook page and Twitter account. Through these media, we connected to 45 individuals and organizations that included substance-using MSM in their membership or who do outreach to substance-using MSM. We also distributed recruitment information through regular Facebook posts and tweets.

Verification of Participants

We used a three-step process to identify and remove possible fraudulent participants from our dataset. Step one occurred at the end of the eligibility screener. The last item in the eligibility screener summarized a participant’s responses to all screening questions and asked the enrollee to confirm all responses. Once the participant confirmed his responses were correct, he could no longer access the eligibility screener. Step two notified participants of their eligibility; ineligible persons were directed to an exit screen that thanked them for their interest in the study and notified them that they did not meet eligibility criteria. We programmed our online survey to prevent persons from toggling between the eligibility and confirmation screen.

In step three, we conducted our cross-validation and de-duplication protocol, adapted from previous studies (Konstan et al., 2005). Our protocol for verifying study samples involved comparisons of data within and between participants (Konstan et al., 2005; Bowen et al., 2008; Bauermeister et al., 2012). At analysis, we validated eligibility by comparing responses to the eligibility-screening questionnaire with responses to the survey for age, US residence, and location of IP address. We also searched for duplicates among the IP addresses, e-mail addresses, and key demographic characteristics of persons who completed the survey. We removed ineligible or duplicate observations from the final dataset.

RESULTS

The 343 participants included in the final dataset were from across the United States (see Figure 1). Because websites charge per click on an advertisement regardless of whether a person chooses to participate in a study, phases one and two proved to be expensive options which resulted in few valid observations (8 each from the mobile and browser banner ads; see Table 1). Both strategies also attracted a high proportion of enrollees who failed to pass our verification protocol (33% for mobile ads and 69% for browser ads). Phase three was the least successful strategy; it resulted in only eight inquires about our study and no one attempted to complete the eligibility screener. The great majority (n=320 or 93%) of observations in our final dataset came from our phase four recruitment efforts. Of the 543 attempts to complete the survey in phase four, 427 (79%) passed the eligibility screener. However, 107 (25%) of “eligible” phase four surveys failed to pass our participant verification protocol. Although the use of social media for recruitment resulted in the greatest number of fraudulent participants, it resulted in proportionally fewer cases of fraud (mobile=33%, browser=69%, and social media=25%).

Figure 1.

Location of participants according to their Internet protocol (IP) address (N=343)

Table 1.

Comparison of four-phase recruitment efforts (N=343)

	Mobile banner ads	Browser banner ads	E-mails	Social media
Cost	$3,000.00	$1,500.00	Free	Free
Advertisement clicks	1679	950	--	--
Attempts to complete the survey	28	190	0	543
Passed eligibility screening	15	16	0	427
Completed but dropped during verification	5	11	0	107
Completed and eligible (included in final sample)	8	8	0	320
Cost per eligible participant	$375.00	$187.50	Free	Free
Proportion of fraudulent cases	33%	69%	--	25%

Note: The proportion of fraudulent cases = number of observations that completed but were dropped during verification/number of observations that passed the eligibility screener.

DISCUSSION

Snowball recruitment through social media proved the most useful strategy to recruit methamphetamine-using MSM into an online study. However, its success was dependent on the collaboration of numerous individuals and organizations. Given the dramatic difference in outcomes, we encourage other researchers wanting to recruit online, and particularly researchers with limited budgets, to direct their online recruitment efforts to social media. We also recommend implementing strong participant verification protocols.

Using advertisements on mobile devices and websites formatted for browsers to recruit participants into our online survey was expensive and resulted in few observations. Had our instrument been brief enough to allow participants to click on the advertisements, consent, and answer questions on their mobile phones, we might have had more success with the mobile advertising method. In our previous work (Rosser et al., 2009; Rosser et al., 2011; Wilkerson et al., 2012), placing advertisements on websites formatted for browsers was an effective strategy to recruit MSM. However, for this study, the strategy did not produce the desired results. Without a sufficient advertising budget, we were unable to run advertisements for the number of weeks required to recruit our desired sample size.

It is unclear why our e-mail snowball failed to result in new observations. We have good relationships with the 14 colleagues we contacted and we believe they forwarded our e-mail to their colleagues. Individuals who received the e-mail might have never viewed it if their spam filter placed the e-mail in a junk folder. Alternatively, individuals might have deleted the e-mail without opening it, or they might have opened the e-mail, intended to complete the survey at a later date, but then forgot about it. We concluded that one e-mail request is insufficient, and we were not willing to ask our colleagues to repeatedly e-mail their contacts. We thought it would be overly burdensome to them and we thought their contacts would not welcome numerous e-mails.

In contrast to the limited contact approach we took with the e-mail snowball, we took a more assertive approach when snowballing with social media, posting and tweeting daily, and asking our colleagues to do the same. With regular postings to social media, exposure to study information was potentially increased, resulting in greater participation. This assertive approach, although labor intensive, proved most effective.

The three methods that resulted in participants also resulted in surveys we rejected as suspicious. The identification and removal of suspicious participants is a critical step in conducting credible research (Bull et al., 2008; Jenkins, 2012; Konstan et al., 2005). Although the most fraudulent participants were from social media, advertising on mobile devices and websites formatted for browsers resulted in proportionately more fraud. Even though these proportions should be interpreted cautiously because of the small number of eligible participants, social media might result in less fraud than advertising on mobile devices or websites formatted for browsers. More research on best practices for online recruitment is needed. To move the field toward the identification of best practices, Internet-based researchers should publish their protocols and share novel techniques to validate the authenticity of participants recruited into online studies. Participant verification protocols need to address two validity challenges: de-duplication to ensure that each enrollee only participates one time, and cross-validation to ensure that responses are consistent and credible (Konstan et al., 2005; Rosser et al., 2011).

We wish to highlight three limitations to our findings. First, the generalizability of our findings is unknown. Methamphetamine-using MSM in the United States represent a distinct sub-population. We simply do not know if these results would be similar for other populations. Second, our recruitment was specific to completing an online cross-sectional survey. It might be that recruitment for other study designs would produce different results. As other researchers publish their recruitment strategies, the empirical literature about online recruitment will become more complete. Third, the data on participant verification are dependent on the items used. The more criteria used will probably result in a greater detection of suspicious responses.

CONCLUSIONS

Recruiting hidden populations—such as methamphetamine-using MSM—into online research is a challenge. Our experience suggests researchers should prioritize approaches that utilize active recruitment on social media to passive recruitment strategies using advertisements on mobile devices and websites formatted for browsers, incorporating the latter only if budgets are sufficiently large. Social media allows for the organic dissemination of and repeated exposure to recruitment materials to persons who might qualify for the study. The use of innovative social media strategies appears to be cost-effective. Potentially, recruitment with social media could result in less fraud. Our success shows researchers with limited funds and the ability to implement a participant verification protocol can use social media to recruit large online samples of substance users.