Using Reddit as a recruitment strategy for addiction science research

Abstract

Reddit is a forum-based social media and message board platform that has been used in the social sciences as a recruitment source of human subject data. In addiction science, Reddit remains a viable but underutilized tool, compared to other websites (e.g., Amazon’s Mechanical Turk, Prolific). The purpose of this commentary is to provide a rationale and recommendations for the successful use of Reddit for addiction science researchers interested in adding it as a recruitment tool. We provide an example of how Reddit can be used to target specific populations of interest, such as individuals struggling with depression or alcohol use disorder. Last, we discuss the limitations of Reddit as a research tool and some considerations for future research to help promote effective use of the platform.

1. Introduction

When recruiting online for human subjects–based research studies, researchers have several options for finding participants. Researchers can partner with websites designed to supply individuals willing to complete online tasks for compensation (known as pay-for-participation, or PFP), such as Amazon Mechanical Turk (MTurk), Prolific, and Qualtrics Survey Panels. A survey created through a third-party company such as Qualtrics or REDCap can also be shared on social media websites, including Facebook, Twitter, or Instagram. The technologies allow for large-scale data collection over a potentially large geographic area, automated screening of participants to reduce researcher burden, and the ability to reach participants outside of medical settings. Access to computers and/or smartphones has increased from 8 % of the US population in 1984 to 89 % in 2016; thus, the Internet is more accessible than ever and, by extension, recruitment through the Internet is less exclusionary than ever (Ryan & Lewis, 2017). Each year, more and more studies are published in addiction science journals with crowdsourcing as keywords; that is, using large populations on the Internet to locate research participants (Mellis & Bickel, 2020). This proliferation is likely to continue with advances in technology and increased availability of Internet access for diverse populations.

Since their inception, PFP platforms such as MTurk have grown exponentially as sources of human subject research for thousands of social science papers (Chandler & Shapiro, 2016; Mellis & Bickel, 2020; Peer et al., 2017). Although these platforms may include an additional fee (Auer et al., 2021), researchers benefit from a readily available, global pool of participants managed by a service that also aids in advertising, screening, and paying participants. These platforms have been examined extensively in several excellent reviews of behavioral research broadly (Aguinis et al., 2021; Chandler & Shapiro, 2016; Cobanoglu et al., 2021; Keith et al., 2017; Litman et al., 2017; Mourelatos et al., 2016; Peer et al., 2017; Standing & Standing, 2018; van Mullen et al., 2021; Woods et al., 2015) and addiction science specifically (Mellis & Bickel, 2020; Strickland & Stoops, 2019), which suggests a sustained demand from the research community for a regulated marketplace of compensated research participants.

An alternative to PFP sources is social media platforms, the most popular of which (Facebook, Twitter, YouTube, and Instagram) host millions or billions of users (Chaffey, 2016). These platforms allow a user to search for information, form public and private groups, and link communication between platforms at no cost. Marketing services are available to researchers for a fee, allowing them to distribute advertisements for their study to individuals targeted by their location, interests, and account activity (Jamie et al., 2018; King et al., 2014). Many social media websites link to a person’s real name or likeness by default, raising concerns for privacy, confidentially, and increased risk for coercion and/or social desirability in responses (Sciutto, 2015). Privacy is always a concern in any online platform, but it is especially important with social media recruitment, as many users differ in how they understand privacy with social media (Taylor et al., 2014) and may not understand the limitations of their own privacy settings (Liu et al., 2011). Finally, users may intend to post information about themselves for the purposes of social connection and not realize this information may be used for research purposes (Gelinas et al., 2017).

Both PFP platforms and social media sources exist as viable crowdsourcing methods for psychological research broadly, and addiction science specifically. Although no online recruitment source is without limitations, the remainder of this commentary is intended to introduce Reddit.com (Reddit) as another potentially beneficial recruitment source that may be particularly helpful for targeted recruitment in addiction science. In this commentary, we provide an overview of Reddit as a website and guidelines for the use of Reddit as a recruitment tool for addiction researchers, based on our experience recruiting participants from this website. Our methods were informed by several peer-reviewed articles (Litman et al., 2017; Mason & Suri, 2012; Peer et al., 2017; Shatz, 2017; Woods et al., 2015) and other online documentation (e.g., blogs and “how to” guides, such as a resource from the University of Kent; Reddit for researchers - Help - University of Kent, n.d.) on the practical implementation of online crowdsourcing generally, and we suggest that readers interested in incorporating these techniques into their own research also review these sources. However, to our knowledge, no prior papers offer a framework for addiction research explicitly. These guidelines will help researchers to develop a plan prior to recruiting participants via Reddit.

1.1. Overview of Reddit as a crowdsourcing platform

Reddit is a free, global social media and message board platform. The website had >52 million active users and 50 billion unique monthly page views as of January 2021, making it one of the top ten largest active websites on the Internet (Reddit, 2022a). A researcher can use Reddit in one of two ways: as a way to recruit potential research participants for procedures conducted beyond the website or as a source of public, online discussions whose content can be analyzed independent of the users who contributed the posts (Proferes et al., 2021). The focus of this article is on the former method. Although it is not as widely cited as other crowdsourcing platforms, Reddit has several differences that make it unique from other social media sites and a potentially useful addition for researchers looking to expand their online recruitment efforts.

The first notable feature is that Reddit is forum-based, meaning that users engage with content that is shared in virtual communities rather than primarily between individual users and their social networks. Although users of other social media websites have the option to join topic-specific subgroups, all Reddit activity falls under a defined subcommunity. Reddit comprises >100,000 of these topic-specific subcommunities called “subreddits” that range in specificity, including humor (r/funny), geographic locations (r/Philadelphia), and shared hobbies (r/guitar lessons). Subreddits even exist for the purpose of research recruitment, such as r/SampleSize. Rather than existing as a feature of the website, these subreddits make up the structure of Reddit itself. Reddit users subscribe and submit content to subreddits that interest them, and they can comment on, rate, and share content from other users who are members of those communities. These topic-specific activities serve as the mechanism by which users connect with each other, as opposed to affiliating by virtue of their profiles or individual identities. Threads on Reddit are defined by a user’s initial post and the subsequent comments and votes on the post by other users. For researchers, this can result in an opportunity for their research post to be seen by thousands of subreddit-subscribing users with a common interest, if it is voted and commented on favorably.

A second notable feature of Reddit is the regulation of user activities, which is largely left up to the individual subreddit communities to manage. Volunteer users designated by Reddit as moderators have administrative privileges to approve and to remove subreddit content or member users based on their individual community rules. Although any user can disclose personal information if they wish to, Reddit’s overall culture favors relative anonymity through the use of user-defined profile usernames over full names and cartoon avatars over photographs, where revealing personally identifiable information is discouraged (Fabian et al., 2015). Individuals can also view posts and search through the website without an account, and although an email address is collected when creating an account, Reddit only releases user information in response to law enforcement or government agencies operating under established legal process (Reddit, 2022d). Additionally, Reddit accounts are equipped with an anonymous browsing mode that allows users to browse content on the site without logging activities such as search history, posts viewed, or account information shared with third parties. A researcher could potentially collect username information from participants and contact them through Reddit for the purposes of follow-up assessment, though not every respondent would necessarily have a username and individuals could create multiple accounts. Despite these considerations, the potential for increased Internet-anonymity may result in more propensity for unfiltered conversations on addiction-related topics on the many subreddits dedicated to asking questions, sharing experiences, and providing support to others, which has been used in studies analyzing text-mined data from the platform (Barker & Rohde, 2019; Brett et al., 2019; Krawczyk et al., 2021; Lu et al., 2019; Sharma et al., 2017; Yao et al., 2020).

A third feature of Reddit, shared with many social media recruitment methods, is the flexibility in participant payment. Although researchers have a degree of flexibility using PFP methods, such as distributing bonuses through a lottery-type structure, PFP platforms require some sort of payment to every participant for every survey completed (Auer et al., 2021). Conversely, Reddit does not have definitive requirements for researchers to pay participant volunteers and does not require a fee for recruitment (Reddit, 2022b). This allows researchers to explore diverse payment methods, including gift card lotteries and variable incentive structures during longitudinal follow-up surveys, which can be particularly attractive for researchers who wish to collect data but do not have a large operating budget (Shatz, 2017).

Finally, perhaps one of the greatest potential advantages of using Reddit as a recruitment source is the specificity of certain subreddits, where researchers can seek out specific communities that are thematically linked to their scientific population of interest. By entering key terms into Reddit’s search bar, researchers can explore both very broad and very specific communities within their topic of interest. Although no subreddit is guaranteed to have participants who exactly fit a researcher’s criteria, targeting specific subreddits may increase the chances of reaching populations of interest. For example, Reddit has been an especially helpful source in recruiting participants who use substances at historically low prevalence, such as psychedelics. Recently, researchers have been able to target specific subreddits catering to the use of psychedelics and related substances (r/microdosing, r/Nootropics, r/Psychonaut, r/RationalPsychonaut, r/tryptonaut, r/drugs, r/LSD, r/shrooms, r/DMT, r/researchchemicals) and pairing this strategy with other recruitment sources to examine psychedelic use patterns, attitudes, and correlates (Anderson et al., 2019; Johnstad, 2020; Lea et al., 2020; Polito & Stevenson, 2019; Rosenbaum et al., 2020). Some studies have been able to examine specific subpopulations within psychedelic use, such as individuals with a history of micro-dosing (Anderson et al., 2019; Lea et al., 2020; Polito & Stevenson, 2019; Rosenbaum et al., 2020). These groups can be difficult to find in large numbers in the general population, but Reddit communities make it possible to reach them.

In summary, the potential for targeting specific populations of interest and flexibility for participant recruitment and compensation may make Reddit an appealing online recruitment option for many researchers in addiction science. Next, we provide a brief case study demonstrating the potential validity of recruiting members with particular backgrounds when targeting subreddits that are consistent with the population of interest.

2. Case study using Reddit to target specific populations: depression and alcohol use

These data are from secondary analyses of a larger project that was conducted during June and July 2020, which corresponded with the early months of the COVID-19 pandemic and widespread lockdowns. The goal of the parent project was to collect a large sample of Adults living in the United States reporting a range of alcohol use and depressive symptoms. Although the parent project recruited participants from multiple social media websites, only the participants who were recruited through Reddit and who listed the specific subreddit from which they were recruited are included in these secondary analyses (N = 519). Eligible participants completed an extensive survey on Qualtrics. For the purposes of this case study, we focused on the 21-item Depression Anxiety Stress Scale-21 (DASS-21; Osman et al., 2012) and the 10-item Alcohol Use Disorder Identification Test (AUDIT; Saunders et al., 1993).

Two research assistants coded the list of subreddits into five categories: those created for the purposes of research recruitment (e.g., r/SampleSize), coronavirus-relevant content (e.g., r/CoronavirusUS), mental health topics (e.g., r/depressed), alcohol use (e.g., r/CripplingAlcoholism), and those that were location-focused (e.g., r/newjersey). The study team conducted analyses to determine whether participants recruited from subreddits related to mental health reported higher severity of depressive symptoms (based on the depression subscale of the DASS-21) than participants recruited from non-mental-health-related subreddits, and whether participants recruited from subreddits related to alcohol reported higher levels of alcohol impairment (based on AUDIT score cut-offs for alcohol use disorder) than participants recruited from non-alcohol-related subreddits.

Fig. 1 shows the mean (+/− standard error of the mean) scores from the DASS-21 depression subscale (Panel A, top) and the AUDIT (Panel B, bottom) for each subreddit category. Because the sample sizes were different across groups (see Fig. 1 for specific sample sizes), we performed nonparametric analyses, with post-hoc comparisons (i.e., Kruskall-Wallis). The results revealed significant differences between subreddit categories across both the DASS-21 and AUDIT. Participants recruited from Mental Health subreddits (Fig. 1, Panel A) had significantly higher ratings of depression than participants recruited from all the other categories (H[4] = 11.94, p = 0.018], with the exception of the Alcohol category. Alcohol use disorder is highly correlated with depression, which is likely to account for this lack of difference between the groups (McHugh & Weiss, 2019). On the other hand, participants recruited from the Alcohol subreddit groups reported significantly higher AUDIT scores than participants recruited from all of the other groups evaluated (H[4] = 47.34, p < 0.001). Although not shown in Fig. 1, anxiety and stress subscale scores were also significantly different based on group. Both subscales were highest in the Mental Health subreddit groups; however, post-hoc analyses revealed that only anxiety was consistently higher among people recruited from the Mental Health subreddits (H[4] = 9.69, p = 0.46), whereas ratings of stress were comparably high in the Mental Health, Alcohol, and Coronavirus subreddits (H[4] = 10.28, p = 0.36). Recall that participants were recruited during the early months of the COVID-19 pandemic, when lockdown procedures were still relatively new, leading to high levels of stress among most Americans.

Fig. 1.

Note: Panel A (top) represents the mean (+/− standard error of the mean) DASS-21 depression subscale scores and the dashed line represents the cut off score for severe depression (21+ represents severe or extremely severe depression). Panel B (bottom) represents the mean (+/− standard error of the mean) AUDIT scores and the dashed line represents the cut off for alcohol use disorder (15+). Each column represents a different subreddit category. DASS-21 depression scores were highest among participants recruited from the Mental Health subreddits, whereas AUDIT scores were highest among participants recruited from the Alcohol subreddits, compared to subreddits unrelated to either topic (e.g., location, coronavirus, research). Note that different sample sizes within a category, but across measures, is due to differences in response completion for each measure.

It is also important to note that the sample was largely homogenous across subreddits: majority female apart from the alcohol use subreddits (which had even gender distribution), average age around 30 years old, around 90 % white, and typically reporting highest education as some college or above. Despite the limited diversity of the sample, the results support the use of selecting topic-based subreddits for targeting participants with depression and alcohol use, arguing for the validity of recruiting participants from specific, targeted populations via Reddit.

3. Guidelines for crowdsourcing on Reddit

Although Reddit may serve as a potentially helpful recruitment source, researchers use it less often compared to other crowdsourcing platforms. Researchers may simply be unaware of Reddit or its potential as a research tool, or they may find the use of Reddit difficult because the website is not designed for research recruitment. Even if researchers can learn the basics of the platform and post their study, they may be met with challenges in how to successfully advertise and stand out among hundreds or thousands of posts competing for the attention of potential participants. Although guides for using Reddit do exist (Amaya et al., 2021; Jamnik & Lane, 2017), we still encountered many issues when we first attempted to use Reddit as a recruitment method for the case study described earlier. Therefore, in this section we provide recommendations on how to best utilize Reddit as a recruitment tool, based on our own successful experience.

3.1. Recommendation 1: be knowledgeable and respectful about rules against posting or soliciting

Researchers should first read the general rules for posting on Reddit as well as the rules for the specific subreddits on which they intend to post. These rules are variable and can include formatting guidelines as well as limits to how often a researcher can repost their survey. Many subreddits have rules against soliciting or advertising to members. Although this is usually designed to prevent businesses from advertising products, it can also prevent researchers from posting their survey links to any page that seems relevant. In some cases, recruitment is not allowed among subreddits that address sensitive topics such as substance use. Reddit’s guidelines are written with the purpose of facilitating content sharing and safe user interaction, making it common for administrators to remove content, revoke posting privileges, suspend individual accounts, or ban entire communities who put those priorities in jeopardy (Reddit, 2022b). Unfortunately, not all subreddits have guidelines for researchers, which leads to our additional recommendations for navigating successful recruitment.

3.2. Recommendation 2: limit posting to subreddits that are relevant to your research

Researchers interested in using Reddit for recruitment may be tempted to post on the biggest and most active subreddit pages with millions of members from around the world, but this strategy may not prove useful. Subreddit pages prioritize posts that are highly engaging through up-votes, shares, and comments. Posts that do not receive early and sustained positive engagement quickly move to the bottom of the front page and are superseded by posts throughout the day. Instead, use the search function to identify subreddits that would be both appropriate for recruitment and have members who would be interested in your research topic.

An excellent starting place is the popular recruitment subreddit r/SampleSize, a community of over 172,000 registered Reddit users whose purpose is to connect researchers with volunteers to complete online surveys (Jamnik & Lane, 2017; Reddit, 2022c). Researchers have supported the use of r/SampleSize and described it as being comparable to data obtained from college student convenience samples (Jamnik & Lane, 2017; Luong & Lomanowska, 2021; Record et al., 2018), indicating results may generalize in a similar manner.

3.3. Recommendation 3: work with moderators to ensure your study is sensitive to the needs of the community

After identifying appropriate subreddits for recruitment, researchers are strongly encouraged to message the moderators of the subreddit page to ask for permission to post their study link. Messaging the moderators and getting their approval ahead of time will prevent the post from being deleted and minimize the likelihood of it being flagged as inappropriate. Getting the support of the moderators from the beginning could result in them pinning the post to the top of their page’s feed for increased visibility or it could result in them making comments of approval directly on the post. Although not always required, we recommend treating these communities in the virtual-world with the respect given to communities in the physical-world by approaching leaders first and demonstrating a willingness to secure buy-in from members before posting. This is common practice in community-based participatory research, which can result in greater trust from the community being studied (Wallerstein & Duran, 2006).

3.4. Recommendation 4: check and engage with posts daily

When possible, take an active role in commenting and directly engaging with Reddit users who would comment on your post to build trust and to keep your post at the top of the subreddit feed. First, initial up/down votes and comments can have an outsized impact on the visibility of the post in the subreddit feed, so if there are initial negative reactions that are not quickly addressed, the post can lose visibility and buy-in from the community. Second, engaging with subreddit members is an opportunity to provide education about the topic and the scientific process. Most users are not familiar with terms such as “IRB approved” and therefore may not understand how their data will be protected. Actively commenting on the post provides an opportunity to answer these questions. There is also an opportunity to debrief and obtain feedback from individuals who do participate, though researchers must be vigilant in divulging information that would cause undue influence on potential participants (and asking moderators to remove such content posted by users), particularly among small populations of individuals who may share among themselves. This qualitative information may be useful in further improving recruitment, with the bonus of attracting more participant views. The ability to engage directly with participants in this virtual platform is important for success in using Reddit as a crowdsourcing platform.

Although engagement is essential to have success on the platform, researchers should also be considerate in creating professional boundaries while communicating with potential participants. Although one study post is unlikely to have a significant impact on the subreddit broadly speaking, posts made by researchers are contributing to that community and may have outsized influence by being there as a scientist and representative of the field. Professional language specifying academic research may help to cue a user to these boundaries but does not guarantee users will interact accordingly. It is critical that researchers consult with their institutional review boards before considering the use of Reddit and consult with the moderators on the platform if they have any questions regarding how to participate ethically.

3.5. Recommendation 5: remove posts after the participation window has closed and consider posting follow-up data when available

Researchers should consider deleting their post once the participation window has closed. Although it was not required by any of the subreddit pages to which we posted, deleting the post can help to reduce confusion among participants who may discover it later. Additionally, researchers should consider writing a follow-up post when results are available. In our experience, users expressed interest in reading about results after participating.

4. Limitations of Reddit (and other online crowdsourcing platforms)

In this commentary, we have discussed the strengths of Reddit as an online crowdsourcing platform and provided evidence of how researchers can use the platform to selectively target populations of interest in addiction science. However, using Reddit as a recruitment strategy has its limits as well, many of which apply to all crowdsourcing platforms. One such issue is data integrity. Self-report often provides a reliable and valid measure of drug-taking behavior (Elman et al., 2000; Napper et al., 2010), but it is possible for Internet-recruited individuals to fabricate information about their drug use to satisfy perceived study inclusion criteria (Devine et al., 2013). Some researchers have opted to indirectly verify substance use through video or sending photographs of test results (Kim & Hodgins, 2017; Raiff et al., 2021; Thrul et al., 2018). Though this is not always feasible, the ability to target recruitment to substance use–related subreddits may increase the likelihood of reaching individuals who actually engage in the substance use measured, potentially increasing validity. Surveys may also be completed by unintended parties, such as individuals who are sent the link from a Reddit user (Amaya et al., 2021), through computer-generated responses (i.e., bots, Al-Fannah, 2017), or through people who are engaging in other tasks while completing surveys (Chandler et al., 2014; Necka et al., 2016). Safeguards can be built into the survey itself to ensure integrity, such as avoiding listing specific inclusion/exclusion criteria and including a question about the recruitment source, including screening questions to ensure eligibility, and including questions that specify the intended answer (e.g., attention checks) that would indicate a human is taking the survey and is paying attention while doing so (Gummer et al., 2021).

A second issue is generalizability and sampling bias. Reddit is available worldwide, which can introduce individual variability regarding access to various substances, as well as exposure to different drug laws. Researchers can specify geographic regions in their inclusion criteria if they wish to limit this variability. Also, although Reddit provides an opportunity to collect nonclinical data outside of medical settings (where individuals may be seeking treatment or engaging in other pro-health behaviors), Reddit limits generalization in other ways. For example, posting survey information on substance use–specific subreddits would be appropriate to sample individuals who engage in substance use but would be inappropriate to estimate substance use prevalence. As with other online recruitment sources, Reddit users tend to differ from the overall population demographically; Reddit users tend to skew younger and are more educated than the general United States population (Shatz, 2017). Researchers should be transparent in the claims made from their data and avoid overgeneralizing their findings (Borodovsky, 2022).

Another sampling issue involves subreddits, a feature of Reddit that is unique to it and differs from other online sources. Each subreddit has a different topic and potentially individualized user base and culture, which means that the types of posts that get upvoted and subsequently are seen by more people may systematically vary, introducing a potential for bias. Researchers can attempt to post to numerous subreddits to keep the recruitment source as varied as possible, though care must be taken to ensure adequate engagement with moderators and potential participants within each subreddit. Additionally, researchers could ask participants to share the subreddit from which they were recruited to potentially check for bias statistically.

Finally, a potential pitfall of using Reddit involves reasonable and secure payment. PFP platforms feature an internal payment structure that is unavailable to researchers recruiting through Reddit. Although this does offer flexibility in compensation methods, researchers must be vigilant in ensuring payments reach participants and attempt to engage in fair compensation (Jamnik & Lane, 2017; Newman et al., 2021). Additionally, though Reddit boasts opportunities for participants to participate in surveys while divulging little personally identifiable information, not all payment methods allow for such anonymity. Researchers could add instructions at the end of their survey to include an email address to which they can be emailed a gift card code that the participant would be free to use without providing additional personal information. Optional compensation through email would allow individuals wishing to remain anonymous to either opt not to receive payment or create an email address for the sole purposes of receiving the compensation. Researchers who intend to use Reddit, as with any crowdsourcing platform, should take these limitations into account.

5. Conclusion

Currently, Reddit is an underutilized recruitment tool for scientific research, potentially due to a lack of awareness of forum-based recruitment methods. As more research is conducted using Reddit, researchers’ understanding of the platform will grow, resulting in greater visibility to its benefits and practical utility. Reddit has great potential as a tool for researchers to recruit study participants, particularly when interested in recruiting specific, hard-to-reach populations. Reddit has the potential to be useful in recruiting participants from specific populations relevant to addiction science. Although Internet recruitment is becoming more widely used among researchers and the literature in this area is growing (Paolacci & Chandler, 2014), more work needs to examine alternative platforms such as Reddit (King et al., 2014). Our goal is not to dissuade researchers from using other sources of recruitment, nor to claim that Reddit is immune from the limitations of remotely collected data, but rather to demonstrate that Reddit is a platform worth adding to the addiction science researcher’s repertoire.