Seeing is believing? Unique capabilities of internet-only studies as a tool for implementation research on HIV prevention for men who have sex with men: A review of studies and methodological considerations

Abstract

Background:

In 2014, Grov and colleagues published a comprehensive review cataloguing both men who have sex with men’s (MSM’s) sexual behavior transitions in online environments from the 1990s through 2013, as well as researchers’ efforts in tandem to utilize the internet in order to engage MSM into research, treatment, and prevention.

Methods:

In the present article, we discuss historical events and research having occurred in the half-decade since the Grov et al. publication.

Results:

Notable transitions include MSM’s expanded use of geosocial networking apps, as well as other forms of social media accessed primarily via mobile devices, as well as the addition of biomedical prevention strategies (e.g., PrEP and Undetectable=Untransmittable) to the proverbial HIV prevention toolkit. In tandem, researchers have rapidly expanded their employment of internet-mediated methods for the recruitment and engagement of key populations for HIV research, treatment, and prevention. In this article, we discuss methodological considerations for using the internet to conduct HIV prevention research with MSM: (1) Sources of recruitment (e.g., geosocial apps, Facebook, crowdsourced online panels); (2) Design (e.g., cross sectional, longitudinal, diaries); (3) Incentives (including disincentivizing fraudulent participants and/or spam bots); (4) confidentiality; and (5) Representativeness.

Conclusion:

We conclude by discussing future directions in HIV prevention research in light of forthcoming technologies such as fifth generation (5G) mobile networks, combined use of self-collected biological data alongside self-report, and the utility of metadata and metaresearch to document, evaluate, and inform best practices.

Introduction

In the U.S., men who have sex with men (MSM) are disproportionally impacted by HIV, especially younger MSM and MSM of color. MSM are 44 times more likely to contract HIV than other men, and accounted for 67% of new infections in 2016 (82% of infections among men).^1,2 In 2016, the CDC estimated that, if current trends in HIV infection go unabated, one-in-six MSM will acquire HIV in their lifetime, including one-in-two Black MSM and one-in-four Latino MSM.³ These ongoing disparities, in spite of promising highly effective biomedical prevention methods (pre-exposure prophylaxis⁴ [PrEP] and treatment as prevention [TasP]^5,6), highlight the continued need for novel strategies that will engage those who would benefit most from effective treatment and prevention.

In 2014, Grov and colleagues⁷ published a comprehensive review cataloguing both MSMs’ sexual behavior transitions in online environments from the 1990s through 2013, as well as researchers’ efforts to utilize the internet to engage MSM in research, treatment, and prevention. Their piece noted how, in the 1990s and early 2000s, MSM used computer-based men-for-men websites to meet partners (e.g., manhunt.net), and more recently MSM have been transitioning to the use of geosocial networking apps installed on smartphones. In the present article, we focus on research and events having occurred in the half-decade since the Grov et al. publication. In so doing, we use an implementation science⁸ lens as it can be applied to internet-mediated research. This includes a discussion of potential pitfalls, as well as areas for caution or further consideration. Of note, however, our foci in the present articles more on the United States and other Westernized countries. For more historical context, see also Chiasson et al.,⁹ for their 2006 piece on the use of the internet for HIV research.

Recent developments in technology and HIV prevention

Today, most U.S. adults own smartphones. Back in 2011, 35% of American adults owned smartphones. By 2018, that number increased to 77%, with 95% ownership among adults under the age of 30.¹⁰ Geosocial networking apps on smartphones—i.e., apps that allow you to interact with other users based on their proximity to you, namely Grindr and Scruff—as well as other forms of social media (Twitter, Facebook, Instagram), have since established themselves as the most common mediums through which MSM meet sex partners. Grindr and Scruff have millions of daily users worldwide and have emerged as novel venues for sexual health research, recruitment, and engagement.^11-18 Beyond benefitting the research community, these apps can be used to seamlessly deliver public health prevention messaging and HIV-prevention interventions.¹⁹

Second, biomedical prevention methods—PrEP, at-home rapid HIV testing, and treatment as prevention—have entered the lexicon and the proverbial HIV prevention toolbox (which includes behavioral prevention methods such as condoms, serostatus disclosure/serosorting). Although PrEP received FDA approval in 2012, we are only now beginning to see PrEP’s potential to reduce HIV incidence at the population level.^20-22 Likewise, it was not until 2015 and 2016 that Scruff and Grindr added “HIV-negative, on PrEP,” as a profile feature its users could check-off (and filter by).^23,24

Also in 2012, the FDA approved OraQuick as an at-home self-administered rapid HIV antibody test, which can now be purchased over-the-counter at many pharmacies, or bought online for about $40-$50.²⁵ In 2015, in an effort to scale up the distribution of home test kits and increase HIV status awareness, the NYC department of health launched a giveaway to distribute in-home tests to MSM at no charge, by mail. Participants (n = 1763) were recruited over the course of 23 days, via ads on dating apps and websites.¹⁹ The success of the project represented the potential for collaboration between health departments and geosocial networking apps for HIV prevention though the distribution of HIV test kits to MSM.

Finally, and most recently, TasP—whereby health messaging explicitly states that HIV cannot be transmitted by someone who is virally suppressed due to antiretroviral medication—has increasingly been recognized as a biomedical prevention method.²⁶ In 2016, the Prevention Access Campaign issued a consensus statement indicating that individuals with undetectable HIV viral loads present negligible risk for HIV transmission to their partners.²⁷ There has since been consistent increasing support that HIV undetectable individuals do not transmit HIV,^28,29 including endorsement by the CDC.³⁰ In 2015 and 2016, Scruff and Grindr further added TasP as a profile feature users could check off. ^23,24 Also in 2016, the Prevention Access Campaign launched its slogan “Undetectable = Untransmittable” (or U=U) to highlight that treatment-as-prevention works.

In the midst of this rapidly shifting technological landscape, researchers have continued to expand their use of online technology to engage MSM in a range of activities—these include identifying and enrolling participants for in person assessments,^31-37 conducting fully online observational studies (i.e. assessments),^38-50 as well as delivering interventions entirely online. ^{34,37,45,51-60} This has afforded researchers the opportunity to identify, recruit, and study MSM who may not live in urban centers or are otherwise geographically dispersed or isolated.^7,47,48 One benefit of using technology to reach a more geographically diverse sample is to deliver at-home rapid HIV antibody tests and self-sampling kits for urethral and rectal STIs, to participants who may otherwise lack access to them.^47,48 Further, advanced technology has enhanced opportunities to quickly enroll online samples at relatively low cost,⁶¹ and to recruit participants into in-person studies at lower cost than prior field-based recruitment efforts.³⁶ In the next section, we discuss methodological considerations for conducting online studies (be they computer-based or smartphone-based).

Methodological Considerations

Researchers have expanded their employment of internet-mediated methods for the recruitment and engagement of key populations for HIV research, treatment, and prevention. However, with few exceptions (c.f. Bauermeister et al.⁶²), little has been published by way of “best practices” to ensure methodological rigor. In the next section, we provide brief overviews of important considerations for conducting online HIV prevention research, and do so through an implementation science lens. Implementation science is a relatively new field,⁶³ having emerged in in the last 15 years. It is defined as “the scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice, and, hence, to improve the quality and effectiveness of health services.”^63,64 And, “implementation science, as a science, takes as part of its mission an explicit goal of developing generalizable knowledge that can be widely applied beyond the individual system under study.”⁸ Online studies are one of several approaches that can be used by implementation researchers to gather information on the uptake and engagement of individuals in HIV prevention interventions, as well as barriers and enablers to uptake and engagement. Importantly, since recruitment requires no more than a smartphone with an internet connection, they have the potential to reach participants who are outside the health care system more easily and quickly than community-based household or telephone surveys. This provides the critical ability to assess: 1. The extent to which access to the health care system is itself a barrier to effective uptake and engagement among populations that implementers intend to reach with an intervention; 2. Whether implementation strategies housed outside the health care system (e.g., online) are effective at improving uptake and engagement. See Figure 1.

Figure 1.

Summary of traditional outcomes research and implementation science research processes highlighting differences between internet-only research and in-person or hybrid research strategies. Select web-based strengths and challenges are emphasized for both traditional outcomes research and implementation science evaluations.

Sources of recruitment

In their 2014 piece, Grov and colleagues⁷ noted that much internet-based HIV prevention research enrolled participants via computer-based men-for-men sexual networking websites, and that there was a trending shift toward the use of geosocial networking apps on smartphones. Today, it is clear that use of geosocial networking apps as well as other forms of social media (e.g., Twitter, Facebook, Instagram) is ubiquitous among MSM, and can be harnessed to engage potential participants. Scruff and Grindr support millions of daily users, whereas there are 2.4 billion Facebook users worldwide (half of all Americans use Facebook), and more than 1 billion people use Instagram each month.

Ads can be targeted to individuals based on their geolocation, and because Facebook owns Instagram, ads purchased on Facebook are currently cross-advertised on Instagram. For a period of time, Facebook advertisers were permitted to select demographic characteristics for whom they wanted their ads shown, but this feature was removed when concerns arose about the systematic exclusion of certain groups (e.g., people of color) from ads for housing—a violation of the Fair Housing Act.⁶⁵ Today, although Facebook does not allow advertisers to target ads directly toward gay and bisexual men, one can target ads to individuals who share likes and interests that are common among LGBT communities (e.g., Lady Gaga, Ru Paul’s Drag Race). Certainly, this can introduce some selection bias, as the likes and interests of LGBT communities are not homogenous.

Facebook is also host to numerous pages of groups who share common interests. For example, as of March 2019, “Gay Meme Nation” (a group devoted to gay memes) had more than 12,000 members and “PrEP facts” (a group devoted to the dissemination of PrEP) had over 21,000. However, in addition to Facebook, other social media platforms should be considered and leveraged especially considering differential usage among various age groups.⁶⁶ For instance, Instagram meme account “Best of Grindr” boasts 1.4 million followers, and numerous accounts devoted to similar content have followings upward of 100,000 users. In other areas of health research, researchers are proposing to use social media monitoring of Twitter hashtags related to specific health topics to recruit participants for studies,⁶⁷ and similar hashtag monitoring has proven feasible for gender minority research.⁶⁸

There has also been recent growth in crowdsourced online panels (e.g., Qualtrics, Amazon’s MTurk) whereby researchers can pre-populate target demographics (e.g., gender, age) and panel members can choose to opt-in and complete surveys for some remuneration.⁶⁹ Beymer et al.⁷⁰ compared samples of MSM obtained via MTurk, Qualtrics, and an HIV/STI clinic and found that the clinic-based sample demonstrated more demographic diversity and greater HIV risk behaviors when compared to the online samples. However, the online samples were more likely than the clinic sample to correctly answer an attention-check question—a pattern that has been observed by other researchers.⁷¹

One of the primary strengths of all of the aforementioned potential sources of recruitment lies in their ability to reach an exceptionally large number of individuals, quickly. However, like all recruitment strategies, consideration must also be given to any requirements related to representativeness. For example, individuals who do not use these platforms would not be represented, and methods designed to hone in on LGBT communities based on broad ‘likes’ and interests can inappropriately assume homogeneity among these communities, which is not the case.

Cross-sectional and longitudinal retention considerations

Fully online studies are well suited for cross-sectional research, whereby a limited commitment from participants is required to gather essential data to answer discrete research questions, or to take a “snapshot” of a given issue (e.g., current prevalence of sexual behavior, substance use, attitudes toward PrEP). Online studies are also well situated to follow participants for short periods of time (e.g., daily or weekly online diaries).^72-75 Some studies have managed to follow participants for several years. For example, the One Thousand Strong cohort study followed a sample of 1,071 gay and bisexual men for a period of 3 years,^47,48 whereby participants were identified via an existing marketing panel of LGBT adults. Participants in that study completed annual online surveys, as well as at-home self-administered rapid HIV testing (results submitted via photo of the test paddle) and self-collected samples for urethral and rectal chlamydia and gonorrhea testing (samples returned to a study laboratory via mail). The study’s retention (i.e., 93% at 36-months) was likely enhanced by the use of brief videos that introduced and explained the content of each assessment, incentives, and by having enrolled a sample that was predisposed to completing online surveys (i.e., via an existing panel). Presently, the NIH is supporting multiple large-scale epidemiological cohort studies that are following thousands of MSM, as well as transgender men and transgender women.⁷⁶ In these cohorts, most participants were recruited via ads on geosocial networking apps to complete annual online assessments and self-collect at-home HIV testing.

However, retention is a concern for longitudinal online studies—rates range from 15% to 93%.^47,48,77-81 Comparatively, some rates seem much lower than typical in-person study retention rates, but the very nature and logistics of an online study mean that standard retention rate goals should be re-evaluated to fit these study venues.⁷⁹ Retention practices should also be tailored to meet the unique needs of online studies.⁷⁸ Some have suggested that retention and participation in online studies, particularly longitudinal intervention studies, could be enhanced by face-to-face components such as video chatting.⁸² Of concern, however, would be that some participants many not feel comfortable with face-to-face chatting given that it will reduce anonymity.

Incentives

It is well documented that incentives improve the speed of recruitment, participation (response rates),⁸³ completion,^84-86 and longitudinal retention, and this remains true for fully online studies.⁷⁹ However, given the convenience that the internet affords study participants (to complete questionnaires and specimen collection at a location of their choosing and often at a time of their choosing), incentives for online studies do not necessarily need to be commensurate with those with a face-to-face or in-person assessments.^{79,84,85,87-92} For example, Beymer et al.⁷⁰—in their study of gay and bisexual men on MTurk, Qualtrics, and in clinic-based settings-first offered MTurk participants $0.50 to complete a ~10 minute survey. Only three MTurk participants responded in the first month of enrollment, so they increased the incentive to $1 and enrolled 264 participants in four days. They paid Qualtrics $6 per participant (of which participants themselves received $2-3) and enrolled 211 participants in seven days. In comparison, they offered $10 gift cards to participants recruited face-to-face in clinic-based settings, and over a period of nine months, 231 participants were enrolled. Enhancing participation and retention are both important considerations; however, so is representativeness. To our knowledge, there is little published by way of examining how differential incentive rates may impact relative representativeness. One study, involving face-to-face and online assessments of MSM taking PrEP noted that 30.1% of participants included financial compensation as among the reasons they joined the study (only 7.8% said it was the sole reason). However, the study also found that White participants were more likely to indicate compensation as a motivation to join than participants of color (39.2% vs. 19.6%).⁹³ Clearly, this is an important area for future research.

However, a word of caution. Given the anonymity the internet can afford study participants and monetary incentives for participating in research, it is also important for researchers to put in place mechanisms to prevent repeat (duplicate) participation, thwart fraudulent participants, and spam bots.⁸⁷ These mechanisms can include using CAPTCHA (completely automated public Turing test to tell computers and humans apart), as well as tracking IP addresses, using cookies, requiring unique email addresses, or requiring text message verification at a unique phone number.⁸⁷ Other mechanisms include the use of attention-control checks and tracking time-to-survey-completion.⁶⁹ Of caution, however, there is some evidence that attention-control checks can introduce other kinds of bias, particularly with regard to false-positives.⁹⁴ For a review of recommendations to avoid fraudulent participants in online studies, see Teitcher et al. 2015.⁹⁵

Confidentiality

As with face-to-face assessments, confidentially is equally important in online settings. The fact that participants can complete assessments in the privacy of their homes and on their personal computing devices has the potential to increase confidentiality; however, special considerations are also needed, as there are multiple opportunities for a confidentiality breach, some of which cannot be controlled by researchers in online studies. This could include evidence of participation in a study by way of browser history, unencrypted survey data being viewed by a third party during its transmission from the participant to wherever data is being stored, and hacking of data stored in a cloud server. As with face-to-face studies, researchers should take care to ensure that participant identifiable information is kept separate from participant data itself, to encrypt and password protect files, and store files on password-protected computers.

Should the online study also include self-collection of biological samples (e.g., at-home HIV testing, STI sampling), the process for directing materials to the participant should be discreet and participants should be asked if it is possible that a third party could intercept their mail (e.g., parent, roommate, partner, co-worker). Alternately, materials could be directed to a self-service parcel delivery service (e.g., Amazon Locker) where participants can pick up materials discreetly at their leisure—assuming such a service is available within a reasonable distance from the participant. Similarly, the process for returning materials to a study location or laboratory should not disclose the nature of the destination (e.g., STI testing lab) or package contents. Participants should be instructed to discreetly and safely dispose of any materials after testing is complete (e.g., HIV test kit packaging). Digital communication with participants also has the potential to breach confidentiality. For example, the content of text messages might appear on their phone even if the device is locked, thus messages should be carefully worded to protect the privacy of participants.

Furthermore, given the recent attention to scandals involving users’ data being harvested on social media and sold to third-parties,^96,97 it is also important to monitor how much end-users (i.e., potential participants) trust that any data they provide as part of a research study will be appropriately used, and that their confidentiality will be protected.⁹⁸ Rendina and Mustanski,⁹⁸ in a 2017 online study of over 11,000 MSM from across the U.S., examined participant perspectives on the issues of trust, privacy, and data sharing in online and mobile research. They found that trust in online research was greater than trust in online and mobile platforms for personal use, such as social and sexual networking apps. Participants expressed the least concerns about privacy when such data were going to be shared anonymously with researchers and the most concern when these data were going to be sold anonymously to third parties. Participants were most willing to share information they disclose publicly within the app (e.g., profile information on characteristics like age and height) and least willing to share information that could be collected by the app automatically (e.g., GPS location or device usage information).

Representativeness

As we have noted, one of the greatest strengths of internet-based studies is their ability to reach a large number of geographically diverse individuals in a relatively short period. Subsequently, it becomes easier to reach narrowly defined sub-groups and to enroll individuals who might not traditionally be represented, or are otherwise more difficult to reach. That being said, representativeness and generalizability—both central features in implementation science—are likely to be limited due to bias in response rates and sample selection bias from online platform sources. For example, Instagram and Snapchat users tend to be younger than Facebook users and many teens today are reticent to join or use Facebook, a platform their parents have been using for a decade. In fact, in 2018, Newsweek declared that “Facebook is officially for old people.”⁹⁹ Meanwhile persons of color tend to be better represented on Twitter than on other platforms.^100,101 Furthermore, in addition to representativeness of populations across varying platforms themselves, as previously noted, the representativeness of those responding on platforms must also be considered. As noted, methods designed to reach Facebook users based on popular interests among LGBT communities could systematically exclude those members who do not share those interests. Likewise, studies have also noted that features of recruitment materials themselves (i.e., using ads that feature models who are persons of color) can impact racial differences in response rate (i.e., participants of color are more likely to respond to ads where they are featured).¹⁰²

It is important to remember, however, that population-level representativeness (e.g., of all MSM) may not be the primary goal of HIV prevention research, which is often targeted to narrower and often more vulnerable groups of individuals. Moreover, it is not a prerequisite to impactful research and knowledge generation, as the participants most randomized trials in HIV prevention research are not representative of the target populations, and this not need compromise their potential usefulness as it relates to other populations and settings.¹⁰³ Any given study’s enrollment criteria (e.g., must be sexually active, must report condomless sex) will automatically circumspect the study’s ability to generalize to, for example, all gay and bisexual men, which is rarely the goal in most HIV prevention research. Further, because not all individuals have access to the internet, nor necessarily use social media or geosocial apps, fully online studies may not immediately generalize to the population who meets enrollment criteria, but were otherwise not reached via online means.

Conclusion

The internet, be it accessed via a smart phone, tablet, or computer, has established itself as a conduit through which researchers can identify potential participants to conduct HIV prevention research as well as to deliver interventions. As a result, it can be an exceptionally valuable medium through which to conduct implementation science research. Compared to studies relying on face-to-face recruitment, online studies afford researchers many advantages including geographic reach and the timely identification and enrollment of participants, and timely availability of data for analysis and dissemination, often with the expenditure of fewer resources. It can also reach populations that might be otherwise difficult to engage in face-to-face settings, and thus could make results more relevant to real world practice settings. The combined growth in popularity of geosocial networking and the forthcoming roll out of fifth generation (5G) high-speed mobile networks are likely to be essential to the future of HIV prevention research, as well as health research more broadly. As a greater number of devices become connected to the internet, this will open new opportunities for researchers to understand human behavior, beyond self-reported online surveys.

However, like any study design attribute, researchers must always evaluate the strengths of using the internet for HIV prevention research in light of its weaknesses—e.g., the potential for participants to be distracted, difficulty ensuring unique and valid participants—and new challenges with regard to privacy and data security. New approaches, such as hybrid online and in person studies, where participants are recruited online and have one or more in-person encounters, may help advance our ability to conduct rigorous prospective studies and to collect more clinical and biological data. There is also a need for metadata and metaresearch around online studies to help document, evaluate, and inform best practices, including those that can maximize retention in longitudinal studies.