Maximizing response rates to ads for free at-home HIV testing on a men-for-men geosocial sexual networking app: Lessons learned and implications for researchers and providers

Abstract

Internet-based recruitment can be effective in reaching large numbers of geographically diverse individuals. Geosocial sexual networking apps on smartphones have emerged as the modal way in which men who have sex with men (MSM) meet sex partners, and as venues for sexual health research. We report on the performance of three types of ads—text-only, text with male figure (no face), and text with male figure (with face)—used on a geosocial sexual networking app to advertise free at-home HIV testing and enroll in an online study. We ran five two-week-long ads on a popular MSM geosocial app between fall 2017 and spring 2018 (~2.19 million impressions). Ads were evaluated in terms of the click-through rate (CTR = advertisement clicks/advertisement impressions), conversion rates (CR = number of enrolled participants/ad-generated clicks), cost per enrolled participant, and demographic composition of survey respondents. We enrolled n = 4,023 individuals, n = 2,430 of whom completed HIV testing—$6.21 spent on advertising per participant enrolled and $10.29 spent for everyone who completed HIV testing. Cost per enrolled participant was associated with the content of the ad used—ads featuring male figures (with or without a face shown) were more cost efficient than ads featuring text alone. These ads also out-performed text-only ads across a range of metrics, including responsiveness among younger MSM as well as MSM of color. Advertising materials that combine text with images may have greater appeal among priority populations.

INTRODUCTION

More than three decades into the HIV epidemic, sexual minority individuals (i.e., gay and bisexual men, and transmen and transwomen who have sex with men) are still disproportionately impacted by HIV—particularly persons of color and younger persons (CDC, 2016a, 2016c, 2017). In 2016, the Centers for Disease Control and Prevention (CDC) estimated that, if incidence trends went unabated, one-in-six men who has sex with men (MSM) would acquire HIV in their lifetime, including one-in-two Black MSM and one-in-four Latino MSM (CDC, 2016b, 2016c). Suffice to say, it remains vital to determine the most effective strategies to engage individuals vulnerable to HIV in prevention, and to link HIV-positive persons to care.

Internet-based recruitment methods can be effective in reaching large numbers of geographically diverse individuals in a relatively short period and can also be effective methods of recruiting populations with characteristics or behaviors that are thought of as “hard-to-reach” populations (e.g., drug users) (Chiasson et al., 2006; C. Grov, Breslow, Newcomb, Rosenberger, & Bauermeister, 2014; C. Grov et al., 2016; Mustanski, 2001; Topolovec-Vranic & Natarajan, 2016; Vial, Starks, & Parsons, 2015). In particular, the Internet serves as a gathering place for rare or otherwise invisible populations that can be hidden in plain sight because they do not possess physical characteristics that visibly differentiate them from other populations. When it comes to recruitment populations vulnerable to HIV, both online and offline recruitment strategies have benefits and disadvantages pertaining to the priority population desired (Vial et al., 2015). Online recruitment strategies are particularly effective in reaching younger MSM, HIV-positive MSM who may not be currently linked to care, and substance using MSM (Parsons, Vial, Starks, & Golub, 2013; Vial et al., 2015). Other research has investigated demographic characteristics and sexual behavioral differences produced by various online sites or social media platforms (e.g., Facebook and dating apps) (Vial, Starks, & Parsons, 2014), with findings suggesting that the most effective online recruitment tools for MSM were dating or sexual networking websites (Vial et al., 2014).

Nevertheless, the landscape of online recruitment is shifting. In 2008, the first “app” store—a digital market where smartphone users can download third-party applications that take advantage of the phone’s features, namely its mobility—launched (Silver, 2018). In the decade since, we have seen growth in popularity of geosocial networking apps like Grindr, Jack’d, Hornet, and Scruff. On these apps, users can search for and communicate with each other based on geo-proximity. These types of apps have subsequently shifted MSM’s use of desktop computer-based websites (e.g., Manhunt.net circa 2008), for the purposes of sexual networking, to smartphone-based apps and web-apps. Consequently, there has been an increase in research of geosocial networking app users (Goedel & Duncan, 2015; Goedel, Halkitis, Greene, Hickson, & Duncan, 2016; C. Grov, Breslow, et al., 2014; Winetrobe, Rice, Bauermeister, Petering, & Holloway, 2014); however, little is known about the best recruitment strategies to engage MSM on these new platforms. Given their smaller size, handheld devices have a decisively different user experience than static websites. For example, one may open and close an app dozens of times throughout a day, as opposed to leaving a website open and running in the background for hours. This presents challenges for designing ads that display well within the app, as well as the frequency with which advertisements appear and re-appear for users on their phones and handheld devices.

Despite the Internet’s and, more recently, apps’ suitability for researchers to reach large numbers of geographically diverse MSM (C. Grov, Rendina, et al., 2014; C. Grov, Rendina, Ventuneac, & Parsons, 2013; Lelutiu-Weinberger et al., 2015; Rendina et al., 2014; Sineath et al., 2013), concerns and challenges remain regarding representation of target populations, particularly those most vulnerable to HIV (including younger MSM and persons of color) (C. Grov et al., 2016; Ricca, Hall, Khosropour, & Sullivan, 2016). Historically, persons of color have been more challenging to engage and retain in research studies (Khosropour, Johnson, Ricca, & Sullivan, 2013; Khosropour & Sullivan, 2011; Sharma, Sullivan, & Khosropour, 2011). Some, albeit limited, research has reported on recruitment methods necessary in order to generate interest and maximize participation (Yancey, Ortega, & Kumanyika, 2006). For example, Sullivan et al. (2011) found that racial minorities being recruited for HIV research had higher odds of clicking on advertisements that featured a racial/ethnic minority model, rather than a white model. However, consideration for user behavior must be accounted for when choosing the appropriate applications on which to advertise (Duncan et al., 2018). In general, app-using MSM tended to be white, better educated, and have a higher annual income (Zou & Fan, 2017). A recent study by Duncan et al (2018) found that while Grindr may be the most commonly used geospatial networking app (Datingsitereviews.com, 2019) it is not necessarily the top used app among young Black MSM in the Southeastern U.S. (Duncan et al., 2018). Sun et al (2018) found similar results suggesting that multiple apps (including Grindr) should be considered when attempting to recruit transwomen of color (Sun et al., 2018). Similar to app users utilizing multiple apps to broaden their networks (Holloway et al., 2014), both of these studies support a broad advertising strategy across multiple apps is most effective to be inclusive of a diverse set of study participants.

Considerable marketing campaigns have relied on social identity theory, which suggests that persons more readily adopt behaviors exhibited by members of a social group to which they identify (Billig & Tajfel, 1973; Tajfel, 1982; Tajfel, Billig, Bundy, & Flament, 1971). This theory has been leveraged for various marketing strategies for differing health outcomes including direct-to-consumer advertising of pharmaceuticals (Mastin, Andsager, Choi, & Lee, 2007), HIV/STD interventions (Sullivan et al., 2011), and anti-smoking campaigns (Moran & Sussman, 2014). Additional research has investigated more subtle influences on health and health behavior related to group-oriented marketing strategies such as racial/ethnic depictions in health-positive (e.g., health foods, weight-loss products) or health-negative (i.e., fast foods, alcohol) and group-oriented magazine placements (Duerksen et al., 2005; Godbold Kean & Prividera, 2007).

In addition to culturally diverse representation, design, layout, and inclusion of images also appear to influence the effectiveness of ad campaigns (Bennett, 2007; Smith, 1991). In health-focused campaigns, design and layout was emphasized in initial studies investigating the use of graphic warning labels to discourage combustible cigarette smoking (Strasser, Tang, Romer, Jepson, & Cappella, 2012). The warning labels that contained graphic images were more successfully recalled at later interview times than those warnings that only contained text. Beyond tobacco product marketing, health promotion campaigns have also utilized marketing materials to promote health-positive behaviors such as the HPV public vaccination campaign (D’Souza, Zyngier, Robinson, Schlotterlein, & Sullivan-Mort, 2011). In addition to using health behavior theories, the HPV campaign highlighted that colorful ad designs and inclusion of famous athletes helped the campaign to be recognized, but did not necessarily translate into remembering the details of a specific campaign (D’Souza et al., 2011).

Current study

Although there is some research evaluating recruitment strategies in online platforms, to our knowledge, there is little published by way of “best practices” on the use ads in geosocial networking apps. To address this gap, we report on the performance of three types of ads—text-only, text with male figure (no face visible), and text with male figure (with face visible)—we used on a geosocial sexual networking app to advertise free at-home HIV testing and encourage participation in an online research study. Given what is known both from the broader advertising literature as well as studies on the engagement of MSM in HIV research, we anticipated that ads featuring a male figure would perform better than text-only ads. Our goal was to inform HIV prevention researchers and providers on the best practices for reaching individuals vulnerable to HIV using geosocial sexual networking apps.

METHOD

Cohort recruitment, enrollment, and surveys

This study uses data collected as part of the Together 5000 study, a U.S. national, internet-based cohort study of men, transmen, and transwomen who have sex with men. The cohort has been described in detail elsewhere (Christian Grov et al., 2019; Nash et al., 2019). In brief, enrollment began October 2017 using ads on men-for-men geosocial sexual networking phone applications, and concluded in June 2018. Eligibility criteria specified that participants were men, transmen, or transwomen; aged 16 to 49; and at risk for HIV (see Grov et al., 2019 for full criteria). Of note, our recruitment strategies were targeted to reach cis-men who have sex with men, but our enrollment criteria did not exclude transmen and transwomen who otherwise met study criteria.

Participants clicking one of our ads were routed from the sexual networking apps to a secured, informed consent and enrollment survey webpage that collected information on eligibility criteria. The first page of the enrollment survey was the informed consent form and this did not differ across ads. Although there was no compensation for completing the enrollment survey, consented and enrolled participants were invited to complete subsequent study procedures (e.g., an additional survey, at-home HIV testing) that were incentivized [see Nash et al., 2019].

Analysis Plan

We sought to evaluate the effectiveness of the digital ads we used to enroll participants—a text-only ad, an ad that featured a partial male figure/model (i.e. no face shown), and two ads featuring full male figures/models (i.e. with face shown). See Figure 1. All ads mentioned free at-home HIV testing and participation incentives.

Figure 1.

Four advertisements used across five ad campaigns

For the present study, we limited the ad-generated responses to one of the two geosocial sexual networking apps used to recruit participants.¹ On this app, we launched five, two-week long advertising campaigns featuring four different digital ads between December 2017 and May 2018. Ads were broadcast nationally in the U.S. and its territories. During each advertising period, when a user opened the app, they were shown a full-screen version of our ad. Visitors could then either click the ad, which would direct them to the study landing page/enrollment survey, or dismiss it to enter the app. If a user dismissed the ad, it moved to their in-app inbox. The ad remained active for the duration of the advertising campaign period such that any user opening the app could be shown our ad once. The company managing the app estimated each of our five ad campaigns were delivered to approximately 437,000 users in the U.S. (total impressions ≈ 2.19 million).

Each ad campaign cost $5,000. For each campaign, we evaluated the number clicks received (estimated by the number of participants landing on our survey page), the number of participants who completed the survey, and the number of participants who met our eligibility criteria and were ultimately enrolled. We then calculated the cost per enrolled participant for each ad campaign (cost per enrolled participant = cost of each ad campaign ($5,000)/number of participants enrolled per ad campaign). We estimated select participant demographic characteristics—percentages of participants under the age of 25, persons of color, and HIV-positive participants—and completion of other study benchmarks—participants who completed the secondary psychometrics questionnaire and returned self-administered HIV test kits—by study campaign and type of ad. Finally, we calculated the click-through rate (CTR = advertisement clicks/advertisement impressions) and conversion rates (CR = number of enrolled participants/ad-generated clicks) for each type of ad.

RESULTS

The five ad campaigns generated a total of 20,539 survey attempts (0.94% of impressions), from an approximately equal number of impressions delivered per campaign. Of these, 10,943 (53.3%) completed the survey. The vast majority of those not completing it stopped immediately (i.e., on the informed consent page), most likely because they had clicked the link unintentionally. Of those completing the survey, 4,023 were eligible and enrolled in the study, and 60.4% of these participants completed in the optional at-home HIV testing.

Table 1 shows the number of participants clicking our ads (i.e., started the survey) for each of the five 14-day ad campaign periods. Regardless of campaign, there was logarithmic decay in the number surveys started in each subsequent day of our ad campaigns. Campaign 3 (text-only) was the least effective ad overall, having generated only 1,676 clicks; whereas, campaign 4 (male figure without face) was the most effective, having generated 8,726 clicks.

Table 1.

Logarithmic decay in the number of survey attempts by day for each ad campaign

Campaign	Day
	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15+
1 Figure w/ Face	1,929	900	212	28	5	4	1	1	3	2	1	12	16	0	0
2 Figure w/ Face	1,158	949	332	124	106	84	68	45	40	13	2	5	4	5	1
3 Text-only	588	267	139	116	147	135	121	57	23	18	25	18	11	10	1
4 Figure w/o Face	4,135	992	643	605	498	562	662	211	102	81	51	46	55	68	15
5 Figure w/ Face	1,446	570	446	357	288	303	275	109	74	51	42	34	37	28	27

Table 2 displays the relative effectiveness of each of the five advertising campaigns across multiple metrics. As alluded to in the prior paragraph, campaign four (male figure without face) was the most cost efficient and successful ad. Campaign 3 (text only) was the least effective across nearly all metrics. Of interest, however, was that consented and enrolled study participants responding to the text-only ad had the greatest proportion of HIV-positive lab results (6.3% of participants, χ²(4) = 14.6, p = 0.006).

Table 2.

Relative effectiveness of ad campaigns in generating survey responses and enrolled participants

Camp.	Start Date	End Date	Survey Attempts	$/Attempt	Surveys Complete	$/Complete	Enrolled	$/Enrolled	HIV Kit Returned	$/Kit	% < 25 years old	% Persons of Color	% HIV-Positive
1. Figure w/ Face	12/11/17	12/25/17	3,114	$1.61	1,713	$2.92	674	$7.42	487	$10.27	18.8%	37.5%	1.6%
2. Figure w/ Face	12/26/17	1/9/18	2,936	$1.70	1,642	$3.05	596	$8.39	394	$12.69	22.0%	38.1%	4.8%
3. Text-only	1/31/18	2/14/18	1,676	$2.98	871	$5.74	315	$15.87	221	$22.62	14.3%	39.1%	6.3%
4. Figure w/o Face	4/16/18	4/30/18	8,726	$0.57	4,585	$1.09	1659	$3.01	966	$5.18	17.1%	38.5%	3.0%
5. Figure w/ Face	5/4/18	5/18/18	4,087	$1.22	2,132	$2.35	779	$6.42	362	$13.81	17.2%	42.5%	2.5%

Table 3 groups the five campaigns into three, based on the type of ad (text-only, vs. figures with face, vs. figure without face). Ads featuring male figures outperformed the text-only ad. Among ads featuring a male figure, those figures without a face were more effective in terms of cost per survey attempt ($0.57 vs. $1.48) and cost per survey completed ($1.09 versus $2.73) compared to those featuring a figure with face. However, the ads featuring a male figure with face were more cost efficient in terms of costs per number enrolled ($2.44 vs. $3.01), costs spent on ads per HIV test kit returned ($4.02 vs. $5.18) and most effective in enrolling young participants (19.1% vs. 17.1%), as well as men of color (39.6% vs. 38.5%).

Table 3.

Relative effectiveness of ad campaigns based on the type of ad used (text-only, figure with face, figure without face)

Ad Type	Spent	Survey Attempts	$/Attempt	Surveys Complete	$/Complete	Enrolled	$/Enrolled	HIV Kit Returned	$/Kit	% < 25 years old	% Persons of Color	% HIV-Positive
No Figure (text only)1	$5,000	1,676	$2.98	871	$5.74	315	$15.87	221	$22.62	14.3%	39.1%	6.3%
Figure w/ Face2	$15,000	10,137	$1.48	5487	$2.73	2,049	$2.44	1,243	$4.02	19.1%	39.6%	2.9%
Figure w/o Face3	$5,000	8,726	$0.57	4585	$1.09	1,659	$3.01	966	$5.18	17.1%	38.5%	3.0%

¹Campaign 3

²Campaigns 1, 2, & 5

³Campaign 4

The CTRs for each type of advertisements were 0.38% for the text-only ad, 0.77% for the ads with full figures, and 2.00% for the ad featuring a male model without a face. The CRs for each of the ad types were 18.8%, 20.2%, and 19.0%, respectively.

DISCUSSION

We report on our advertising efforts on a geosocial networking app to engage at-risk individuals to participate in a sexual health study—an area in which there is little published research. In total, we spent $6.21 on advertising for every participant enrolled and $10.29 for every participant who completed HIV testing. However, cost per enrolled participant varied with the content of the ad used. Although all ads specified free HIV testing and remuneration as incentive, ads featuring male figures (with or without a face shown) were more cost efficient than ads featuring text alone. These findings are consistent with other HIV researchers who have noted target populations are most likely to respond to ads featuring individuals who look like them (Vinjamuri, 2015). Further, our findings highlight that ads featuring images were more influential than text alone (Bennett, 2007; Smith, 1991), and provide further support for social identity theory (Billig & Tajfel, 1973; Tajfel, 1982; Tajfel et al., 1971).

Although it is well documented that the Internet is an effective medium through which to engage MSM in HIV behavioral research (C. Grov, Rendina, et al., 2014; C. Grov et al., 2013; Lelutiu-Weinberger et al., 2015; Rendina et al., 2014; Sineath et al., 2013), there is limited research on cost comparison, and virtually none on advertising via apps. Grov et al. (2009) reported on recruitment costs associated with direct marketing (via targeted sampling) of a substance use intervention to MSM compared to a foot-in-the-door strategy (whereby participants were asked to complete a one-time survey and then offered an opportunity to join a clinical trial). They reported it was substantially cheaper to use a foot-in-the-door approach ($356 versus $497 per enrolled participant spent on recruitment/advertising). Although the target population (at-risk MSM) and longitudinal nature of their study was similar to ours, their study was enrolling for a behavioral intervention (whereas ours is observational). Nevertheless, it is worth highlighting the cost difference between the two studies ($356-$497 per participant versus $6–10 per participant). A similar, Facebook-oriented recruitment strategy for a study administering mental health surveys found a similar cost-per-(initial) survey completion of $9.82 after recruiting through the social network (Batterham, 2014).

In addition to reporting on the cost-efficiency of online advertising, we also sought to compare the effectiveness of ads that used full and partial model images on recruitment. Some studies have looked explicitly on the effects of images vs. text on influencing health behaviors, such as the influence of graphic images used on cigarette packaging (Strasser et al., 2012). This study found that graphic warning labels left more of a lasting impact and were better recalled than text-only advertisements (Strasser et al., 2012). Other studies simply used either text-only or image-based advertising. The CTRs for studies that only used text-based advertising were relatively low (0.08%) (Bakker, Heskes, Neijt, & Kappen, 2004) compared to studies that used images on their ads (3.5%) (Fenner et al., 2012). One study has suggested that positive advertising images increase positive affect (Chowdhury, Olsen, & Pracejus, 2011). This may be relevant to the current study in that our advertisements featuring attractive male models may have aided in study recruitment. However, there is limited research on the influence type of images (partial vs. full model) have. Additionally, our ads were designed to feature racially diverse MSM to support our study aim of having our total enrollment reach 50% persons of color. We used ads that were inclusive of men of color to reach this goal following the tenets of social identity theory. We were successful in reaching our enrollment goal of 50% persons of color and surmise that our inclusive advertising helped us reach these goals; however, our study was not designed to test this nor to test the impacts of social identity theory on our marketing. Our study contributes to this limited body of research by suggesting that ads featuring full male models were more effective in engaging participants through multiple levels of study participation (i.e. enrollment, survey completion, and HIV testing).

Our findings should be understood in light of their limitations. Although the ads we used shared common elements (e.g., mentioned the same incentive and HIV-testing), they differed in subtle ways (e.g., font, colors) in addition to major ones (e.g., model vs. no model). We believe the performance of ads were more so a factor of the major differences between the ads, but we recognize that subtle variations can impact response (Jacques, Perry, & Kristensson, 2015; Magnini & Kim, 2016; Pochun, Brennan, & Parker, 2018). As noted in the advertising literature (Jacques et al., 2015; Magnini & Kim, 2016; Pochun et al., 2018), it would also be important to experimentally evaluate the role that additional subtle variations (e.g., with or without smile, with or without eye contact, varying states of undress, serious tone versus playful, subtle variations in the messaging of the ad) can play in ad performance. These, however, were beyond the scope of the present study, but may have important implications on initial study participant interest and enrollment.

We lacked data on the actual number of times our ad was viewed and dismissed, thus are not able to calculate a true response rate. The app developer estimated that each of our ad campaigns were delivered to approximately 437,000 users in the U.S., thus approximating click-through rates between 0.38% and 2.00% for each ad campaign. Click-through rates for some advertisements highlighting health interventions illustrate the varying ranges of CTRs between 0.08% to 2.3% (Arcia, 2014; Birnbaum et al., 2017; Sanders et al., 2018). Nevertheless, this also means greater than 98% of individuals seeing our ad did not click it; however, these include users for whom the ad was not relevant (e.g., HIV-positive individuals, individuals who recently tested for HIV). Comparatively, our overall CR for enrolled participants was 19.6% which is higher than a conversion rate for enrollment in a smoking cessation intervention (9.1%) and for an observational study of birth preferences among pregnant women (18%) (Arcia, 2014; Graham, Milner, Saul, & Pfaff, 2008). Future researchers wishing to calculate an actual (as opposed to estimate) CTR should seek out venues where the ad purchase is based on impressions delivered. This was not the case with the app we reported on, and that limitation should be noted. Finally, because our ads ran sequentially, we cannot determine if there were seasonal effects or secular trends that influenced response to our advertisements (Polus, 2016).

Conclusion

Although there is some data on using static computer-based websites to engage MSM in health research, relatively little is known about doing so in app environments. Given the transition MSM have made from static computer-based websites for the purposes of sexual networking, to smartphone-based apps, researchers and service providers seeking to engage this population may also find themselves using such apps for outreach and recruitment. We successfully used app platforms to advertise and engage a high priority population, and found that ads featuring a male model outperformed text-only ads across a range of metrics—responsiveness among younger MSM and MSM of color, and at lower cost per enrolled participant. Advertising materials that combine text with images may have greater appeal among priority populations highlighting the importance of social identity theory and cultural diversity in health research advertising (Bennett, 2007; Smith, 1991). Our findings also highlight the importance of real-time evaluations of the relative effectiveness of outreach strategies, as such information provides valuable “lessons learned” data that other researchers and providers can use.