Using web analytics data to identify platforms and content that best engage high-priority HIV populations in online and social media marketing advertisements

Abstract

Background

Online advertisements on social media platforms are an important tool for engaging relevant populations in public health research. However, little is known about what platforms and ad characteristics are most effective in engaging high-priority HIV populations, including racial/ethnic and sexual minority individuals.

Methods

Data from this study were drawn from advertising campaigns conducted on popular websites and social media platforms that recruited for several nationwide randomized controlled trials of various HIV prevention and testing strategies among sexual minority men (SMM) from December 2019 until March 2022. Descriptive statistics and LASSO regression models were used to determine which platforms and ad characteristics were associated with significantly higher odds of engagement.

Results

Ads on Google search, Facebook, and Instagram yielded the most cost-effective engagement, while gay-oriented dating platforms and TrafficJunky yielded the highest percentage of users who appeared to meet basic eligibility criteria. The highest percentages of Black users were screened through ads on Jack’d, TrafficJunky, and Google search; for Hispanic/Latino users, Google search, Grindr, Facebook, and Instagram. Analyzing ad characteristics, we found ads that used suggestive content, animation, and included study or institution logos were associated with greater engagement. Ads that emphasized convenience of the research (e.g. mentioned participating “from home”) and that depicted people of similar races/ethnicities were also associated with greater engagement among Black and Hispanic/Latino sexual minority men.

Conclusions

We found that advertisements on mainstream social media sites are most cost effective. Although gay-oriented dating platforms were much more effective at reaching the target population, they were considerably more expensive. We also identified ad characteristics that were particularly effective in engaging users. These results could inform the design of online public health outreach campaigns for similar populations to improve their engagement and reach. Findings also demonstrated the value of conducting focused research on the effectiveness of various online marketing strategies.

Introduction

Daily internet use continues to increase in the United States and globally. ¹ In 2022, people in the United States spent an average of 7 hours online each day, ² with at least 2 of those hours spent specifically on social media platforms. ¹ Moreover, these rates are expected to continue increasing, ³ highlighting the continued importance of the internet and social media for engaging the public in health-related campaigns. Although internet/social media use varies slightly across demographic groups, vast majorities of nearly every demographic group own multiple internet-connected devices and spend significant amounts of their time online.^4,5 Average daily internet use is highest among racial/ethnic minority individuals in the United States, including Black/African-Americans (B/AA) and Hispanics/Latino/as (H/L).^6,7 Higher percentages of B/AAs and H/Ls also have accounts on nearly every major social media platform, relative to other groups.^8,9 Online public health outreach may be uniquely important for B/AA and H/L individuals in the United States, since they experience health disparities across a number of conditions, ¹⁰ including HIV. Online outreach efforts may also be especially important for HIV interventions and research, given that many populations at elevated risk (e.g. adolescents/young adults, LGBTQ  +  populations) spend significant amounts of their time online and on these platforms. ¹ As such, research focused on identifying effective strategies for reaching and engaging relevant populations in public health outreach is critical for the field. It is also consistent with our charge to serve people where they are. ¹¹

How do online outreach campaigns measure effectiveness?

The vast majority of available research on the effectiveness of online public health outreach is focused on campaigns intended to engage participants in public health research. ¹² These and other public health outreach campaigns generally have two key goals: (1) reaching a clinically relevant population and (2) eliciting engagement from users in that population (e.g. reading a message, clicking/pressing on content, and entering information into a form). To accomplish these goals, researchers have so far overwhelmingly relied on paid advertising campaigns (as opposed to organic content, such as text, images, and videos posted from an organization's own accounts on a given platform), primarily because paid advertising (ads) allow researchers to consistently reach users beyond just those who follow the researchers’ accounts. Organic content also often reaches only those with similar interests. ¹³

Most platforms that offer paid online advertising collect and report web analytics data that reflect performance of the campaign and individual ads. These data can provide unique insights into the specific characteristics of campaigns that most effectively accomplish its goals. One such useful performance metric is an ad's click-through rate (CTR), which is a ratio of the number of times users clicked on a given ad over the number of times that ad was rendered on a user's screen (impressions), and thus reflects the level of engagement an ad generates. Since many platforms charge campaigns for the number of times a given ad is shown, the cost per click (CPC) is another important metric of engagement that reflects a ratio of the total cost of showing the ad over the number of times users clicked on it. However, the ultimate engagement goal of many public health campaigns, especially campaigns recruiting for public health research, is to encourage users to express interest in participating in research and provide contact information, so that research teams can contact them to discuss specific studies. To facilitate this, users who click on an ad are often directed to survey websites that provide basic information about open research and collect interest forms. Pairing web analytics data from advertising platforms with data from other websites can provide useful insights into how effective certain ads were in generating “conversions,” or encouraging users to express interest in research. As such, the number of interest forms completed after clicking on a particular ad is likely among the most important metrics of ad performance. Finally, many campaigns also aim to reach specific populations, such as those who are especially at-risk for a given condition, and certain characteristics of online marketing strategies may be more or less effective in this task. As such, metrics like the number of qualified users who were identified through each ad and the total costs of identifying each qualified user, such as the cost per eligible (CPE), can be helpful in assessing how effective the campaign was in reaching these individuals.

What are effective elements of online public health outreach campaigns?

To date, nearly all published research has focused primarily on demonstrating the feasibility of online marketing for recruitment to public health studies on various platforms. Together, these studies have shown that it is feasible to recruit a number of at-risk populations on a variety of websites and social media platforms, including Facebook, Instagram, Snapchat, Google search, and various online dating platforms.^14–18 However, few of these studies compared the performance of various marketing strategies, making it difficult to identify specific strategies that are effective. Two key aspects of online campaigns likely have the most impact on their performance: (1) the platforms used and (2) ad/post content. Although the campaign's messengers also likely influence performance (i.e. who is posting or “speaking” in the posts/ads), the following sections discuss the available research on the first two factors, given that they were the primary focus of this project.

Platform

The specific platforms that researchers choose to advertise or post on are likely to have an important impact on how effective campaigns are in reaching and engaging their focal populations. Past studies have shown that a wide variety of platforms and websites, such as Facebook, Instagram, Reddit, Snapchat, Google search, and various online dating sites, can feasibly recruit a range of very specific populations, such as individuals who have considered abortion, individuals who cultivate cannabis, and nurses who have experienced workplace violence.^{14,16,19–24} A handful of studies have shown that online campaigns engage more relevant users and are more cost efficient than traditional outreach methods, like direct emails or printed flyers.^25–27 Others found that online samples are more diverse and representative relative to those recruited via traditional approaches,^23,28–30 perhaps because online campaigns “push” content to users who otherwise are not intentionally seeking out information on the condition or topic the research involves. ²³ Several other studies have reported similar results for HIV-relevant populations, such as sexual minority men (SMM), showing more efficient reach and engagement in online outreach campaigns than traditional methods,^15,21,31,32 although at least one also showed that local venue advertisements were particularly effective. ²¹ Still, few studies have compared the relative effectiveness of a wide variety of online platforms for recruiting HIV-relevant populations, like SMM.

Content

An ad or post's content, such as its specific text, still images, graphics, animations, or videos, is also likely to influence the effectiveness of online public health campaigns. Like platforms, however, the type of content that is most effective likely depends heavily on the population researchers hope to engage. Very few studies have systematically compared whether the specific attributes of an ad or post's content influences its engagement performance. One study recruiting adult men in Australia for mental health research showed that ads depicting masculine strengths (e.g. a man licking an axe) tended to elicit more engagement and were most cost effective, but generated fewer contact form entries. ³³ In a study of US adolescents to increase human papillomavirus vaccination (HPV), ads that used text and images emphasizing the risks of HPV elicited more engagement than those highlighting protection, ownership, or other benefits. ³⁴ Two similar campaigns focused on SMM similarly showed that ads on social media and gay-oriented dating apps that mentioned sexually transmitted infections (STIs) or sexual innuendo had higher CTRs and enrollments than those that emphasized cancer prevention or technology,^18,35 with Reiter et al. ³⁵ adding that ads with images of SMM couples had higher CTRs and conversions (i.e. users who reached the study consent form) than images of a single man, a group of men, or a doctor. Finally, one of the few relevant studies focused on HIV showed that ads for HIV self-testing on gay-oriented dating sites that featured suggestive images (e.g. bare upper bodies) generated the most clicks and had the lowest cost per conversion (screen), versus images depicting men that included faces or text-only and icon-only ads. ³⁶ However, ad images depicting men that included faces had lower CPEs and recruited a higher percentage of racial/ethnic minority SMM than the other categories. Together, these studies show that the effectiveness of online marketing campaigns depends in part on the content depicted. However, none of these studies explored more than a few characteristics of ad content and most compared their performance pairwise and across single dimensions of content (e.g. only different variations of text). More robust studies and approaches to analysis are needed to help design more engaging online public health outreach campaigns.

The current study

In this project, we used analytics data drawn from several advertising platforms generated during campaigns to recruit participants for large national studies of SMM to explore: (1) whether specific websites and social media platforms were more effective in engaging users and reaching SMM and (2) whether specific characteristics of ads were more engaging, such as the specific content of their text, still images, animations, graphics, and videos. Given the increased risk for HIV among B/AA and H/L SMM, we also explored which platforms and ad content was most effective in engaging B/AA and H/L SMM in particular. Given findings from past studies, we expected that social media platforms like Facebook and Instagram, as well as gay-oriented dating apps, would reach the most eligible users. We also expected that suggestive and romantic ad images, and text mentioning HIV, would generate the most engagement overall, and ads depicting B/AA and H/L men would generate the most engagement among B/AA and H/L SMM.

Methods

Data and participants

Data in these analyses were cross-sectional and were drawn from an advertising campaign that was intended to reach prospective participants for several open trials of HIV prevention strategies. ³⁷ Studies were “siteless” trials conducted entirely online and recruited participants from several areas with a high incidence of HIV in the United States using online outreach. In these analyses, we evaluated whether various platforms and ad content characteristics were effective in reaching SMM who lived in the targeted areas. The basic criteria shared across all parent studies included participants who were: (1) 18 years or older, (2) assigned male sex at birth, (3) male gender, (4) reported receptive or insertive anal sex with a man in the past year, (5) had a stable residence in one of the study's primary recruitment areas, (6) used an iOS/Android smartphone with a data plan or home Wi-Fi, (7) able to speak/read either English or Spanish, and (8) who met one of the three criteria suggesting some risk for HIV in the past 6 months: (a) anal sex without condoms outside of a monogamous partnership with a recently tested HIV-negative or unknown status male, (b) an STI diagnosis, or (c) an ongoing sexual partnership with an HIV-positive male.

Procedures

We continuously ran paid advertising campaigns from branded accounts on several popular websites and social media platforms from December 2019 until March 2022. During this time, we ran campaigns on the social media sites Facebook and Instagram, gay-oriented dating sites Grindr, Scruff, and Jack’d, and other platforms like Google search and TrafficJunky, an online advertising network that includes a variety of popular websites, apps, and games. Within each platform, we adjusted audience characteristics whenever possible to present ads primarily to users within a 25 mile radius of several cities/regions that have high HIV incidence: Los Angeles, CA; New Orleans, LA; Baton Rouge, LA; Shreveport, LA; Jacksonville, FL; Orlando, FL; Miami/Ft. Lauderdale, FL; Tallahassee, FL; Boston, MA/Providence, RI; and the state of Mississippi. These cities/regions were selected when designing the parent studies in order to balance representation in many regions of the United States and focus on areas where HIV incidence is particularly high. ³⁸

Over the course of the campaign, we created and uploaded new ad content at least 2 to 3 times per month, with the goal of both keeping content fresh for users and enabling us to explore attributes of ad content that could be particularly engaging. Throughout the campaign, we ran 122 different ads that used different combinations of text, still images, graphics, animations, and videos (see Figure 1 for example graphic ads). However, some platforms (Grindr, Jack’d, Scruff) only allow campaigns to run one or two ads at a time. Others, like Facebook and Instagram (both run through Facebook's advertising platform), allow up to 500,000 ads within each campaign. Ads can be designed for placement on users’ timelines or newsfeeds, which is the centerpiece of each site and is a continuously updating list of all content (text, images, and video) posted by the user's friends or followed accounts (hereafter called “feed ads”), or among users’ “stories,” which are short sequences of photos or video clips created by users (hereafter called “story ads”). On Facebook and Instagram, we enabled campaign optimization features that empower the platform's software to promote ads that had previously performed well (e.g. elicited more clicks) more often than those that had not.

Figure 1.

Example ads run on various platforms and websites during the campaigns.

Note. Many ads contained stock photos of real people, licensed under public domain. Only graphic ads were included as examples to protect the confidentiality of these individuals.

Users of these platforms who saw a campaign ad were prompted to click on it to learn more about open studies. Doing so referred users to a landing webpage in on a survey website that provided brief, bulleted information about open studies and invited users to complete a brief form that assessed basic eligibility criteria for open studies. Entry into the interest form was protected by reCaptcha v2 widget. Users were also prevented from submitting multiple entries from the same browser and were deemed ineligible if their reCaptcha v3 scores were < 0.5. reCaptcha is a bot mitigation tool: v2 uses screening puzzles to differentiate between bots and real users, while v3 assigns scores without any user interaction. Data collected in this initial interest form was completely anonymous. Those who were eligible based on this survey were then referred to a separate webpage for more detailed information about specific studies they may be eligible for and, if interested, subsequent forms collected informed consent and contact information so that research staff could follow up with them about participating in those studies. All procedures were carried out in accordance with relevant guidelines and regulations. The Brown University Institutional Review Board reviewed these procedures and determined that they were exempt (Study #00000133).

Measures

Web analytics. Data on basic ad engagement, including clicks, forms started, and forms completed were collected using basic web analytics methods in which a unique alpha-numeric session ID number was appended onto the URLs each ad re-directed users to. The survey website then captured this ID, enabling us to determine whether specific users simply clicked on an ad and ended their session, started the interest form but did not finish, completed the full interest form, and/or met basic eligibility criteria. A unique code assigned to each ad also identified the specific ad and platform that each user was referred through.

Engagement metrics. We used three primary metrics to assess overall engagement with all ads delivered on a given website or platform: (1) CTR, which is a ratio of the number of times users clicked on a given ad and the number of times it was rendered (impressions); (2) CPC, or the amount of money (in US dollars) paid for each click; (3) a binary variable reflecting whether or not users completed the initial interest form was also used to evaluate the effectiveness of specific ad content in engaging users.

Reach metrics. We used two key metrics to assess the effectiveness of each platform in reaching SMM who may be at elevated risk for HIV: (1) the overall percentage of those who completed the initial interest form who met basic eligibility criteria for open studies (noted in Data and participants section) and (2) the CPE.

Statistical analysis

To calculate overall metrics of engagement and reach by platform, we first extracted the number of impressions, clicks, and amount spent for ads run on that platform and used them to calculate the CTR, cost per screen (CPS), CPE, and percent eligible. We then compared these metrics descriptively across platforms. Within ads run on Facebook and Instagram, we also compared metrics across feed and story ads. To identify attributes of ad content that were associated with more engagement and better reach, we coded whether each ad depicted or contained any of 40 possible attributes of the text, images, graphics, animations, or videos included in each ad (a full list of these attributes is included in Supplemental Appendix A). To do so, co-authors (TBW, DJC, and EMSO) first met as a group and identified as many distinguishable attributes as possible. Then, we reviewed the list and eliminated any duplicate or nondistinct attributes, discussing areas of disagreement as needed. Next, two co-authors (DJC and EMSO) rated each of the ads independently according to whether it contained each attribute, giving each ad a binary value (1/0) for each attribute. We then compared both raters’ responses, and while disagreement was extremely rare, discussions with a third co-author (TBW) helped to resolve them. The result was a list of all ads run during the campaign and set of codes reflecting whether or not that ad contained or had specific content or attributes. Using this approach to code ad content allowed us to explore the performance of a large number of broader attributes of ad content that were often shared across several ads, rather than the performance of specific ads themselves or a handful of broad types of ads. We then examined frequencies in which each coded attribute appeared in each category of the outcomes (screened and eligible) and excluded attributes that appeared in <5% or >95% of ads users responded to, because attributes appearing extremely rarely or nearly all the time would not be useful in analyses. This excluded four attributes from further analysis. We also explored pairwise correlations between all of the attributes. We retained only one attribute within any cluster of attributes with r > .80, since these attributes likely only appeared together in the same ads, and so, would not aid in prediction. This resulted in excluding an additional 9 attributes, resulting in a final set of 27 attributes that were included in the analysis. Next, we estimated least absolute shrinkage and selection operator (LASSO) regression models with ad-level CTR and CPC as primary outcomes and all ad attributes as predictors. We chose LASSO models because of their strengths in variable selection, particularly when attributes are strongly correlated. ³⁹ Since CTR and CPC were both highly skewed, we log-transformed these variables prior to analysis. Afterward, skewness for both variables were within an acceptable range (CTR  =  1.2 and CPC  =  0.8). ⁴⁰ We then compiled the results of each model and summarized them using forest plots, which depict the multiplicative effects of the top ad attributes on the outcomes and the corresponding 95% confidence intervals (CIs) for a sense of plausible range of magnitude of each attribute's effects. To test the influence of ad content attributes on the user-level odds of interest form completion, we used a similar approach as above but with a binary outcome variable reflecting whether a given user completed the form and the attributes of the ad that the user clicked through as predictors. We also tested whether specific ad attributes were associated with greater engagement among B/AA and H/L SMM by coding a variable reflecting whether participants reported one of these racial/ethnic categories versus those who did not and estimating a similar model. The impact of these attributes on these outcomes was quantified using odds ratios (ORs). All analyses were conducted in R (version 4.1.2). Significance was defined as p < 0.05.

Results

Figure 2 shows the flow of users across all campaigns and platforms during the recruitment period. During the study period, there were over 7.8 million impressions with 123,328 unique clicks by users; 19,196 started the online questionnaire with 7881 completing it. A total of 1365 (13.1%) met the basic eligibility criteria. Table 1 summarizes the demographic characteristics of users who completed the initial interest form. Unsurprisingly, given the content of all ads, participants who completed the interest form were overwhelmingly assigned male sex at birth, current male gender, and reported gay sexual orientation. Figure 3 shows frequencies of all coded ad attributes across all ads. Most ads used images depicting at least one man (82.5%), at least one racial/ethnic minority man (66.7%), used bright colors (61.7%), and depicted all racial/ethnic minority men (56.7%).

Figure 2.

Flow of users through the ad campaign and interest form sequence.

Table 1.

Demographic characteristics of users completing the initial interest form (N  =  7881).

Characteristic	Mean (SD) or N (%)
Age (range: 18–90)	35.0 (12.6)
Gender
Male	7619 (96.7)
Female	8 (0.1)
Trans	130 (1.7)
Other	121 (1.5)
Assigned male at birth	7677 (97.4)
Race
White	5332 (67.7)
Black or African-American	566 (7.2)
Asian	516 (6.6)
American Indian/Alaska Native	128 (1.6)
Pac. Isl./Native Hawaii	25 (0.3)
Multiracial	429 (5.4)
Chose not to respond	885 (11.2)
Ethnicity (Hispanic or Latino)	2761 (35.0)
Sexual identity
Gay	6187 (78.5)
Bisexual	1104 (14.0)
Other	117 (1.5)
Not sure	82 (1.0)
Chose not to response	321 (4.1)
Spanish language	846 (10.7)
Region of residence
Northeast	1600 (20.3)
South	3294 (41.8)
West	2229 (28.3)
None of these	758 (9.6)

Figure 3.

Percent of all ads that contained specific attributes.

Note. Green reference lines show attributes that were represented in <5% of all ads or >95% of all ads. Attributes that were below or above this cutoff were not included in analyses.

Table 2 summarizes the key metrics of campaign performance by platform, as well as the percentage of users who completed the interest form through each platform who identified as B/AA or H/L. Ads on Google search, Facebook, and Instagram yielded the highest CTRs, and the lowest CPSs and CPEs. However, gay-oriented dating platforms and TrafficJunky yielded the highest percentage of eligible users. The highest percentages of B/AA users were screened through ads on Jack’d, TrafficJunky, and Google search, and while these platforms also had some of the highest percentages of eligible respondents, they were also among the most expensive campaigns and generated the least number of total conversions. The highest percentages of H/L users were screened through ads on Google search, Grindr, Facebook, and Instagram. Story ads outperformed feed ads generally across most metrics, with story ads yielding a higher CTR at a slightly higher CPC than feed ads, but a significantly lower CPS and CPE compared to feed ads. The percentage of all participants who completed the interest form that were ultimately eligible was slightly lower among story ads versus feed ads, but this is partly explained by the higher percentage of users who clicked through and screened through story ads, which was significantly higher than feed ads.

Table 2.

Selected metrics of campaign performance by platform.

Platform	CTR	CPC ($)	CPS ($)	CPE ($)	% Eligible	% B/AA	%H/L
Facebook	1.8	0.63	9.60	82.52	16.2	10.3	33.3
Instagram	1.8	0.63	8.37	83.35	15.9	10.2	38.0
Feed ads a	1.6	0.56	11.97	63.29	18.9	10.4	34.0
Story ads a	2.1	0.78	2.80	17.92	15.7	10.2	36.3
Grindr	0.4	1.33	44.32	250.00	17.7	12.9	52.7
Scruff b	1.1	2.91	39.00	229.41	17.0	9.7	20.8
Jack’d b	1.1	2.91	41.93	150.00	28.0	29.6	17.4
Google search	1.4	5.99	11.21	75.29	14.9	17.5	52.9
TrafficJunky	0.1	1.66	105.68	581.25	18.8	16.7	27.3

^a Feed ads are those that appeared in users’ timelines or “newsfeeds” on Facebook and Instagram, while story ads were those that appeared among other short, time-limited photo sequences or clips in which users convey their status or other information.

^b The same ad campaign ran across both of these platforms, with a single combined metric provided by both.

B/AA: Black/African-American; CPC: cost per click; CPE: cost per eligible; CPS: cost per screen; CTR: click-through rate; H/L:  Hispanic/Latino/a. Represents the percentage of users who completed the initial interest form and also reported this race/ethnicity.

LASSO regression identified six attributes with the strongest effects in predicting the log-transformed CTR (see Figure 4). A forest plot (see Figure 4) showed that ads depicting suggestive images of men and ads that included animation were significantly positively associated with higher CTRs. One other attribute, whether the ad depicted an image of a man holding a phone, was significantly associated with lower CTRs. A similar model showed that approximately five attributes appeared to be most important in predicting CPC (see Figure 5). Many of the same attributes that were associated with CTR were also associated with CPC. Ads including suggestive images were positively associated with CPC, meaning that ads with this attribute cost less to elicit each click. Ads depicting men holding phones and men wearing glasses were negatively associated with CPC, meaning ads with these attributes cost more to generate each click.

Figure 4.

Forest plot of coefficients for the top predictors of CTR.

Solid black squares reflect the odds ratio estimate. Black lines represent the 95% confidence interval for each estimate. CTR: click-through rate.

Figure 5.

Forest plot of coefficients for the top predictors of CPC.

Solid black squares reflect the odds ratio estimate. Black lines represent the 95% confidence interval for each estimate. CPC: cost per click.

In a LASSO model of the user-level odds of conversion (see Figure 6), seven attributes were among the most important in predicting whether users completed the initial interest form, versus those who clicked on an ad but elected not to complete it. The odds of users electing to complete the form were higher among those who clicked through a brightly colored ad (adjusted OR (aOR) = 1.41, 95% CI  =  1.29, 1.54), among those who clicked through an ad with a logomark on it (i.e. a study or university logo; aOR = 1.28, 95% CI  =  1.02, 1.61), and among those who clicked through an ad that included the text “paid” (aOR  =  1.16, 95% CI 0.99, 1.35), although the effects of this latter attribute overlapped slightly with an OR of 1, limiting strong conclusions about the importance of this particular factor. Two additional attributes were associated with lower odds of completing the interest form. Users had a lower odds of completing the interest form who clicked through ads depicting a man looking at their phone (aOR = 0.78, 95% CI  =  0.69, 0.89) and among those who clicked through ads that included images depicting B/AA men (aOR = 0.49, 95% CI  =  0.40, 0.58).

Figure 6.

Forest plot of odds ratios for the top predictors of conversion.

Solid black squares reflect the odds ratio estimate. Black lines represent the 95% confidence interval for each estimate.

Finally, LASSO models of the user-level odds of conversion among B/AA and H/L SMM versus other racial/ethnic groups selected nine attributes as among the most important (Figure 7). A forest plot showed that four attributes were significantly associated with an increased odds of completing the initial interest form. The odds of a B/AA or H/L user completing the form were significantly higher among those who clicked through ads depicting either B/AA (aOR = 2.97, 95% CI  =  2.29, 3.88) or H/L men (aOR = 1.91, 95% CI  =  2.13, 4.06), versus those that did not. The odds of these users electing to screen were also higher among those who clicked through ads that included text suggesting that the program could be completed “at home” (aOR = 2.93, 95% CI 2.13, 4.06), and among those who clicked through ads that suggested the program was primarily “online” (aOR = 1.39, 95% CI  =  1.15, 1.69). Surprisingly, two attributes that were associated with higher numbers of clicks, ads containing particularly bright colors (aOR = 0.54, 95% CI  =  0.45, 0.64) and ads that included suggestive images (aOR = 0.76, 95% CI  =  0.61, 0.96), were associated with a lower odds of completing the screening among B/AA and H/L, specifically.

Figure 7.

Forest plot of odds ratios for the top predictors of conversion among users who identified as Black/African-American or Hispanic/Latino.

Solid black squares reflect the odds ratio estimate. Black lines represent the 95% confidence interval for each estimate.

Discussion

This project's findings show that several decisions about what platforms to focus on and the content of posts have an important impact on the performance of online public health marketing campaigns. They also demonstrated the value of evaluating the effectiveness of various online marketing strategies. Given the promise of online marketing campaigns for reaching and engaging relevant and representative populations,^23,28–30 the results of studies like these could inform the design of more inclusive and effective public health outreach campaigns.

Platforms

Our findings showed that campaigns on Facebook, Instagram, and Google search generally engaged more users, and did so cost effectively, which is consistent with a large body of research showing that these platforms can feasibly recruit a variety of populations that are relevant to public health.^{14,16,19–24} However, our findings are unique in showing that these platforms generally elicited more engagement among SMM relative to gay-oriented dating platforms, which many expect will be especially successful among this population. Consistent with past research,^21,36 our results showed that these dating platforms reached a higher percentage of SMM who may be at-risk for HIV than social media or other platforms, but were generally much more expensive. Particular dating apps, including Jack’d and Grindr, may also be uniquely effective in reaching particularly high-priority subpopulations for HIV, including B/AA SMM. However, social media platforms were very effective in reaching high percentages of H/L SMM. Together, these findings underscore that a key challenge for those designing online marketing campaigns for HIV is balancing the high cost of advertising on dating apps with its strengths in reaching certain high-priority subpopulations, like B/AA SMM. They also highlight the importance of using platform-level metrics of engagement and reach to design a campaign strategy that balances the potential reach of advertising on various platforms with their cost.

Among Facebook and Instagram ads, our findings also suggest that story ads may yield significantly more conversions at a lower cost than feed ads. Although story ads had a slightly higher CPC and lower percentage of eligible users than feed ads, the higher CTR of story ads, as well as a much higher percentage of users who clicked ultimately elected to complete the initial interest form significantly lowered the CPS and CPE for story ads versus feed ads. This pattern of findings suggests that the higher clicks and conversion rates among story ads likely compensate for smaller differences in eligibility percentages and yields better overall performance than feed ads.

Ad content

Table 3 lists key findings about specific ad attributes that were associated with higher engagement. Models testing whether specific characteristics of ad content were associated with generally higher and more cost-efficient engagement consistently echoed findings from past studies showing that suggestive images consistently elicited more engagement among SMM at lower cost than other ads. ³⁶ These results suggest that, while sometimes controversial, using suggestive images in public health outreach campaigns that are relevant to sex is likely a generally effective strategy for engaging certain populations, like SMM. However, our findings also extend past research by showing that several other attributes of ads may be particularly effective in eliciting engagement, including brightly colored ads, ads that incorporate animation and other techniques that help ads stand out from other content users normally see on their feeds (e.g. text, photos, and videos from friends). Although these attributes were among the only factors that were statistically significant in their association with ad-level engagement metrics (CTR and CPC), other nonstatistically significant attributes may also be instructive to campaign designers. Those attributes with sizable coefficients (such as ads depicting men interacting romantically) could still suggest that they had some influence on engagement, but because the confidence intervals of these attributes overlapped with zero, we cannot be firmly confident about their contribution to higher or lower levels of engagement.

Table 3.

List of ad attributes that were associated with higher engagement outcomes and their direction, by sample subset.

Attribute
Ad-level analyses
Man without shirt
Included animation
All sexual minority men (SMM)
Used bright colors
Used a study/institution logomark
Included the word “paid”*
Black/African-American or Hispanic/Latino SMM
Included the word “online”
Included the words “at home”
Depicted a Hispanic/Latino man
Depicted a Black/African-American man

Note: *Attribute was p ≤ 0.07.

Our results also extend past research by showing that several other previously unexplored aspects of ad content were also represented more frequently among those who elected to complete the initial interest form, versus those who did not. These analyses were important both because of the breadth of attributes they explored and because of their focus on the engagement behavior that is the ultimate goal of many public health outreach campaigns: Encouraging users to complete a health screening and/or opt-into a public health program. Our findings showed that users chose to express interest in research significantly more often when they clicked through ads that were brightly colored. Notably, “brightly colored” was a distinct attribute reflecting whether coders agreed that the ad contained vivid colors (versus more muted colors). We also coded a separate variable for whether the ad contained any color (vs. black and white), but this variable was not included in analyses because all ads we ultimately ran were in color. Our results also showed that users elected to screen more often when they clicked through ads containing a study logomark, perhaps because including logomarks convey that the study is legitimate or is supported by a trusted institution. While not statistically significant, unsurprisingly, users also appeared to be more likely to complete the initial interest form after clicking through ads that included the word “paid.” Designing ads that incorporate these factors could increase the odds that users go beyond simply clicking and actually express interest in the program being advertised.

Finally, our results also showed that several attributes of ad content also distinguished between B/AA and H/L users versus others. The odds of B/AA and H/L users electing to complete the interest form were significantly higher among those who clicked through ads that explicitly mentioned that users could participate in the research from home. These results are consistent with past research showing that people are generally more likely to use a health intervention and participate in research if it is convenient for them, ⁴¹ but extends these findings to suggest that convenience may be of even greater importance for engaging high-priority populations who may be more skeptical of research or inexperienced with engaging in research, like B/AA and H/L SMM. B/AA and H/L SMM also had higher odds of clicking through ads that featured either B/AA or H/L men, confirming findings from past research that racial/ethnic minority individuals often find ads featuring individuals that are similar to them more appealing and engaging than those that do not.^42–44 These results add further confidence to the notion that campaign designers who aim to engage specific, high-priority populations should create content that depicts individuals with similar appearances, and that emphasizing convenience may uniquely appeal to B/AA and H/L SMM. Although these findings may appear to conflict with the model of conversions among all users, it is important to note that the overall model showed that, among all participants, the odds of completing the initial interest form were lower after users clicked through ads depicting only B/AA men, whereas this model showed that, among B/AA or H/L participants, the odds of conversion were higher after users clicked through ads depicting at least one B/AA man or at least one H/L man. The former finding could suggest that, in a large sample of SMM (many of whom were not B/AA), some may have simply believed that the study was only recruiting B/AA men, or it may reflect some more pernicious process (e.g. racial bias) or other factors. On the other hand, the finding that B/AA and H/L were more likely to complete the initial interest form after clicking through ads that depicted at least one B/AA or H/L man suggests that racial/ethnic concordance is important in digital advertising. Finally, this model also showed that the odds of B/AA and H/L electing to complete the initial interest form were significantly lower among those who clicked through ads that included especially bright colors, particularly dark colors, and those that included suggestive images. Although these findings appear to conflict with those examining overall ad engagement among SMM in general, this pattern of results could suggest that these characteristics appeal less to B/AA and H/L who are willing to engage in public health research and programs. That is, B/AA and H/L who are open to engaging in outreach campaigns may view ads that contain these characteristics to be too garish or bold, and thus, may appeal less to them. These interpretations are speculative, however, and should be confirmed in future work.

Limitations

Although this work had a number of significant strengths, several limitations are important to note. Although our analyses allowed us to explore whether certain attributes of ad content might have been appealing to racial/ethnic minority users who completed the initial interest form, determining ad content that might encourage these users to initially click on an ad or post could also be uniquely helpful for designing campaigns that better engage them, especially those that may not typically engage in public health research or programs. However, we were not able to examine these factors because demographic information is unavailable at that stage. This limitation underscores the need for future research that goes beyond web analytics data and instead focuses on understanding what users in these groups engage with while using these platforms naturally. This would allow researchers to develop ad content that more closely resembles content that may be more likely to engage those who rarely get involved in public health programming. Second, our ad content was limited by the stock images available for commercial use. There are far fewer options for stock images featuring B/AA and H/L SMM, compared to the images available that include white SMM. This makes it difficult to distill whether engagement with images showing only B/AA SMM is lower due to the content of the images, or because of the quality of the images, since there were not as many options for this type of image. Third, the features of ad content that we coded represented over 40 attributes of content, which is more than many similar past studies, these attributes are unlikely to capture all of the characteristics that are important to engagement among SMM or that could be especially potent in engaging them (e.g., the age of the men depicted in the ads).

Conclusions

In this project, we found that Facebook and Instagram ads generated the most engagement and were most cost effective overall. However, gay-oriented dating platforms reached a higher percentage of SMM who may be at increased risk for HIV and a higher percentage of B/AA SMM, but were considerably more expensive. We also found that ads that depicted suggestive content, used animation, and that included study or institution logos were associated with greater engagement. Ads that emphasized convenience of the research and that depicted people of similar races/ethnicities were also associated with greater engagement among B/AA and H/L SMM. These results can inform the design of online public health outreach campaigns for similar populations to improve their engagement and reach.