Abstract
Background.
Routine human papillomavirus (HPV) vaccination is recommended for young adults, yet many young gay, bisexual, and other men who have sex with men (YGBMSM) remain unvaccinated. We report the efficacy of Outsmart HPV, a web-based HPV vaccination intervention for YGBMSM.
Methods.
From 2019–2021, we recruited YGBMSM in the United States who were ages 18–25 and unvaccinated against HPV (n=1,227). Participants were randomized to receive either: (a) Outsmart HPV content online and monthly interactive text reminders (interactive group); (b) Outsmart HPV content online and monthly unidirectional text reminders (unidirectional group); or (c) standard information online about HPV vaccine (control group). Regression models compared study groups on HPV vaccination outcomes.
Results.
Overall, 33% of participants reported initiating the HPV vaccine series and 7% reported series completion. Initiation was more common among participants in the interactive group compared to the control group (odds ratio [OR]=1.47, 98.3% confidence interval [CI]: 1.03–2.11). Completion was more common among participants in both the interactive group (OR=3.70, 98.3% CI: 1.75–7.83) and unidirectional group (OR=2.26, 98.3% CI: 1.02–5.00) compared to the control group. Participants who received Outsmart HPV content reported higher levels of satisfaction with online content compared to the control group.
Conclusions.
Outsmart HPV is an efficacious and acceptable HPV vaccination intervention for YGBMSM. Future efforts are needed to determine how to optimize the intervention and disseminate it to settings that provide services to YGBMSM.
Impact.
Outsmart HPV is a promising tool for increasing HPV vaccination among YGBMSM with the potential for wide dissemination.
Keywords: human papillomavirus (HPV), HPV vaccination, gay or bisexual, MSM (men who have sex with men), intervention, young adult
Introduction
Oncogenic human papillomavirus (HPV) types can cause several types of cancer in men (i.e., anal, oropharyngeal, and penile cancer), and nononcogenic HPV types cause almost all cases of genital warts (1, 2). Rates of HPV infection and HPV-related disease tend to be higher among gay, bisexual, and other men who have sex with men (GBMSM) (3–5). For example, the anal cancer incidence rate among human immunodeficiency virus (HIV)-negative GBMSM is about 10 times higher than the United States (US) national rate, and the incidence is even higher for HIV-positive GBMSM (5). HPV vaccination has been approved by the US Food and Drug Administration to prevent anal cancer, oropharyngeal cancer, and genital warts among males (6). Indeed, HPV vaccine protects against infection with the types of HPV that cause almost all genital warts and cases of anal cancer (1, 2). Routine HPV vaccination is currently recommended for males and females who are ages 11–12, with vaccination also recommended for people through age 26 who have not already been vaccinated (7). Shared decision-making between people ages 27–45 and their healthcare providers is suggested (7) in order to discuss the potential health benefits of HPV vaccination for these individuals.
Despite recommendations, many young adults in the US, including young gay, bisexual, and other men who have sex with men (YGBMSM), remain unvaccinated against HPV. In fact, only about 40% of age-eligible YGBMSM have initiated the HPV vaccine series (8). We developed a web-based HPV vaccination intervention, called Outsmart HPV, to increase HPV vaccination among YGBMSM (9, 10). Our previous work shows that the intervention improves cognitive outcomes (i.e., knowledge, attitudes, and beliefs) related to HPV vaccination and may also improve vaccine uptake (9–11). The current report takes extends this line of research by including findings from a large randomized controlled trial (RCT) on the efficacy of Outsmart HPV on increasing HPV vaccine uptake and participants’ satisfaction with intervention content.
Materials and Methods
Participants and Procedures
The methods for the RCT have been described previously (12) and will be covered briefly here. We recruited a convenience sample of YGBMSM via advertisements on social media/dating sites, existing research panels, and university-based organizations. Interested individuals were directed to a mobile-friendly project website where they had an opportunity to complete an eligibility screener. Eligibility criteria included self-identifying as: (a) cisgender male; (b) 18–25 years of age; (c) either gay, bisexual, or queer; ever having oral or anal sex with a male; or being sexually attracted to males; (d) living in the US; (e) not vaccinated against HPV (i.e., indicating no HPV vaccine doses received); and (f) not having previously participated in Outsmart HPV. Age 25 was the upper age limit for eligibility so individuals did not “age out” of the recommended age range for routine HPV vaccination (7) during study participation. Eligible individuals’ participation was then contingent upon the completion of informed consent online and the creation of aa project website account.
Participants next completed a baseline survey (“T1 survey”) on the project website and were then immediately randomized via an automated process on the project website with a 1:1:1 allocation scheme to either: (a) Outsmart HPV content that included monthly interactive vaccination reminders (interactive group); (b) Outsmart HPV content that included monthly unidirectional vaccination reminders (unidirectional group); or (c) standard information about HPV vaccine (control group). We examined both interactive and unidirectional reminders since past research has suggested that people may prefer interactive reminders and that they may be more effective in increasing vaccine uptake (13, 14). After randomization, participants viewed their assigned content about HPV vaccine (either Outsmart HPV content or standard information) on the project website and then completed a second survey (“T2 survey”). Completion of the T2 survey concluded a participant’s initial session on the project website. Additional follow-up surveys on the project website then occurred three and nine months later (“T3 survey” and “T4 survey,” respectively). In exchange for completing study surveys, participants could earn up to $95 in gift cards.
From October 2019 through June 2021, a total of 1,227 participants were randomized for this study (n=408 in the interactive group, n=407 in the unidirectional group, and n=412 in the control group). Over 80% of these participants stemmed from recruitment via social media/dating sites. Retention was 90% for the T2 survey (interactive group=89%, unidirectional group=88%, and control group=92%), 70% for the T3 survey (interactive group=71%, unidirectional group=67%, and control group=72%), and 69% for the T4 survey (interactive group=68%, unidirectional group=67%, and control group=72%). Study activities were paused for several months during 2020 due to the coronavirus disease 2019 (COVID-19) pandemic. This study was conducted in accordance with ethical guidelines (e.g., Belmont Report, US Common Rule) and was approved by the Institutional Review Board at The Ohio State University. The RCT is registered at ClinicalTrials.gov: NCT04032106 (available at: https://clinicaltrials.gov/ct2/show/NCT04032106).
Study Materials
Intervention Content.
The interactive and unidirectional groups received the same population-targeted, individually tailored Outsmart HPV content delivered through the project website and differed only in the type of HPV vaccination reminders received, as described further below. Outsmart HPV content was informed byfeatures of the Protection Motivation Theory (15), Information-Motivation-Behavioral Skills Model (16), and the Minority Stress Model (17). Intervention content was patterned heavily after the materials used in a previous pilot test of Outsmart HPV, which were guided by input from YGBMSM through focus groups and in-depth interviews (9, 10). Prior to the start of the current RCT, we conducted additional focus groups of YGBMSM in order to update and refine the materials from our pilot study via an iterative process (18). The additional focus groups also helped further ensure the appropriateness of the intervention content.
The project website presented Outsmart HPV content through a variety of formats (e.g., infographics, testimonials), and participants could view content throughout the study period, except during surveys. The online content included four sequential sections:
“Learn About HPV” provided population-targeted information about HPV and HPV-related diseases among GBMSM. For example, this section included content about the occurrence of HPV infection and anal cancer among GBMSM. At the end of this section, the project website prompted participants to identify the most important thing they learned about HPV.
“Learn About the Vaccine” provided information about HPV vaccine (e.g., effectiveness, safety, and recommendations). This section also included population-targeted content for YGBMSM (e.g., HPV vaccine acceptability previously reported by YGBMSM (19)). At the end of this section, the project website prompted participants to identify their motivations for wanting to become vaccinated.
“Get Answers” addressed common barriers and concerns about HPV vaccination among YGBMSM (19–24) using a question and answer format. The website individually-tailored the content within this section based on a participant’s responses on the T1 survey about barriers and concerns. Thus, the barriers and concerns indicated by a participant on the T1 survey were prioritized and appeared at the top of this section in order to highlight content that was likely most relevant to that participant.
“Get Vaccinated” provided content about vaccination logistics, including vaccine accessibility, cost and health insurance, and talking with a healthcare provider about HPV vaccine. Participants were also able to identify potential questions they would have when discussing HPV vaccination with a doctor. The website then prompted participants to create an individually-tailored “Action Plan” that included setting a goal date for receiving their first dose of HPV vaccine, schedule for subsequent doses, and suggested next steps for getting vaccinated.
In addition to the online Outsmart HPV content, both intervention groups received theoretically-informed HPV vaccination reminders that provided a cue to action. The first vaccination reminder was sent about a month after a participant joined the study and continued on a monthly basis until study participation ended. However, the reminders were different for the interactive and unidirectional groups. The reminders sent to participants in the unidirectional group were automated text messages that were unidirectional (i.e., participants did not have the option to respond to these reminders). Unidirectional reminders contained text-based information about HPV and HPV vaccine, steps for getting vaccinated, and content addressing common barriers and concerns.
The text message reminders sent to the interactive group covered similar information as those for the unidirectional group, but these reminders provided the opportunity for participants to respond. To help increase participant engagement and reinforce behavior, some of the interactive reminder messages also contained a meme or brief animation in Graphics Interchange Format (GIF). Prior to the RCT, the study team determined which messages would include a meme or GIF and identified the exact memes and GIFs to be included. Messages for the interactive reminders were sent via a predominantly automated process. For each month, an automated initial text message was sent and included a question that prompted a response from participants. If a participant responded to this initial message, subsequent automated messages with additional questions, information, and resources were sent. The content was individually-tailored to participants based on their responses to questions. At the end of the automated sequence of messages, participants had the opportunity to send any remaining questions via text message. A study team member reviewed each received question and then used a library of potential responses to guide their manual response.
Control Group Content.
Participants in the control group received standard information about HPV vaccine on the project website. This content was modeled closely after the Vaccine Information Statement (VIS) for HPV vaccine that has been created by the Centers for Disease Control and Prevention (25). We modeled control group content after the VIS because it was designed to provide information that is easy to understand, it is publicly available, and healthcare providers are required to give a VIS to patients before vaccination. Similar to the unidirectional and interactive groups, participants in the control group could view their content throughout the study, except during surveys. Participants in the control group did not receive any HPV vaccination reminders.
Measures
HPV vaccination.
We used self-reported HPV vaccination data from the T3 and T4 surveys to examine both HPV vaccine initiation (receipt of one or more doses) and HPV vaccine completion (receipt of all three doses recommended for the age range of our participants (26)). Both outcomes were dichotomous (yes or no for each). We categorized participants who did not complete the T3 and T4 survey as “no” for each outcome.
Participant satisfaction.
The T2 survey included Likert-type items on participants’ satisfaction with their online content (either Outsmart HPV content or standard information for the control group) (10, 27). Items assessed participants’ satisfaction with the quality of information received (e.g., the information was easy to understand), the quality of the project website (e.g., it is easy to locate information), and their study experience (e.g., it improved my ability to make healthier choices). For participants in the interactive and unidirectional groups, the T4 survey assessed satisfaction with their monthly HPV vaccination reminders using Likert-type items (e.g., the reminders were helpful). All Likert-type items had a possible range of 1–5 and were coded so that higher values indicate greater levels of agreement with a statement.
Demographic and health characteristics.
The T1 survey included items on demographic and health characteristics (Table 1). This included participants’ disclosure of their sexual orientation to their healthcare provider and concealment of their sexual orientation (3 items, α=0.75, possible range=1–5) (28). Perceived discrimination in healthcare was assessed by asking participants about their experiences of receiving poor quality healthcare due to their sexual orientation (1 item, possible range=1–5) (29). The survey also examined participants’ electronic health (e-health) literacy (4 items, α=0.83, possible range=1–5) (30).
Table 1.
Participant characteristics by study group (n=1,227).
| Interactive Group (n=408) |
Unidirectional Group (n=407) |
Control Group (n=412) |
Total (n=1,227) |
|
|---|---|---|---|---|
|
|
||||
| n (%) | n (%) | n (%) | n (%) | |
|
| ||||
| Age (years) | ||||
| 18–21 | 143 (35) | 152 (37) | 142 (35) | 437 (36) |
| 22–25 | 265 (65) | 255 (63) | 270 (66) | 790 (64) |
| Race / ethnicity | ||||
| Non-Hispanic White | 193 (47) | 198 (49) | 188 (46) | 579 (47) |
| Non-Hispanic Black | 47 (12) | 45 (11) | 37 (9) | 129 (11) |
| Hispanic | 110 (27) | 112 (28) | 130 (32) | 352 (29) |
| Non-Hispanic Other | 58 (14) | 52 (13) | 57 (14) | 167 (14) |
| Relationship status | ||||
| Single and not having sex | 61 (15) | 44 (11) | 65 (16) | 170 (14) |
| Single and having sex or casually dating | 254 (62) | 269 (66) | 270 (66) | 793 (65) |
| In a relationship | 93 (23) | 94 (23) | 77 (19) | 264 (22) |
| Education level | ||||
| High school or less | 116 (28) | 143 (35) | 121 (29) | 380 (31) |
| Some college or more | 292 (72) | 264 (65) | 291 (71) | 847 (69) |
| Employment status | ||||
| Employed full time or part time | 123 (30) | 105 (26) | 121 (29) | 349 (28) |
| Other | 285 (70) | 302 (74) | 291 (71) | 878 (72) |
| Sexual identity | ||||
| Gay | 273 (67) | 267 (66) | 275 (67) | 815 (66) |
| Bisexual | 108 (27) | 104 (26) | 105 (26) | 317 (26) |
| Some other identity | 27 (7) | 36 (9) | 32 (8) | 95 (7) |
| Ever had sex with a male | ||||
| No | 26 (6) | 22 (5) | 37 (9) | 85 (7) |
| Yes | 382 (94) | 385 (95) | 375 (91) | 1,142 (93) |
| Sexually attracted to males | ||||
| No | 11 (3) | 16 (4) | 11 (3) | 38 (3) |
| Yes | 397 (97) | 391 (96) | 401 (97) | 1,189 (97) |
| Region of residence | ||||
| Northeast | 72 (18) | 79 (19) | 77 (19) | 228 (19) |
| Midwest | 74 (18) | 79 (19) | 71 (17) | 224 (18) |
| South | 146 (36) | 137 (34) | 146 (35) | 429 (35) |
| West | 116 (28) | 112 (28) | 118 (29) | 346 (28) |
| Health insurance | ||||
| Private insurance | 251 (62) | 258 (63) | 259 (63) | 768 (63) |
| Public insurance | 70 (17) | 65 (16) | 71 (17) | 206 (17) |
| None/don’t know | 87 (21) | 84 (21) | 82 (20) | 253 (21) |
| Last preventive health visit | ||||
| Within last year | 177 (43) | 189 (46) | 195 (47) | 561 (46) |
| More than a year ago | 231 (57) | 218 (54) | 217 (53) | 666 (54) |
| Disclosure of sexual orientation to healthcare provider | ||||
| Provider definitely knows | 138 (34) | 138 (34) | 135 (33) | 411 (34) |
| Provider probably knows or might know | 164 (40) | 156 (38) | 161 (39) | 481 (39) |
| Provider definitely does not know | 106 (26) | 113 (28) | 116 (28) | 335 (27) |
| Concealment of sexual orientation, mean (SD)a | 2.66 (1.19) | 2.59 (1.24) | 2.67 (1.20) | 2.64 (1.21) |
| Perceived discrimination in healthcare, mean (SD)b | 2.45 (1.21) | 2.32 (1.11) | 2.39 (1.21) | 2.38 (1.18) |
| HIV status | ||||
| Negative | 381 (93) | 391 (96) | 391 (95) | 1,163 (95) |
| Positive | 27 (7) | 16 (4) | 21 (5) | 64 (5) |
| Ever have genital warts | ||||
| No | 398 (98) | 388 (95) | 398 (97) | 1,184 (96) |
| Yes | 10 (3) | 19 (5) | 14 (3) | 43 (4) |
| Electronic health literacy, mean (SD)c | 3.96 (0.80) | 3.95 (0.76) | 3.90 (0.75) | 3.94 (0.77) |
Note. Percentages may not total 100% due to rounding. Interactive group=Outsmart HPV content with monthly interactive vaccination reminders; unidirectional group=Outsmart HPV content with monthly unidirectional vaccination reminders; SD=standard deviation; HIV=human immunodeficiency virus.
3-item scale; items had a 5-point response scale ranging from “never” to “always” (coded 1–5)
1 item with a 5-point response scale ranging from “strongly disagree” to “strongly agree” (coded 1–5)
4-item scale; items had a 5-point response scale ranging from “strongly disagree” to “strongly agree” (coded 1–5)
Data Analysis
We examined descriptive statistics for demographic and health characteristics. However, as suggested for randomized trials (31, 32), we did not compare study groups on these characteristics using statistical tests. We used logistic regression models to produce unadjusted odds ratios (ORs) and compare study groups on HPV vaccine initiation and completion via an intent-to-treat approach. For each outcome, we made all three pairwise comparisons between study groups and used a Bonferroni corrected two-sided alpha of 0.017 to control the overall type I error rate. We report 98.3% confidence intervals (CIs) from these models to be consistent with the Bonferroni correction. Following this primary analysis, we used logistic regression models to determine if differential intervention efficacy existed across demographic and health characteristics (i.e., moderation). These models included an interaction term between each potential moderator and study group. We considered moderation to be present if an interaction term had p<0.05.
Lastly, we used independent sample t-tests to compare study groups on participant satisfaction. For satisfaction with online content, we compared the control group to a combined intervention group that included both the interactive and unidirectional groups. We combined them into a single analytic group for this part of the analysis since they received identical Outsmart HPV online content, and participants answered these satisfaction items prior to their first vaccination reminder. For satisfaction with monthly HPV vaccination reminders, we compared the interactive and unidirectional groups. These statistical tests were two-sided with an alpha of 0.05. Analyses were performed with Stata version 15.0 (Statacorp, College Station, TX).
Data Availability
Deidentified data may be made available upon request.
Results
Participant Characteristics
Most participants were ages 22–25 (64%), were single and having sex or casually dating (65%), and had at least some college education (69%) (Table 1). Just over half of participants indicated a minoritized racial/ethnic identity (53%). A majority of participants self-identified as gay (66%) or bisexual (26%), reported a history of oral or anal sex with a male (93%), and reported being sexually attracted to males (97%). Few participants indicated that they were HIV-positive (5%) or had a history of genital warts (4%). Baseline characteristics were highly similar between the three study groups.
HPV Vaccination
Overall, 33% of participants (400/1,227) reported initiating the HPV vaccine series, and 7% (91/1,227) reported completing the three-dose series. A greater proportion of participants in the interactive group reported initiating the HPV vaccine series compared to the control group (36% vs. 28%; OR=1.47, 98.3% CI: 1.03–2.11; p=0.010)(Figure 1). HPV vaccine initiation was more common among participants in the unidirectional group compared to the control group, though the difference did not reach statistical significance after the Bonferroni correction (34% vs. 28%; OR=1.36, 98.3% CI: 0.94–1.95; p=0.045). There was no difference in HPV vaccine initiation between the interactive group and unidirectional group (36% vs. 34%; OR=1.09, 98.3% CI: 0.76–1.54; p=0.580).
Figure 1.
HPV vaccine initiation (receipt of one or more doses) and completion (receipt of all three doses) by study group. Bars indicate standard errors. *p<0.017 in comparison to the control group.
Compared to the control group, a greater proportion of participants in the interactive group (12% vs. 3%; OR=3.70, 98.3% CI: 1.75–7.83; p<0.001) and in the unidirectional group (7% vs. 3%; OR=2.26, 98.3% CI: 1.02–5.00; p=0.014) reported HPV vaccine completion. The interactive and unidirectional groups did not differ from one another on HPV vaccine completion after the Bonferroni correction (12% vs. 7%; OR=1.64, 98.3% CI: 0.91–2.94; p=0.045). None of the demographic and health characteristics moderated the intervention’s effects on HPV vaccine initiation or completion (all interaction terms had p>0.05).
Participant Satisfaction
As shown in Table 2, participants in the intervention group (combining the interactive and unidirectional groups) reported higher levels of satisfaction with their online content compared to the control group. For information quality, participants in the intervention group more strongly endorsed that their information was easy to understand, credible, important, accurate, and relevant (all p<0.05). Participants in the intervention group also more strongly endorsed that they liked how the project website looked, it was easy to use, it loaded all text and graphics quickly, it was easy to go back forth between pages, and it was easy to locate information (all p<0.05). Participants in the intervention group also more strongly endorsed that the study improved their ability to make healthier choices, that they would recommend the study to their friends, and that they were satisfied with the study overall (all p<0.05). For satisfaction with monthly HPV vaccination reminders (Table 3), the interactive and unidirectional groups reported similar levels of satisfaction for all items (all p>0.05).
Table 2.
Participant satisfaction with online content by study group.
| Intervention Group (n=722) |
Control Group (n=379) |
||
|---|---|---|---|
|
|
|||
| mean (SD) | mean (SD) | p | |
|
| |||
| Information quality | |||
| The information was | |||
| Easy to understand | 4.61 (0.58) | 4.37 (0.71) | <0.001 |
| Credible | 4.36 (0.72) | 4.24 (0.75) | 0.008 |
| Important | 4.62 (0.60) | 4.49 (0.74) | 0.001 |
| Accurate | 4.30 (0.77) | 4.19 (0.77) | 0.030 |
| Relevant to me | 4.44 (0.73) | 4.25 (0.80) | <0.001 |
| Project website quality | |||
| Like how it looked | 4.51 (0.67) | 4.22 (0.80) | <0.001 |
| Easy to use | 4.52 (0.69) | 4.31 (0.74) | <0.001 |
| Loads all the text and graphics quickly | 4.56 (0.66) | 4.39 (0.74) | <0.001 |
| Easy to go back and forth between pages | 4.32 (0.89) | 4.16 (0.89) | 0.004 |
| Easy to locate information | 4.35 (0.81) | 4.18 (0.79) | 0.001 |
| Took too long to view information | 2.54 (1.34) | 2.38 (1.21) | 0.060 |
| Study experience | |||
| Improved ability to make healthier choices | 4.31 (0.69) | 4.21 (0.69) | 0.018 |
| Would recommend to friends | 4.32 (0.80) | 4.18 (0.78) | 0.007 |
| Overall satisfaction | 4.43 (0.69) | 4.33 (0.66) | 0.018 |
Note. The “Intervention Group” combined the interactive and unidirectional since these groups received identical Outsmart HPV content online. SD = standard deviation. All items were from the T2 survey and were Likert-type items with a possible range of 1–5. Items were coded so that higher values indicate greater levels of agreement with a statement. p-values represent results from independent samples t-tests. Statistical tests were two-tailed with a critical alpha of 0.05.
Table 3.
Participant satisfaction with monthly HPV vaccination reminders by study group.
| Intervention Group (n=275) |
Unidirectional Group (n=271) |
||
|---|---|---|---|
|
|
|||
| mean (SD) | mean (SD) | p | |
|
| |||
| The reminders were… | |||
| Helpful | 4.00 (0.88) | 3.98 (0.89) | 0.807 |
| Relevant to me | 3.91 (0.98) | 3.89 (0.93) | 0.879 |
| Annoying | 2.74 (1.10) | 2.65 (1.11) | 0.330 |
| Easy to understand | 4.33 (0.69) | 4.29 (0.68) | 0.502 |
| Text messages are a good way to send reminders | 4.17 (0.86) | 4.10 (0.98) | 0.392 |
| The reminders helped me decide whether or not to get the first shot of the HPV vaccine | 3.41 (1.22) | 3.44 (1.15) | 0.781 |
| The reminders helped me remember to get my second and/or third HPV vaccine shotsa | 3.87 (1.12) | 3.84 (1.12) | 0.808 |
Note. HPV=human papillomavirus; interactive group=Outsmart HPV content with monthly interactive vaccination reminders; unidirectional group=Outsmart HPV content with monthly unidirectional vaccination reminders; SD=standard deviation. All items were from the T4 survey and were Likert-type items with a possible range of 1–5. Items were coded so that higher values indicate greater levels of agreement with a statement. p-values represent results from independent samples t-tests. Statistical tests were two-tailed with a critical alpha of 0.05.
Item asked only to those participants who reported initiating the HPV vaccine series
Discussion
Given the current disparities related to HPV infection and HPV-related disease among GBMSM (3–5) and the potential of HPV vaccine to prevent these health outcomes (6), efforts are needed to increase HPV vaccination among YGBMSM. To our knowledge, Outsmart HPV is among the first HPV vaccination interventions for YGBMSM in the US to be developed and evaluated. Building upon promising results from our past work involving this intervention (9–11), results from the current RCT indicate that Outsmart HPV increases both HPV vaccine initiation (when interactive reminders were used) and completion (when either interactive or unidirectional reminders were used). Looking at general patterns in our vaccination outcomes, the interactive group had the largest increases in each outcome, followed by the unidirectional group, and then the control group. This pattern is very similar to a previous study that examined both types of reminders in increasing influenza vaccination (14). We think our findings suggest that Outsmart HPV is efficacious at increasing HPV vaccination among YGBMSM and that interactive reminders potentially add an additional effect in increasing vaccination compared to unidirectional reminders. Importantly, efficacy did not differ across demographic and health characteristics, suggesting that the effects of Outsmart HPV were similar across various subgroups of participants.
There are a few important points to consider when interpreting our results. First, we found positive effects on HPV vaccine uptake even when comparing the intervention groups to a strong control group (i.e., a group that received information about HPV vaccine that was patterned after the VIS) and when using the conservative Bonferroni correction to account for multiple comparisons between study groups. Second, the COVID-19 pandemic began only a few months after recruitment opened for the RCT, and uptake of HPV vaccine and other vaccines declined greatly in the US following the onset of the pandemic (33–35). Our intervention’s effects on HPV vaccine initiation are similar to those from pre-pandemic interventions (36, 37), but it appears that participants in our study were especially challenged in completing the HPV vaccine series during the pandemic. Indeed, HPV vaccine completion was low overall in our study, even among those who reported initiating the vaccine series. Only about a quarter of participants who reported initiating the HPV vaccine series indicated that they had completed it. In comparison, national data show that, prior to the COVID-19 pandemic, over 40% of young men in the US who initiated the HPV vaccine series had completed it (38). As delivery of healthcare services rebounds from the pandemic, future efforts involving Outsmart HPV will allow us to better determine the potential impact that the pandemic had on the current study, including how vaccine hesitancy and trust in healthcare may have changed due to the pandemic (39).
Participants who received Outsmart HPV reported higher levels of satisfaction compared to the control group, including satisfaction with the quality of information received (e.g., understandability, accuracy, relevance), quality of the project website (e.g., appearance, usability), and their study experience (e.g., the ability to make healthier choices). We think these findings are attributable to Outsmart HPV presenting information through a combination of online text and more visual formats (e.g., infographics), whereas the control group materials presented information solely through online text. More visual formats are preferred by adolescents and young adults and may also result in better recall of information (40–42). Additionally, the iterative process used to develop and refine Outsmart HPV, including input from YGBMSM (10, 12, 18), likely increased the intervention’s relevance and cultural appropriateness. Interestingly, satisfaction with vaccination reminders was similar between the interactive and unidirectional groups, which is in contrast to past research that showed parents preferred reminders for their children’s vaccinations to include an interactive component (13).
Given its web-based format, we believe that Outsmart HPV has the potential to be utilized across clinical and other healthcare settings that provide HPV vaccination services to YGBMSM. One of the goals of Outsmart HPV is to help optimize patient-provider interactions about HPV vaccination during clinical visits by priming YGBMSM to talk with healthcare providers about getting vaccinated. Thus, we believe our intervention can help complement other strategies for increasing utilization of preventive healthcare among YGBMSM (e.g., ensuring welcoming spaces in healthcare settings (43), reducing lack of health insurance coverage (44)) and also help reduce missed opportunities for HPV vaccination during clinical visits. Moving forward, it will be important to consider the dissemination and sustainability of Outsmart HPV. This includes possible routes of dissemination and how the intervention can be further optimized for impact and sustainability (e.g., which intervention components should be better emphasized, how can content be more engaging, how can vaccination reminders be improved and sustained).
Potential study limitations include reliance on self-reported data for all variables. However, self-reported HPV vaccination data result in only a 2% net bias compared to medical records among young adults (45). We recruited a convenience sample of participants through online avenues, which could limit generalizability, although nearly all young adults use the internet (46) and over 80% use social media (47). Participants in our study were also demographically similar to YGBMSM from other national studies (19, 48, 49). The current study included only cisgender individuals; given barriers to care that result in low uptake of preventive services and vaccination among transgender and gender diverse young adults (50–52), future research is needed to adapt Outsmart HPV for these populations. Fraudulent accounts occur in web-based research, so we used several recommended strategies for detecting such accounts and minimizing this risk (e.g., reviewing information for similarities between accounts, requiring potential participants to verify their text message number during account setup, inspecting survey data for illogical responses, providing incentives only after study activities are completed) (53, 54). We think the chance of contamination between study groups was low since participants were from throughout the US, the project website required login, and participants received only the materials for their randomized study group.
Findings from this RCT indicate that Outsmart HPV is an efficacious web-based HPV vaccination intervention for YGBMSM. Results also show that Outsmart HPV is highly acceptable to YGBMSM, with participants reporting high levels of satisfaction with intervention content. Future efforts are needed to determine how Outsmart HPV can be best disseminated across settings that provide HPV vaccination services to YGBMSM and to identify how the intervention can be further optimized for impact and sustainability.
Acknowledgements
The research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health under Award Number R37CA226682 that was awarded to P.L. Reiter and A.L. McRee. Additional support was provided from the Center for Health Communications Research at the University of Michigan (P30CA046592) and the Recruitment, Intervention, and Survey Shared Resource at The Ohio State University Comprehensive Cancer Center (P30CA016058). This work was prepared while Dr. McRee was employed at the University of Minnesota. The opinions expressed in this article are the authors’ own and do not reflect the view of the National Institutes of Health, the Department of Human Services or the Unites States Government.
Footnotes
Conflicts of Interest: The authors declare no potential conflicts of interest.
References
- 1.Gillison ML, Chaturvedi AK, Lowy DR. HPV prophylactic vaccines and the potential prevention of noncervical cancers in both men and women. Cancer. 2008;113(10 Suppl):3036–46. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Lacey CJ, Lowndes CM, Shah KV. Chapter 4: Burden and management of non-cancerous HPV-related conditions: HPV-6/11 disease. Vaccine. 2006;24 Suppl 3:S3/35–41. [DOI] [PubMed] [Google Scholar]
- 3.Smith JS, Gilbert PA, Melendy A, Rana RK, Pimenta JM. Age-specific prevalence of human papillomavirus infection in males: A global review. J Adolesc Health. 2011;48(6):540–52. [DOI] [PubMed] [Google Scholar]
- 4.Wei F, Gaisa MM, D’Souza G, Xia N, Giuliano AR, Hawes SE, et al. Epidemiology of anal human papillomavirus infection and high-grade squamous intraepithelial lesions in 29 900 men according to HIV status, sexuality, and age: A collaborative pooled analysis of 64 studies. Lancet HIV. 2021;8(9):e531–43. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Clifford GM, Georges D, Shiels MS, Engels EA, Albuquerque A, Poynten IM, et al. A meta-analysis of anal cancer incidence by risk group: Toward a unified anal cancer risk scale. Int J Cancer. 2021;148(1):38–47. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.U.S. Food and Drug Administration. Gardasil 9. 2020. Available from: https://www.fda.gov/vaccines-blood-biologics/vaccines/gardasil-9.
- 7.Meites E, Szilagyi PG, Chesson HW, Unger ER, Romero JR, Markowitz LE. Human papillomavirus vaccination for adults: Updated recommendations of the advisory committee on immunization practices. MMWR Morb Mortal Wkly Rep. 2019;68(32):698–702. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Nadarzynski T, Frost M, Miller D, Wheldon CW, Wiernik BM, Zou H, et al. Vaccine acceptability, uptake and completion amongst men who have sex with men: A systematic review, meta-analysis and theoretical framework. Vaccine. 2021;39(27):3565–81. [DOI] [PubMed] [Google Scholar]
- 9.Reiter PL, Katz ML, Bauermeister JA, Shoben AB, Paskett ED, McRee AL. Increasing human papillomavirus vaccination among young gay and bisexual men: A randomized pilot trial of the outsmart HPV intervention. LGBT Health. 2018;5(5):325–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.McRee AL, Shoben A, Bauermeister JA, Katz ML, Paskett ED, Reiter PL. Outsmart HPV: Acceptability and short-term effects of a web-based HPV vaccination intervention for young adult gay and bisexual men. Vaccine. 2018;36(52):8158–64. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Reiter PL, Gower AL, Kiss DE, Shoben AB, Katz ML, Bauermeister JA, et al. Effects of a web-based HPV vaccination intervention on cognitive outcomes among young gay, bisexual, and other men who have sex with men. Hum Vaccin Immunother. 2022:2114261. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Reiter PL, Gower AL, Kiss DE, Malone MA, Katz ML, Bauermeister JA, et al. A web-based human papillomavirus vaccination intervention for young gay, bisexual, and other men who have sex with men: Protocol for a randomized controlled trial. JMIR Res Protoc. 2020;9(2):e16294. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Hofstetter AM, Vargas CY, Kennedy A, Kitayama K, Stockwell MS. Parental and provider preferences and concerns regarding text message reminder/recall for early childhood vaccinations. Prev Med. 2013;57(2):75–80. [DOI] [PubMed] [Google Scholar]
- 14.Hofstetter AM, Vargas CY, Camargo S, Holleran S, Vawdrey DK, Kharbanda EO, et al. Impacting delayed pediatric influenza vaccination: A randomized controlled trial of text message reminders. Am J Prev Med. 2015;48(4):392–401. [DOI] [PubMed] [Google Scholar]
- 15.Rogers RW. Cognitive and physiological processes in fear appeals and attitude change: A revised theory of protection motivation. In: Cacioppo JT, Petty RE, editors. Social Psychophysiology: A Source Book. New York: Guilford Press; 1983. p. 153–76. [Google Scholar]
- 16.Fisher JD, Fisher WA. Changing AIDS-risk behavior. Psychol Bull. 1992;111(3):455–74. [DOI] [PubMed] [Google Scholar]
- 17.Meyer IH. Prejudice, social stress, and mental health in lesbian, gay, and bisexual populations: Conceptual issues and research evidence. Psychol Bull. 2003;129(5):674–97. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Malone MA, Gower AL, Reiter PL, Kiss DE, McRee AL. “What does it matter?” young sexual minority men discuss their conversations with sexual partners about HPV vaccination. J Am Coll Health. 2023;71(2):489–95. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Reiter PL, McRee AL, Katz ML, Paskett ED. Human papillomavirus vaccination among young adult gay and bisexual men in the united states. Am J Public Health. 2015;105(1):96–102. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Gerend MA, Madkins K, Phillips G 2nd, Mustanski B. Predictors of human papillomavirus vaccination among young men who have sex with men. Sex Transm Dis. 2016;43(3):185–91. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Nadarzynski T, Smith H, Richardson D, Jones CJ, Llewellyn CD. Human papillomavirus and vaccine-related perceptions among men who have sex with men: A systematic review. Sex Transm Infect. 2014;90(7):515–23. [DOI] [PubMed] [Google Scholar]
- 22.Fontenot HB, Fantasia HC, Vetters R, Zimet GD. Increasing HPV vaccination and eliminating barriers: Recommendations from young men who have sex with men. Vaccine. 2016;34(50):6209–16. [DOI] [PubMed] [Google Scholar]
- 23.Wheldon CW, Daley EM, Walsh-Buhi ER, Baldwin JA, Nyitray AG, Giuliano AR. An integrative theoretical framework for HPV vaccine promotion among male sexual minorities. Am J Mens Health. 2018;12(5):1409–20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Cummings T, Kasting ML, Rosenberger JG, Rosenthal SL, Zimet GD, Stupiansky NW. Catching up or missing out? human papillomavirus vaccine acceptability among 18- to 26-year-old men who have sex with men in a US national sample. Sex Transm Dis. 2015;42(11):601–6. [DOI] [PubMed] [Google Scholar]
- 25.Centers for Disease Control and Prevention, (CDC). Vaccine information statements (VISs) - HPV (human papillomavirus). 2019. Available from: https://www.cdc.gov/vaccines/hcp/vis/vis-statements/hpv.html.
- 26.Meites E, Kempe A, Markowitz LE. Use of a 2-dose schedule for human papillomavirus vaccination - updated recommendations of the advisory committee on immunization practices. MMWR Morb Mortal Wkly Rep. 2016;65(49):1405–8. [DOI] [PubMed] [Google Scholar]
- 27.Horvath KJ, Oakes JM, Rosser BR, Danilenko G, Vezina H, Amico KR, et al. Feasibility, acceptability and preliminary efficacy of an online peer-to-peer social support ART adherence intervention. AIDS Behav. 2013;17(6):2031–44. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Meidlinger PC, Hope DA. Differentiating disclosure and concealment in measurement of outness for sexual minorities: The nebraska outness scale. Psychology of Sexual Orientation and Gender Diversity. 2014;1(4):489–97. Available from: https://search.proquest.com/docview/1634728645. [Google Scholar]
- 29.Rainbow Health Initiative. Voices of health: A survey of LGBTQ health in Minnesota. 2013. Available from: http://www.rainbowhealth.org/resources-for-you/research-and-publications/.
- 30.Norman CD, Skinner HA. eHEALS: The eHealth literacy scale. J Med Internet Res. 2006;8(4):e27. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.de Boer MR, Waterlander WE, Kuijper LD, Steenhuis IH, Twisk JW. Testing for baseline differences in randomized controlled trials: An unhealthy research behavior that is hard to eradicate. Int J Behav Nutr Phys Act. 2015;12:4-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: Updated guidelines for reporting parallel group randomised trials. J Clin Epidemiol. 2010;63(8):1. [DOI] [PubMed] [Google Scholar]
- 33.Daniels V, Saxena K, Roberts C, Kothari S, Corman S, Yao L, et al. Impact of reduced human papillomavirus vaccination coverage rates due to COVID-19 in the united states: A model based analysis. Vaccine. 2021;39(20):2731–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.O’Leary ST, Trefren L - Colorado Department of Public Health and Environment, Roth H, Moss A, Severson R, Kempe A. Number of childhood and adolescent vaccinations administered before and after the COVID-19 outbreak in colorado. JAMA Pediatr. 2021;175(3):305–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Santoli JM, Lindley MC, DeSilva MB, Kharbanda EO, Daley MF, Galloway L, et al. Effects of the COVID-19 pandemic on routine pediatric vaccine ordering and administration - united states, 2020. MMWR Morb Mortal Wkly Rep. 2020;69(19):591–3. [DOI] [PubMed] [Google Scholar]
- 36.Smulian EA, Mitchell KR, Stokley S. Interventions to increase HPV vaccination coverage: A systematic review. Hum Vaccin Immunother. 2016;12(6):1566–88. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Walling EB, Benzoni N, Dornfeld J, Bhandari R, Sisk BA, Garbutt J, et al. Interventions to improve HPV vaccine uptake: A systematic review. Pediatrics. 2016;138(1): 10.1542/peds.2015,3863. [DOI] [PubMed] [Google Scholar]
- 38.Adjei Boakye E, Lew D, Muthukrishnan M, Tobo BB, Rohde RL, Varvares MA, et al. Correlates of human papillomavirus (HPV) vaccination initiation and completion among 18–26 year olds in the united states. Hum Vaccin Immunother. 2018;14(8):2016–24. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Liu R, Li GM. Hesitancy in the time of coronavirus: Temporal, spatial, and sociodemographic variations in COVID-19 vaccine hesitancy. SSM Popul Health. 2021;15:100896. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Trevena LJ, Zikmund-Fisher BJ, Edwards A, Gaissmaier W, Galesic M, Han PK, et al. Presenting quantitative information about decision outcomes: A risk communication primer for patient decision aid developers. BMC Med Inform Decis Mak. 2013;13 Suppl 2:S7,S7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Houts PS, Doak CC, Doak LG, Loscalzo MJ. The role of pictures in improving health communication: A review of research on attention, comprehension, recall, and adherence. Patient Educ Couns. 2006;61(2):173–90. [DOI] [PubMed] [Google Scholar]
- 42.Garcia-Retamero R, Cokely ET. Effective communication of risks to young adults: Using message framing and visual aids to increase condom use and STD screening. J Exp Psychol Appl. 2011;17(3):270–87. [DOI] [PubMed] [Google Scholar]
- 43.McClain Z, Hawkins LA, Yehia BR. Creating welcoming spaces for lesbian, gay, bisexual, and transgender (LGBT) patients: An evaluation of the health care environment. J Homosex. 2016;63(3):387–93. [DOI] [PubMed] [Google Scholar]
- 44.Bosworth A, Turrini G, Pyda S, Strickland K, Chappel A, De Lew N, et al. Health insurance coverage and access to care for LGBTQ+ individuals (issue brief no. HP-2021–14). Office of the assistant secretary for planning and evaluation, U.S. department of health and human services. 2021. Available from: https://aspe.hhs.gov/sites/default/files/2021-07/lgbt-health-ib.pdf. [Google Scholar]
- 45.Rolnick SJ, Parker ED, Nordin JD, Hedblom BD, Wei F, Kerby T, et al. Self-report compared to electronic medical record across eight adult vaccines: Do results vary by demographic factors? Vaccine. 2013. Aug 20;31(37):3928–35. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Pew Research Center. Internet/broadband fact sheet. 2021. Available from: https://www.pewresearch.org/internet/fact-sheet/internet-broadband/.
- 47.Pew Research Center. Social media fact sheet. 2021. Available from: https://www.pewresearch.org/internet/fact-sheet/social-media/.
- 48.Fasula AM, Oraka E, Jeffries WL 4th, Carry M, Banez Ocfemia MC, Balaji AB, et al. Young sexual minority males in the united states: Sociodemographic characteristics and sexual attraction, identity and behavior. Perspect Sex Reprod Health. 2016;48(1):3–8. [DOI] [PubMed] [Google Scholar]
- 49.Strutz KL, Herring AH, Halpern CT. Health disparities among young adult sexual minorities in the U.S. Am J Prev Med. 2015;48(1):76–88. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.McRee A, Gower AL, Reiter PL. Preventive healthcare services use among transgender young adults. Int J Transgend. 2018;19(4):417–23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Rider GN, McMorris BJ, Gower AL, Coleman E, Eisenberg ME. Health and care utilization of transgender and gender nonconforming youth: A population-based study. Pediatrics. 2018;141(3):e20171683. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Amiling R, Winer RL, Newcomb ME, Gorbach PM, Lin J, Crosby RA, et al. Human papillomavirus vaccination coverage among young, gay, bisexual, and other men who have sex with men and transgender women - 3 U.S. cities, 2016–2018. Hum Vaccin Immunother. 2021;17(12):5407–12. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.Teitcher JE, Bockting WO, Bauermeister JA, Hoefer CJ, Miner MH, Klitzman RL. Detecting, preventing, and responding to “fraudsters” in internet research: Ethics and tradeoffs. J Law Med Ethics. 2015;43(1):116–33. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Xu Y, Pace S, Kim J, Iachini A, King B, Harrison T, et al. Threats to online surveys: Recognizing, detecting, and preventing survey bots. Social Work Research. 2022;46(4):343–50. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
Deidentified data may be made available upon request.