Feasibility, acceptability, and process indicators for Guy2Guy, an mHealth HIV prevention program for sexual minority adolescent boys

Abstract

Purpose:

Guy2Guy is one of the first mHealth HIV prevention programs for sexual minority boys aged 14–18 years old, evaluated nationally. Here, we examine the program’s feasibility and acceptability and explore participants’ feedback about program content and components intended to invigorate program engagement.

Methods:

Guy2Guy was tested in a randomized controlled trial (RCT) of 302 youth assigned to either the intervention or an attention-matched control group. At three-month follow-up, participants completed a survey that included questions about feasibility and acceptability. Focus groups were conducted with a subset of intervention participants (n=45) to further understand their program experience.

Results:

The protocol and program appeared to be feasible: 94% completed the 3-month follow-up survey. The intervention also appeared to be acceptable: 93% of intervention participants said they somewhat or strongly agreed that they liked the program. Although ~20% boys agreed that the program sent too many messages, only 10% said they stopped reading the messages by the end. Focus group participants were largely enthusiastic about program content and generally appreciated receiving information and skills-building messages that talked about HIV risk reduction. Some indicated a desire for more content that addressed condom negotiation. Program engagement components, particularly the weekly ‘level up’ quiz, also were generally well received.

Conclusions:

Sexual minority boys are willing to engage in Guy2Guy, an intensive, multiweek sexual health intervention via text messaging, and most would recommend the program to their friends.

Background

Adolescents and young adults in the U.S. are among the highest groups at risk for HIV, representing more than 20% of those with new infections each year.¹ Male youth who identify as gay or bisexual are substantially overrepresented among those infected. Indeed, 8 of every 10 new cases of HIV among youth occur in gay and bisexual men.¹

Close to two-thirds (65%) of the new HIV infections among young gay men occur through male-to-male sexual contact without condoms.² These sobering data underscore a critical gap in our HIV prevention efforts. To impact the trajectory of the HIV epidemic in the U.S., we must develop, evaluate, and disseminate evidence-based interventions that focus on modifiable risk factors and that are effective in reaching and engaging groups at elevated risk for HIV acquisition.³

Because adolescents are among the most prolific consumers of cell phones and text messaging,⁴ it is useful to consider whether text messaging could be used as a delivery method of content aimed to change complex adolescent behaviors. Several studies, research syntheses, and meta-analyses for behaviors such as medication adherence, smoking, sexual health, nutrition, and physical activity suggest that behavior change is possible with programs sent by text message.^5–10 Moreover, text messages are low cost, simple to deliver, and easy to replicate or adapt for different groups and subgroups of interest.

The proliferation of technology-based initiatives for health promotion demands that we dig deeper to learn which elements of a program are more or less compelling so that we can maximize program impact.¹¹ As of now, we know only that exposure to technology-based programs can affect behavior. To better explore questions of engagement, we must first consider message design. Emerging research suggests that careful application of health communication theory can facilitate greater engagement with messages. There is evidence that engagement with messages wanes over time, and messages that are presented as quizzes or questions may be persuasive.¹² On social media, messages that are more often shared within networks are those that make the sender look intelligent or funny, offer an easy-to-act-upon trigger, evoke a powerful emotion, remind readers of a brand or product, or tell a story that resonates.^12–13 Variation in engagement with message content by gender, sexual orientation and race/ethnicity suggests that we can potentially have the capacity to better engage some groups and ensure that messages are appealing for the intended audience.¹²

Although this literature offers strategies in making our messages more compelling, we are only beginning to understand the relationship between greater engagement with text message content and its impact on program effects. In this article, we share feedback from male youth identifying as gay, bisexual, and/or queer about their experience in the Guy2Guy (G2G) text messaging-based HIV prevention program. The content was guided by the Information Motivation Behavioral (IMB) Model of HIV preventive behavior.¹⁴ Intervention development has been described elsewhere.¹⁵ Randomized Controlled Trial (RCT) outcomes have also been previously reported. In brief, analyses suggested that youth in the intervention were three times more likely to be tested for HIV than youth in the control group.¹⁶ There also was suggestion that condomless sex acts may have been lower for the intervention versus control group, although age mediated the relation. Among sexually inexperienced youth, those in the intervention versus control were three times as likely at follow-up to be in the “high motivation” group, which includes intentions to use condoms.¹⁷ Here, we examine the feasibility and acceptability of this intensive intervention, and explore whether adolescents thought that the program content affected their condom use behavior, and how they perceived components intended to invigorate program engagement.

Methods

Protocols for the study were reviewed and approved by the Chesapeake Institutional Review Board (IRB). A waiver of parental permission was granted. As such, youth provided verbal informed assent/consent at the time of enrollment.

Guy2Guy is a mobile phone-based program using text messages to deliver educational and skill-oriented information to 14- to 18-year-old males who identify as gay, bisexual, and/or queer. The program was evaluated in a two-arm RCT. Eligible participants were 14- to 18-year old English-speaking cisgender males who identified as gay, bisexual, and/or queer and exclusively owned a cell phone with an unlimited text messaging plan. Youth needed to have had at least 6 months of text messaging experience and intentions to maintain their unlimited text messaging plan for 6 months. Participants were recruited to participate in the Guy2Guy study through online advertising on Facebook and Instagram between June and October of 2014. Persons clicking on the ads were linked to an online screener and those determined eligible from the screener were contacted and enrolled by a study coordinator over the telephone. After completing an online baseline assessment, youth were randomly assigned to the intervention (n=150) or control (n=152) arm. To ensure attention control, those in the control arm received an equal number of text messages as those in the intervention arm. Control messages focused on general health topics such as self-esteem, positive body image, and dealing with bullying.

Primary outcomes for the Guy2Guy intervention were condom use and abstinence. HIV testing was a secondary outcome. To promote behavioral determinants, an average of 8-10 text messages were sent daily over a 5-week period. The number of messages were confirmed to be acceptable in the formative focus groups (FGs).¹⁵ The number varied based on the topic (e.g., how to put on a condom was a longer day). A similarly intensive 1-week booster was delivered 6 weeks after the end of this ‘core’ 5-week period to reinforce new behaviors. Four strategies were used to increase participant engagement during the intervention: 1) Participants could earn “Badges” that were virtual emblems to signify skill mastery if they successfully achieved a task (i.e., getting, carrying, or using condoms, getting tested for HIV). 2) They could “Level Up” (i.e., move to the next level) each week by correctly answering a quiz question. Questions tested comprehension of critical points made in the past week’s content. 3) Participants also had access to G2Genie, an on-demand advice feature on pre-determined topics such as sexually transmitted infection testing, relationships, and coming out. 4) Participants were also paired and encouraged to discuss what they are learning in the program with their “Text Buddy”. This person was another intervention participant with whom they could chat and discuss the program content anonymously (a study by Ybarra et al. [18] gives more detail on the method used to protect anonymity and prevent participants from meeting).

Recruitment and data collection

Detail on the RCT recruitment and enrollment process for the study can be reviewed in Ybarra et al.¹⁶ In brief, RCT participants were invited to complete an online assessment at 3-month postintervention. Those who did not respond were invited to complete a briefer, text messaging-based assessment that queried only the outcome measures. Intervention acceptability and tolerability for all program features were queried in the online survey. Participants were emailed a $20 Amazon gift card for their completion of the online survey. If they completed the survey within 48 hours of receiving the survey invitation, they earned an additional $10 incentive.

We conducted three online, asynchronous FG discussions with Guy2Guy participants who had completed the 3-month online quantitative survey. Because of the format, the FGs can be larger than traditional groups. As such, instead of 8–10 participants, we targeted 15–20 per group. Of the 132 Guy2Guy intervention study participants who completed the 3-month postintervention online survey, 45 were purposively selected to take part in the asynchronous online discussions. To query participants on which messages were more or less helpful in invigorating condom use specifically, we established three groups: One group with participants who reported no change in their condom use, a second reporting an increase in condomless sex acts, excluding oral sex, in the past 3 months; and a third who reported a decrease in condomless sex acts, excluding oral sex, in the past 3 months. Invited youth visited an online, password-protected bulletin board and responded anonymously to the moderator’s questions and commented on other study participant responses.

Data analysis

Data from the quantitative survey were tabulated. Differences between control and intervention participants were tested using chi-square statistics. Review of transcripts and thematic analyses were used to analyze content from the FG discussions. The first stage in this process included open coding, where transcripts were coded based on a priori set of codes that were anticipated based on the FG topic guide. In the second stage of coding, we reviewed transcripts to ascertain the relationships between codes, and in the final stage, we summarized key themes and outliers to synthesize results.¹⁹

Measures

We assessed intervention acceptability and tolerability for both intervention and control participants using measures adapted from those created for a text messaging-based smoking cessation program for young adults.²⁰ Five items regarding the text buddy aspect of Guy2Guy were asked of those in the intervention arm. In addition, three questions about the text messaging aspect of the intervention were assessed (e.g., “How much did the text messages talk about things that you and people you know experience?”), and a final open-ended item was included: “If you have any other opinions or thoughts about Guy2Guy that you would like to share, please write in the text below.”

Intervention feasibility was measured by recruitment and retention rates. We determined that a recruitment rate of at least 40 participants per month and a retention rate of at least 80% at 3-months would be supportive of a hypothesis of feasibility.

We asked participants in the FGs discussion to comment on how the program changed their views on condoms, if at all; how well the use of badges, “leveling up”, and the use of reviews at the end of each week helped them to improve intentions and behaviors for key actions, including buying, carrying, or using condoms, and testing for STIs including HIV.

Results

Feasibility

We recruited an average of 15.9 participants per week for the RCT, equivalent to 63.6 participants in a 4-week month. This was well above our 40 participants per month threshold identified as the number needed for feasibility. At 94% (n = 283), the 3-month retention was well above the 80% threshold for feasibility.

Acceptability and tolerability

A total of 275 RCT participants completed the online survey and 16 completed the text messaging-based survey at 3 month follow-up (8 completed both surveys). As shown in Table 1, among the intervention participants who completed the online assessment (n=132), there was strong indication of intervention acceptability and tolerability. For example, 93% said they somewhat or strongly agreed that they liked the program and 75% said they were somewhat or very likely to recommend the program to other guys their age. More than 90% agreed that the text messaging content was easy to understand. That said, 22% agreed that the program talked too much about condoms and 10% agreed that G2G talked too much about HIV and other STDs. Although 20% agreed that the program sent too many messages, only 10% said they stopped reading the messages by the end of the program. Five percent said G2G disrupted their daily schedule. When asked to rank the program components, 85% said the content delivered through text messages was their favorite; <10% said that the Text Buddy, Level up, G2Genie, and badge features were their favorite.

Table 1.

Guy2Guy acceptability and feasibility ratings among intervention participants (n=132)

Acceptability of intervention components
General likability
I liked the “level up” questions	81.8% (108)
I liked G2Genie	69.7% (92)
I liked the G2G badges	54.5% (72)
Rating of program components (No. of participants who  ranked the component #1 of the 5 components)
Text messages (content)	84.9% (112)
Text buddy	6.8% (9)
Level up questions	6.1% (8)
G2Genie	1.5% (2)
Badges	.8% (1)
Specific feedback about the program components
My text buddy became a really good friend	25.0% (33)
G2Genie topics spoke to issues that teens like me are going through	81.1% (107)
The badges helped me feel more comfortable with condoms (getting, carrying, and using them)	39.4% (52)
The badges made it easier for me to get an HIV test	11.4% (15)
I would have liked more days in between the badge messages	27.3% (36)
The “level up” questions made it easier to remember things in the G2G program	78.0% (103)

Control participants not asked these questions. Percentages reflect the number who Agreed or Strongly agreed with the statement.

Similarly, youth in the control group appraised their experience highly. For example, 85% said that the content talked about things relevant to themselves and their friends. Less than one in 10 (9%) said they stopped reading the messages by program end. This suggests that the attention control was also acceptable and tolerable. Aside from the statement: “G2G talked too much about condoms”, which was endorsed significantly more often by intervention than control youth (p<0.001), youth in both arms rated the Acceptability and Content items similarly (all p>.05), suggesting that the control group was successfully masked.

Program use statistics

Ninety-eight percent of intervention participants sent or received at least one Text Buddy message. The median number was 129 (Range: 0–3167). Similarly high numbers completed the weekly level-up questions: In Weeks 1-5, more than 90% answered the questions correctly. Four in five (80%) intervention participants earned at least one condom-related badge, and almost one in three (31%) earned the HIV testing badge. The level of participation with these features (Text Buddy, the badges, leveling up) did not demonstrate a pattern of benefit with relation to either condom use or abstinence among intervention participants, however.

Qualitative feedback about the program content and engagement strategies

Of the 45 participants invited to take part in the asynchronous FGs discussions, 19 (42%) posted responses to the queries about engagement with various program elements: Five from the group who made no change in condom use, six from the group who increased condomless sex, and eight from the group who reported decreased condomless sex over time.

Participants were largely enthusiastic about program content, and generally appreciated receiving information and skills-building content related to HIV risk reduction. “I liked the whole goal of the program,” “It was useful,” and “It made [sex] seem less scary” represent enthusiasm for the overall program content. “It made me realize the importance of condoms” and “G2G provided me with the useful information…to make the best decisions for myself around condom use” are examples of appreciation for skills-building content.

Participants were asked whether G2G influenced their knowledge of or attitudes towards condom use. Participants who had increased condomless sexual activity and those whose condom use did not change, in general, did not indicate any change in their knowledge about condoms or attitudes towards them, and had both positive (“I’ve always felt they are generally smart to use”) and negative (“They just aren’t for me”) attitudes towards them. Participants who had decreased condomless sexual activity explicitly expressed improved attitudes towards condoms as exemplified by this quote:

“It has made me realize the importance of condoms. Just about every time I didn’t use a condom, I thought about G2G later on. I primarily didn’t use a condom because I was in a relationship but when it ended, I had plans on using condoms, and would just get caught in the moment”

Although the enthusiasm for program content was consistent across FGs participants, when queried about strategies and approaches to improve program content, participants indicated a desire for more content that addressed specific skills, that is, “How to tell a persistent guy you want to use a condom,” “Tips on how to have condoms on hand,” “Ways to get condoms,” and “Where to keep condoms that are not in your wallet.”

Participants in the FGs discussions had a mixed response to the idea of using badges to incentivize behavior. Those who had no change in condom use throughout the program found these useful, whereas others felt that this was not as valuable as an approach to increasing engagement with content. “[Badges] were useful and encouraged me to carry condoms” said one young man whose condom use did not change during the program; “To me it felt gimmicky…I mean what are we the sex ed Boy Scouts?” said another young man who had increased condom use over the course of the program.

In contrast, the idea of “leveling up” was more universally appreciated. “It encouraged me to review the discussed material. There should be more incentives and rewards for leveling up. Maybe a leader board to show who is on top,” said one participant who had no change in condom use during the program period. Others who had increased condomless sex indicated, “It was kind of fun to level up,” and “These were useful in proving how well we sifted through all the info that was sent at us.” Although participants did seem to appreciate the level up concept and activities, they had suggestions for how to improve this engagement strategy. One indicated that “The questions were too easy,” suggesting we make this more competitive, and another suggested we offer the opportunity to “level up” more frequently than once a week.

We also explored participant reactions to the booster content delivered 6 weeks after the end of the ‘core’ intervention. This too was well received by most FG participants, who indicated the review week “Helped me to remember and retain the information,” “Helped…retain more knowledge by doing the review just like studying for a test,” and “These were useful in proving how well we sifted through all the info that was sent at us.” At the same time, participants also had suggestions for how to improve this feature, including making it more frequent (e.g. every week, rather than over a longer period). Unlike the leveling up component, participants felt it would be helpful for the booster to be more educational and less competitive.

Discussion

Results presented here offer important contributions to help us better understand how to make text messaging-based health promotion programs engaging for youth. Perhaps the most crucial take away is the importance of ensuring that the content itself is engaging. Gamification may be helpful, but what will keep teens engaged is the quality of the content. Indeed, the most popularly rated element of the Guy2Guy program was the text messages. Moreover, unlike previous work that found engagement with messaging waned over time,¹² the vast majority of participants in the program (>90%) – both intervention and control – said that they continued to read the text messages through the end of the program. It is also useful to note that the intensity and length of the program seem to be feasible. As such, questions about whether youth are willing to receive a multitude of messages each day over a long period of time seem to be answered with a resounding ‘yes’, at least for this population and about this topic. Extensive formative work was conducted to ensure that the content was salient and spoke to the lived experiences of gay, bisexual, and queer adolescent men.²¹ These findings suggest that this developmental work was fruitful.

It is interesting to note 22% of intervention youth agreed that the content focused too much on condom use. This highlights the importance of discussing additional topics beyond those that are singularly focused on HIV preventive behaviors. Indeed, Guy2Guy discussed healthy relationships, relationship violence, and other non-condom-related content. Perhaps in future iterations of the program, youth could be given the option to receive additional messages about different topics (e.g., fitness) if they feel like they are hearing too much about condoms. Also, although only 10% said they stopped reading the messages by the end, 20% agreed that there were too many text messages. This suggests that for a minority of youth, a less intensive program may be better received. Future iterations might let youth decide how many messages they would like to receive a day, and perhaps the program content could be spaced out accordingly.

Of the features aimed at increasing program engagement, the “level up” questions appeared to be the most popular – 82% of youth agreed that they liked them in the quantitative survey. Less acceptable seemed to be the badges, which only half of the youth who completed the quantitative survey said they liked. This relative mix of approval ratings from FG and RCT participants for these various features has been noted in other studies of “gamification” of mobile and digital health solutions, ^22–23 suggesting we still do not have a standard “recipe” of features that lead to greater engagement. Perhaps instead of one recipe for all, future interventions could offer youth the choice to opt-in or -out of each of these features, thereby allowing the user to tailor their intervention experience.

Limitations.

Perhaps the largest limitation is that these data were collected from youth who completed the follow-up survey as well as those who agreed to take part in the FGs. Efforts to engage youth who had dropped out of the study to understand their intervention experience may have provided additional clues for how to improve program engagement for future users. Also, youth were largely recruited into the Guy2Guy RCT through Facebook and Instagram. It is possible that these or other gamification features would be differently appraised by youth recruited through other venues (e.g., game sites, school). It also should be noted that the RCT sample comprised youth who owned a cell phone and were enrolled in an unlimited texting. Youth who do not meet these criteria may have reported a different program experience.

Future directions

Feedback from gay, bisexual, and/or queer adolescent teens in the Guy2Guy RCT suggest that this intensive, text messaging-based HIV prevention program is feasible and acceptable, and that 14–18 year old sexual minority boys self-reported that they would recommend the program to their friends.

Although the evidence for text messages to impact behaviors is increasingly strong, we still lack the precise knowledge of how to design engaging content, how to calibrate dose and timing of messages, and whether a specific dose is needed to achieve or increase targeted behavioral outcomes.^7–8 Future research could exact a more systematic exploration of which program elements contribute to better program engagement, and whether this increased engagement is associated with greater odds of behavior change. This latter effort can be accomplished through dose-response assessments, which would require careful tracking of whether and how many messages within a program are read and responded to, and then linking this dose with the outcomes for participants. The advances in machine learning and tracking of technology-based interventions, including texting interventions, might make this more feasible.

Implications and contributions:

Little is known about whether intensive mHealth programs are feasible and acceptable to adolescents. This process evaluation of Guy2Guy suggests that sexual minority boys will engage in a multi-week sexual health intervention via text messaging, and that 14–18 year-old sexual minority boys would recommend the program to their friends.