The effect of a text messaging based HIV prevention program on sexual minority male youths: A national evaluation of information, motivation and behavioral skills in a randomized controlled trial of Guy2Guy

Abstract

Introduction:

There is a paucity of literature documenting how the constructs of the Information-Motivation-Behavioral Skills (IMB) model are affected by exposure to technology-based HIV prevention programs. Guy2Guy, based on the IMB model, is the first comprehensive HIV prevention program delivered via text messaging and tested nationally among sexual minority adolescent males.

Methods:

Between June-November 2014, 302 14–18 year old gay, bisexual, and/or queer cisgender males were recruited across the US on Facebook and enrolled in a randomized controlled trial testing Guy2Guy versus an attention-matched control program.

Results:

Among sexually inexperienced youth, those in the intervention were more than three times as likely to be in the “High motivation” group at follow-up as control youth (aOR=3.13; P-value=.04). The intervention effect was not significant when examined separately for those who were sexually active. HIV information did not significantly vary by experimental arm at 3 months post-intervention end, nor did behavioral skills for condom use or abstinence vary.

Conclusions:

The increase in motivation to engage in HIV preventive behavior for adolescent males with no prior sexual experience is promising, highlighting the need to tailor HIV prevention according to past sexual experience. The behavioral skills that were measured may not have reflected those most emphasized in the content (e.g., how to use lubrication to reduce risk and increase pleasure), which may explain the lack of detected intervention impact.

Introduction

Youth between the ages of 13 and 24 account for one in five (22%) of all new HIV diagnoses in the United States ^[1]. Among these cases, adolescent gay and bisexual men are disproportionally represented, accounting for 80% of diagnoses among youth. HIV infections among adolescent gay and bisexual men are largely transmitted via condomless sex. Among male adults and adolescents, 76% of new infections are attributed to male-to-male sexual contact ^[2]. Given the epidemiology of HIV transmission among adolescents in the United States, there is an urgent need to design salient, theory-driven HIV prevention programs that address the unique sexual health needs of adolescent men who have sex with men and focus on modifiable risk factors that have the potential to change the trajectory of new transmissions ^[3].

Using technology to reach adolescent gay and bisexual men.

Certainly, a key aspect of engaging hard-to-reach populations, such as adolescent gay and bisexual men, is being able to go to where they ‘are’. Technology, particularly the Internet and text messaging, represent such an opportunity. Importantly, cell phone ownership is high across racial and ethnic groups, and income levels ^[4], and overcomes many structural challenges and access issues, which is critical when targeting underserved populations. Technology-based interventions are also cost-effective: Compared to the high personnel and infrastructure costs of in-person interventions, web-based interventions cost the price of the server and software updates; text messaging-based interventions have the additional price of sending and receiving text messages (<2 cents per message). As such, interventions targeted to adolescent gay and bisexual men are integrating technology into their delivery model more and more ^[5,6,7,8].

Using the Information-Motivation-Behavioral Skills (IMB) Model to guide intervention content.

A review of HIV prevention programs finds that effective programs are based on strong theoretical models ^[9]. The IMB model ^[10,11] is one such theoretical model that is often used to guide the development of content that aims to affect HIV preventive behavior. It posits that one’s information and knowledge about how to prevent HIV, one’s motivation to perform HIV preventive behaviors, and one’s behavioral skills to engage in HIV preventive behavior are all needed to affect HIV preventive behavior. For example, improving one’s motivation to engage in preventive behaviors is expected to lead to an increase in HIV preventive behavioral skills, which then leads to an increase in enactment of HIV preventive behavior.

Associations between the IMB model and sexual behavior are generally well supported ^{[10,11,12,13,14,15,16,17]}, including in samples of MSM adults ^[14] and youth ^[18]. Furthermore, research has shown that intervention programs guided by the IMB model have resulted in increases in HIV preventive behavior among both sexual minority adults ^[19] and adolescents ^[20], as well as adolescents more generally ^[21,22]. That said, there is a paucity of literature that examines the effect of technology-based HIV interventions on all three constructs. Two technology-based studies targeted to young men who have sex with men have reported the programs’ associations with the IMB Model descriptively; however, both have sample sizes of less than 75 and neither report a control group ^[6,8]. Keep It Up!, an interactive, online HIV prevention intervention among HIV-negative, sexually active young MSM recruited across several LGBT-focused clinics in Chicago resulted in lower rates of unprotected anal sex and condom failure ^[20]. Improvements in each IMB construct underpin this finding, although the constructs themselves were not specifically tested. Bowen and colleagues report statistically significant increases in condom use with anal sex partners among adult, rural MSM in an Internet-delivered intervention, backed by improvements in knowledge, motivation, and behavioral skills ^[19], although again, the three components of the IMB model do not appear to have been tested individually.

To our knowledge, none of these previously published studies include sexually inexperienced participants. Adolescence is a period of intense sexual development and exploration ^[23]. As children mature through childhood into adolescence, they gain a greater sense of their sexual self ^[24], enhanced by interplay of biological and social changes. At the same time that sexual interest increases, development of brain capacities that moderate risk taking (e.g., impulse control, delay of gratification) continue to be underdeveloped ^[25]. This tension creates a ‘perfect storm’ that may heighten the likelihood of engaging in HIV risk behaviors. It also suggests that the impact of behavior change and associated indicators may vary based upon one’s sexual experiences.

Gap in the literature addressed in this paper.

Guy2Guy is an HIV prevention program that was developed for and tested among adolescent gay and bisexual men. Content is delivered via text messaging based on the hypothesis that proactive, automated delivery may increase the likelihood of messages being read versus reactive programs that require a log on or other action by the respondent to access the messages ^[26]. Also, most teenagers carry their cell phones with them everywhere, making program messages close at hand. The intervention is innovative in several key ways: It is the first HIV prevention program for sexual minority youth that has been developed and tested nationally; it is a comprehensive HIV prevention program delivered via text messaging; and includes sexual minority youth as young as 14 years of age. The inclusion of sexually inexperienced youth alongside sexually experienced youth further increases the program’s novelty. Here we report how Guy2Guy affected the IMB model constructs in a randomized controlled trial with an attention-matched control group. To our knowledge, this is one of the first manuscripts to examine the effect of a low-cost technology-based HIV intervention on all three constructs, using an experimental design. Behavioral outcomes have been reported separately ^[27].

Methods

The research protocol was reviewed and approved by both the Chesapeake Institutional Review Board and the Northwestern University Institutional Review Board. A waiver of parental permission was obtained to prevent youth from needing to disclose their sexual identity to their parents to participate in the study ^[28,29]. The clinical trial registration number is: NCT02113956. The protocol for this trial and supporting CONSORT checklist are available upon request.

Between June – November 2014, 302 gay, bisexual, and/or queer males from across the United States were enrolled into a randomized controlled trial testing a text messaging-based HIV prevention program (Guy2Guy) ^[30,31]. In order to participate in this study, individuals needed to: 1) be between 14 and 18 years of age at baseline; 2) be cisgender (i.e., male sex and male gender identity); 3) be English speaking; 4) identify as gay, bisexual, and/or queer; 5) be the exclusive owner of a cell phone with an unlimited text messaging plan; 6) have used text messaging for at least 6 months; and 7) intend to have the same number for the next 6 months.

All participants were recruited through national advertisements on Facebook. Multiple ad images and text were used. An example advertisement read: “We need gay&bi teen guys to take part in a healthy sexuality program.” An example ad headline said: “Be part of research!” To ensure a demographically representative sample, targeted ads were also used to enroll individuals from particular subgroups who are typically more difficult to engage in research (e.g., 14-year-olds, Black males). Anyone who clicked on the link in the Facebook ad was directed to an online screener form to complete. Individuals who appeared eligible were contacted by research staff, sequentially based on recruitment targets, to confirm eligibility; those who were ineligible were emailed HIV prevention resources. Recruitment targets included the following: sexual experience (50% experienced), race (65% White, 20% Black, and 15% other race), ethnicity (20% Hispanic), age (40% 14–15 years and 60% 16–18 years), and urban versus rural living situation determined by ZIP code (80% urban and 20% rural). These recruitment targets were developed through reviewing U.S. Census and American Community Survey data ^[32]. We also endeavored to oversample minority populations (e.g., 13% of people living in the United States identified as Black, so the target was increased to 20%). Once eligibility was determined, the participant was provided more details about the study and verbal assent/consent was obtained. The participant was then sent a survey link to the baseline survey via text message and email.

An individual was not considered enrolled in the study until they completed the baseline survey and were randomized into one of the study arms. Of the 321 gay, bisexual, and/or queer males who were eligible and provided consent, 302 (94.1%) completed the baseline assessment and were enrolled (see Supplementary Figure I).

As described elsewhere ^[27], once enrolled, participants were randomly assigned to either the Guy2Guy intervention or an attention-matched control group. The assignment (1:1 ratio) was done by a computer program that minimized imbalance between the study arms with respect to sexual identity and prior sexual activity at baseline. Participants, but not researchers, were blind to arm allocation.

Both programs consisted of multiple daily messages over 5 weeks, with a 1-week booster delivered 6 weeks after the 5 week period ended. Guy2Guy intervention content discussed HIV information (e.g., what it is and how to prevent it), motivation (e.g., reasons why adolescent gay and bisexual men may choose to use condoms), and behavioral skills (e.g., how to correctly use a condom). Additional topics also covered the importance of HIV testing, healthy and unhealthy relationships, coming out, and bullying. Although the same concepts were discussed, content was tailored to emphasize different perspectives by sexual experience (e.g., “when you are in a healthy relationship and start having sex…” versus “when you have sex…”). Interactive features included a “Text Buddy”, another program participant with whom the youth was paired and could discuss program content, weekly “level up” questions to assess understanding of the program material, and “G2Genie”, an on-demand tool that provided pre-programmed advice on a variety of topics (e.g., how to break up with a boyfriend).

Control group participants received a text messaging-based program matched on the length of the intervention. Content focused on general health topics (e.g., self-esteem, dealing with bullying) (Table I). Level-up questions, badges, Text Buddy, and G2Genie were not available to the control group.

Table I.

Distribution of youth characteristics of participants in the Guy2Guy RCT (n=273)

	Baseline	3 month post-intervention
	Intervention    Control   (N=133)     (N=140)	Intervention    Control   (N=133)     (N=140)
	Mean (std.    Mean (std.    dev) or %     dev) or %	Mean (std.    Mean (std.    dev) or %     dev) or %
Baseline characteristics
Age	16.02 (1.26)    16.24 (1.43)	--        --
Number of sexual partners in the past 3   months	1.08 (4.53)      0.92 (5.14)	--        --
Sexual experience (had sexual intercourse   at least once)	49.6%       49.3%	--        --
Information
Information-answering more than 75%   correctly (total 13 questions)	48.9%       45.0%	74.4%       66.4%
Behavioral Skills
Behavioral skills-abstinence (high skills)	15.2%       17.9%	21.8%       19.3%
Behavioral skills-condom use (high skills)	42.4%       45.7%	60.9%       55.0%
Motivation
Attitude-abstinence (positive)	40.6%       47.9%	51.9%       47.9%
Attitude-condom use (positive)	69.9%       81.4%	81.2%       80.0%
Subjective norms-abstinence (positive)	16.5%       23.7%	24.8%       21.4%
Subjective norms-condom use (positive)	48.1%       49.6%	51.9%       57.1%
Behavioral intention-abstinence (positive)	25.8%       27.9%	33.3%       23.6%
Behavioral intention-condom use (positive)	53.4%       63.3%	60.9%       62.9%

Measures

Data were collected online at baseline; and 3 months after the 5 weeks of main intervention content was delivered (i.e., approximately 6 weeks after the booster). Baseline survey data were collected between October – November 2014. Three-month follow-up assessments were collected between October 2014 – April 2015. Survey items are shown in Supplementary Table I.

Main outcome measures

Information ^[33]:

A total of 13 true/false questions were asked to test participants’ knowledge about HIV prevention. Answering more than 75% of the 13 questions correctly was coded as 1 (high information) and answering less than 75% of the 13 questions correctly was coded as 0 (low information).

Motivation ^[34,35]:

Six motivation subscales were examined: Attitudes towards abstinence (2 items, e.g., “not having sex until I’m older”; answer categories are 0 “very bad”, 1 “somewhat bad”, 2 “neither good or bad”, 3 “somewhat good”, 4 “very good”; Spearman-Brown reliability test = .25), attitudes towards condom use (4 items, e.g., “buying condoms or getting them for free during the next three months”; answer categories ranged from 0 “very bad” to 4 “very good”; Cronbach’s alpha=.68), subjective norms toward abstinence (4 items, e.g., “Friends that I respect think I should not have sex until I’m older”; answer categories are 0 “Very untrue”, 1 “Somewhat untrue”, 2 “Neither untrue nor true”, 3 “Somewhat true”, and 4 “Very true”; Cronbach’s alpha=.69), subjective norms toward condom use (8 items, e.g., “Friends that I respect think I should buy condoms or get them for free, during the next three months”; answer categories ranged from 0 “Very untrue” to 4 “Very true”; Cronbach’s alpha=.89), behavioral intentions toward abstinence (2 items, e.g., “I’m planning not to have sex until I’m older”; answer categories ranged from 0 “Very untrue” to 4 “Very true”; Spearman-Brown reliability test = .32), and behavioral intentions toward condom use (4 items, e.g., “During the next three months, I’m planning to buy condoms or get them for free.”; answer categories ranged from 0 “Very untrue” to 4 “Very true”; Cronbach’s alpha=.77). Two additional scales, Attitudes towards abstinence and behavioral intentions toward abstinence, each measured with two items, were also examined.

For each scale, item average was computed and dichotomized according to empirical distribution and substantive meaning of the answer categories. For the attitude scales, the item average score was cut at 2.99. As such, 1 represents positive attitude (“somewhat good” or “very good”) and 0 represents negative attitude (“very bad”, “somewhat bad”, or “neither good or bad”). For subjective norms and behavioral intention scales, the item average score was cut at 2.99; that is, 1 represents positive subjective norms or behavioral intentions (“somewhat true” or “very true), and 0 represents negative subjective norms or behavioral intentions (“very untrue”, “somewhat untrue”, or “neither untrue nor true”).

Behavioral skills ^[34,35]:

Behavioral skills for abstinence was measured by 2 items (e.g., “How hard or easy would it be for you to make sure you do not have sex until you’re older”; answer categories provided are 0 “very hard to do”, 1 “hard to do”, 2 “neither hard nor easy to do”, 3 “easy to do”, 4 “very easy to do”; Cronbach’s alpha=.66). Behavioral skills for condom use was measured by 7 items (e.g., “How hard or easy would it be for you to get condoms [buy them or get them free] at a place close to your home or school”; answer categories provided ranged from 0 “very hard to do” to 4 “very easy to do”; Cronbach’s alpha=.81). For both behavioral skills scales, item average was computed and then dichotomized according to empirical distribution and substantive meaning of the answer categories. The scales were cut at 2.99; that is, 1 represents high skills to stay abstinent or high skills to use condoms (“easy to do” or “very easy to do”).

Data analyses

Identifying the analytical sample.

Out of the 302 youth who completed the baseline survey and were randomized, those who: 1) had no missing data on HIV preventive information measured at follow-up; 2) had a non-missing answer to at least one of the 6 HIV preventive motivation questions (i.e., attitude, subjective norms, and behavioral intentions for abstinence and condom use) measured at follow-up; 3) had no missing data on HIV preventive behavioral skills for abstinence nor in condom use measured at follow-up; and 4) had no missing data on attitudes towards condom use measured at baseline were included in the analytical sample (n=273). Those included versus excluded were similar in terms of baseline sexual experience, intervention status, age, and number of sexual partners.

Analyses.

Prior to the main analysis, we conducted preliminary analyses to determine whether subjects assigned to intervention and control conditions differed on baseline IMB measures. Baseline age and number of previous sexual partners were also examined for differences by experimental arm. The only variable that was not balanced was attitudes towards condom use; that is, those assigned into the intervention condition were less likely to hold a positive attitude than those assigned into the control condition (69.9% versus 81.4%). We thus controlled for this variable in all analyses. Additionally, baseline sexual experience was included as a covariate, and also was examined as an effect modifier.

Information, motivation and behavioral skills were each analyzed separately. All analyses were conducted using Mplus version 7.3 ^[36]. Instead of using a scoring scheme to identify motivation subscales, we used latent class analysis, which is a person-centered approach, to identify motivation. Motivation classes were identified by explicitly modeling the inter-relationship between the 6 motivation subscales: attitudes, subjective norms, and behavioral intentions towards both abstinence and toward condom use. Information, behavioral skills for abstinence, and behavioral skills for condom use were examined in a similar way (see Supplementary Figure II). However, as only one outcome was presented, rather than six as with motivation, class membership was fixed rather than freely estimated. Although item endorsement probabilities were fixed, this model nonetheless accounted for measurement error by having class assignment occurring in a probabilistic fashion ^[37]. That is, a latent variable C captured two discrete memberships, i.e., low information and high information, or low behavioral skills and high behavioral skills.

By design, only 1 case in the final sample missed one of the 6 motivation scales. Partial missing data on the latent class analysis indicators were accounted for by using the full information maximum likelihood estimation method ^[38,39].

Latent class analysis was conducted on the 3 month follow-up data. The selection of the class solution was guided by both statistical criteria and substantive meaning of the classes. Results were examined via multinomial logistic regression ^[40] to determine how intervention status and other control covariates were related to these profiles by regressing latent class “C” on covariates. If the intervention affected an IMB construct, we would expect it to be a significant predictor in the model.

Because epidemiologic studies are often underpowered to assess interactions during the data analysis phase, the Type I error rate was increased to p<=.10 as suggested by Selvin ^[41].

Results

Table I shows the distribution of each study variable by experimental arm, for baseline and follow-up separately. Youth were, on average, 16 years of age. By design, about half had prior sexual experience at baseline; and participants had, on average, one sexual partner in the last 3 months. Most outcome measures had a higher value at 3 month follow-up than at baseline, and this was true for both intervention and control groups. That said, some exceptions were noted (e.g., attitude-abstinence and behavioral intention-condom use for the control group).

No participants were harmed in the study.

Information to make HIV preventive decisions

Because HIV information was dichotomized, a 2-class latent variable model was estimated. Intervention status did not have a significant impact on HIV information post-intervention (aOR=1.43; P>.05), nor did sexual experience moderate the intervention effect.

Motivation to make HIV preventive decisions

Statistical criteria, specifically the Bayesian information criterion and Lo-Mendell-Rubin test-likelihood ratio test p-value, indicated that both a 3-class solution and a 4-class solution were candidate models (with the caveat that the Bayesian information criterion for the 4-class solution was slightly lower). We thus compared the two models, and ultimately selected the more parsimonious 3-class model (Log Likelihood=−845.306 (DF=20), Bayesian information criterion=1802.802; see Supplementary Table II), representing three variations in motivation for HIV prevention post-intervention (see Figure I). This model was chosen firstly, because statistical inference indicated a diminishing return pattern of the Bayesian information criterion ^[42] such that the decrease of Bayesian information criterion from a 1-class solution to a 2-class solution (2038.9–1871.6=167.3) and that from a 2-class to a 3-class solution (1871.6–1802.8=68.8) were both much larger than that from a 3-class to a 4-class solution (1802.8–1794.9=7.9). Said another way, the return on the Bayesian information criterion from a 3-class to a 4-class solution was minimal and would have sacrificed parsimony. Secondly, with only 10% of the sample in the additional class, the 4-class solution did not add a substantive, meaningful profile beyond the 3-class solution.

Figure I.

Conditional 3-class model describing post-intervention motivation profiles among participants in the Guy2Guy RCT (n=273)

About one-third (29.4%) of the youth in this sample represented a “High motivation” class, where they had 0.9 probability of having positive attitudes towards abstinence and condom use, a 0.6 probability of having positive subjective norms about abstinence and condom use, a 0.9 probability of perceiving abstinence as easy or very easy, and 0.7 probability of perceiving condom use as easy or very easy.

Two out of five (39.5%) youth were assigned to the “High condom use motivation” class, where they had a high probability of endorsing condom use-related items but a low probability of endorsing abstinence-related items. Slightly less than one-third (31.1%) represented the “Low motivation” class, where there was limited motivation to use condoms and even lower motivation to abstinent.

As shown in Table II, a trend suggested that at 3 months post-intervention, youth in the intervention condition were more likely to be in the High motivation class vs. the Low motivation class, compared to those assigned to the control condition (aOR=2.56, p =.05). As baseline attitudes towards condom use were not balanced between treatment and control conditions, it was included as a control variable in the model.

Table II.

The effect of G2G on HIV preventive motivation profiles at 3 months post- intervention (n=273)

	High motivationb (29.4%)		High condom use motivationc (39.5%)
	aORa	P-value	aORa	P-value
Intervention	2.56	0.05	1.12	0.85
Attitudes toward condom use (Baseline)	1.54	0.29	7.31	<.001
Sexual experience (Baseline)	0.46	0.13	1.50	0.40
Intervention*Sexual experience (Baseline)	0.29	0.11	0.79	0.78

^aAdjusted Odds Ratio

^bYouth in this class have a 0.9 probability of having a positive attitude towards abstinence and condom use, a 0.6 probability of having positive subjective norms about abstinence and condom use, a 0.9 probability of perceiving abstinence as easy or very easy, and 0.7 probability of perceiving condom use as easy or very easy

^cYouth in this class have high probability of endorsing condom use related items but low probability of endorsing abstinence related items

For both models, low motivation class is the reference class

While baseline sexual experience did not have any main effect on motivation class membership, it moderated the effect of the intervention, with a Type I error rate of 0.10 (p=.11) ^[41]. To interpret the interaction effect, we computed the predicted probability by intervention status and sexual experience. The results suggested that the differences were substantially larger for youth who did not have sexual experience compared to those who had sexual experience when they entered the study. Indeed, for those who were initially sexually inexperienced, the predicted probability of being in the High motivation class differed substantially by intervention status (less than 0.3 for the control youths vs. over 0.5 for the intervention youths). We also conducted analyses where class membership was conditioned on intervention status. The sample was stratified by baseline sexual experience and models controlled for attitudes toward condom use at baseline. As presented in Table III, the intervention effect on class membership was only significant for those without sexual experience at baseline: Youth in the intervention group were over 3 times (aOR=3.13; P-value=.04) as likely to be in the High motivation class compared to the control youth.

Table III.

The effect of G2G on HIV preventive motivation profiles at 3 months post-intervention, stratified by baseline sexual experience (n=273)

	No sexual experience at baseline (n=138)				Sexually experienced at baseline (n=135)
	High motivationb (38.7%)		High condom use motivationc (35.5%)		High motivationb (16.1%)		High condom use motivationc (47.6%)
	aORa	P- value	aORa	P- value	aORa	P- value	aORa	P- value
Intervention arm Attitudes towards condom use (Baseline)	3.13 1.80	0.04 0.34	1.34 4.33	0.62 0.07	0.72 1.22	0.56 0.73	1.00 8.40	0.99 <.001

^aAdjusted Odds Ratio. Intervention arm was coded with control arm as the reference group.

^bYouth in this class have a 0.9 probability of having a positive attitude towards abstinence and condom use, a 0.6 probability of having positive subjective norms about abstinence and condom use, a 0.9 probability of perceiving abstinence as easy or very easy, and 0.7 probability of perceiving condom use as easy or very easy

^cYouth in this class have high probability of endorsing condom use related items but low probability of endorsing abstinence related items

For both models, low motivation class is the reference class

Behavioral skills to enact HIV preventive decisions

Because behavioral skills to use condoms and behavioral skills for abstinence were both treated as dichotomous indicators, a 2-class model was estimated for both scales. Intervention status was not significantly associated with behavioral skills for abstinence (aOR=1.15; P>.05), nor did sexual experience moderate the intervention effect. Similarly, intervention status did not predict behavioral skills for condom use post-intervention (aOR=1.45; P>.05), nor did sexual experience moderate the intervention effect.

Discussion

The Guy2Guy intervention is associated with significant differences between the intervention and control groups in terms of motivation to engage in preventive HIV actions. We did not find any evidence for intervention effects on HIV information or behavioral skills however. Of particular note is the significant intervention effect on motivation among sexually inexperienced youth: Guy2Guy is associated with significantly higher motivation to engage in HIV preventive acts for adolescent gay and bisexual men who had never had sex when they entered the intervention. Indeed, by presenting extensive information on HIV risk, anal sex, and condom use prior to sexual debut, we were able to increase inexperienced adolescent gay and bisexual men’s motivation to use condoms early in their sexual trajectory. As prior research has shown ^{[43,44,45,46,47]}, condom use at first sex is one of the strongest predictors of condom use over time. Early intervention with adolescent gay and bisexual men, preferably before sexual debut, is therefore critical if we are to affect the greatest impact on their subsequent sexual risk behaviors.

Information about HIV increased for youth in both intervention and control groups over time: At follow-up, more than 70% of participants correctly answered at least three-quarters of the questions queried. Perhaps HIV information is easy to improve; both arms were exposed to one week of HIV information messaging. It also is possible that there was a ceiling effect, which would have diminished the likelihood of observing differences between the two groups. This ceiling effect around HIV information has been seen in other studies of adolescent gay and bisexual men, and would suggest that the questions posed may have been too easy to answer ^[48]. Future intervention programs should employ better measures of HIV information, incorporating more difficult questions.

The lack of intervention impact on behavioral skills, specifically to engage in condom use and abstinence, is somewhat more difficult to interpret – particularly within the context of other intervention results. In a previously published study of the behavioral outcomes associated with Guy2Guy, the relative odds of self-reported condom use at follow-up are higher for youth in the intervention compared to the control group at intervention end, although power issues introduced a lack of clarity ^[27]. Furthermore, growth curve analyses suggest that the intervention may have a positive impact on abstinent behavior, with similar but non-significant trends for condom use for youth in in the intervention versus control group over time (Ybarra, Liu, Prescott, in progress). While far from conclusive, these promising findings seem to be in contradiction with the behavioral skills findings presented here. Perhaps this is because the IMB-based behavioral skills scales measured aspects of the two behaviors that, while theoretically important, did not reflect intervention components that were emphasized in Guy2Guy. The content for Guy2Guy was purposefully crafted to address key aspects of healthy sexuality ^[49], particularly those that emerged in the focus groups that were conducted in the intervention development phase ^[50]. For example, several messages were written to demystify anal sex because many adolescent gay and bisexual men who had never had sex were worried that it might be painful. Messages encouraged participants to feel empowered to have sex when they wanted to and were ready, and to say no when they did not. This was based upon focus group data that suggested sometimes even youth who had had sex before could identify times when they chose to ‘take a break’ ^[51]. Other youth in the focus groups talked about times when they did not necessarily want to have sex but did not feel like they could say ‘no’ ^[51]. The items that query behavioral skills for abstinence in the IMB scale ask how difficult it may be for youth to wait to have sex until they are older, and so perhaps do not tap into these nuanced issues. That said, it is also possible that the intervention was ineffective at improving behavioral skills and that there was another mechanism that contributed to the noted behavioral change. The null findings here in comparison to the changes noted in motivation may indicate that it is easier to affect one’s motivation than actual behavioral skills.

Similarly, several important aspects of condom use that were emphasized within the Guy2Guy content were not directly assessed by the IMB-based behavioral skills scales. For instance, items measuring condom negotiation were specific to condom use with a boyfriend or girlfriend. The measure did not assess these behaviors with less serious partners (e.g., one-night stands). This approach was taken because most HIV transmissions and condomless sex acts among adolescent gay and bisexual men occur in the context of a serious relationship ^[52,53]. However, this measurement approach might explain why condom use attitudes were comparatively higher than condom use behavioral skills. The attitude questions were much less specific (e.g., “I like having sex with condoms”) and were not quite as closely tied to a specific partner. It is possible that youth in this study had behavioral skills for condom use that were commensurate with attitudes about using them, but that their posited behaviors within a serious partnership were not the same as with a theoretical partner of any seriousness. While focusing participant responses on a tangible partner might have improved the reliability of their answers, it may have also decreased their broad applicability to their overall behavior. Therefore, we suggest that future behavioral skills assessments either drop the reference to “boyfriend or girlfriend” or add in parallel items for partners other than a boyfriend or girlfriend.

Limitations

This study had several limitations. All data were collected via self-report questionnaires, so may be affected by recall and social desirability bias. In order to minimize recall bias, all questions were time-anchored, and participants were requested to provide a memorable event that happened approximately three months prior to the interview to serve as a clear time point. Additionally, surveys were administered via an online portal in order to reduce social desirability bias that might be present with an interviewer-administered assessment.

Because all participants were recruited via Facebook, findings are likely generalizable to the larger population of adolescent gay and bisexual men using this social media site. The sample may not be representative of adolescent gay and bisexual men who use the Internet less frequently or are not ‘out’ on social media.

Conclusion

Guy2Guy was a highly innovative text messaging-based intervention developed for adolescent gay and bisexual men in order to improve their information, motivation, and behavioral skills for HIV prevention. Although we did not see marked changes in information and behavioral skills in the nationally implemented RCT, we believe this to be more indicative of how the measures mirrored and in some cases did not mirror the intervention content, rather than any true lack of change. Additionally, the increase in motivation for HIV prevention, particularly among sexually inexperienced youth within the treatment group, serves as a strong indicator that future HIV prevention interventions for adolescent gay and bisexual men need to be initiated prior to sexual debut to have the greatest impact.