Iteratively Developing an mHealth HIV Prevention Program for Sexual Minority Adolescent Men

Abstract

Five activities were implemented between November 2012 and June 2014 to develop an mHealth HIV prevention program for adolescent gay, bisexual, and queer men (AGBM): (1) focus groups to gather acceptability of the program components; (2) ongoing development of content; (3) Content Advisory Teams to confirm the tone, flow, and understandability of program content; (4) an internal team test to alpha test software functionality; and (5) a beta test to test the protocol and intervention messages. Findings suggest that AGBM preferred positive and friendly content that at the same time, did not try to sound like a peer. They deemed the number of daily text messages (i.e., 8–15 per day) to be acceptable. The Text Buddy component was well received but youth needed concrete direction about appropriate discussion topics. AGBM determined the self-safety assessment also was acceptable. Its feasible implementation in the beta test suggests that AGBM can actively self-determine their potential danger when participating in sexual health programs. Partnering with the target population in intervention development is critical to ensure that a salient final product and feasible protocol are created.

Introduction

Adolescent gay, bisexual, queer, (AGBM) and other men who have sex with men bear a disproportionate HIV incidence burden, representing over 70 % of new HIV infections among youth [1]. Furthermore, AGBM are more likely than other adolescents to have not used a condom at last intercourse [2]. Despite these concerning data, few validated prevention programs are available for AGBM [3, 4], and none are available for adolescents under the age of 16 years. Programs that are developmentally appropriate and compelling for this vulnerable population are urgently needed.

Text messaging may be a particularly efficient way to reach and engage adolescents [5]. Recent reviews suggest optimism for mHealth interventions [6–9]. To capitalize on this medium as a potentially effective way of reaching AGBM, we designed Guy2Guy, a text messaging-based (i.e., “mHealth”) healthy sexuality and HIV prevention program for 14- to 18-year-old AGBM in the United States. The content was crafted based upon the Information-Motivation-Behavioral skills (IMB) Model of HIV prevention [10]. The intervention’s impact on condom use and abstinence was tested in a randomized controlled trial, which concluded in April 2015. Results are forthcoming.

Despite encouraging findings that are emerging in the mHealth literature, a significant paucity of program development experiences have been reported [11–14]. When developing mHealth interventions, attention to message content alone is inadequate. Formative work needs to test the interface and user experience when users are engaged with the technology. A robust literature finds that beta testing can help the development team identify important opportunities to improve the usability of programs before they are released [11, 12, 15–17]. While consensus as to the best methodology is lacking, many agree that having the intended users review mHealth content and usability can facilitate troubleshooting, identify potential technological problems, and reduce user dissatisfaction [15, 18].

To contribute to the burgeoning mHealth and HIV prevention program development literatures, we detail the iterative development process of Guy2Guy. Similar to iterative development methods used in designing Internet-[16, 19] and mHealth-based programming [11, 12], Ybarra and colleagues have described a stepwise approach to developing and testing health behavior change content delivered via text messaging [17, 20]. Likewise, prior to testing Guy2Guy in a randomized controlled trial (RCT), the development phase includes: (1) focus groups (FG) to inform program development; (2) drafting and ongoing development of content; (3) pre-testing content with target users in Content Advisory Teams (CAT); (4) alpha testing internally within the research team; and (5) lastly beta testing among target users.

Sequential Methods and Results

The Chesapeake Institutional Review Board (IRB) and the Northwestern University IRB reviewed and approved protocols. Youth provided informed assent or consent, depending on their age, and a demonstration of decisional capacity [21–24]. A waiver of parental permission was granted to prevent against disclosure of the participants’ sexual identity to their parents, which could potentially place young people at risk for physical or emotional harm [23].

Recruitment advertisements were purchased on Facebook. Incentives were not mentioned in the advertisements or screener to increase the generalizability of the study sample to a “real world” sample of youth who would opt-into the program if it were available publicly. Interested candidates completed an online eligibility screener. Those who appeared eligible were contacted by phone by research staff to confirm eligibility and to obtain verbal assent/consent. Ineligible candidates were emailed HIV prevention resources. The eligibility criteria for all formative research activities matched those of the planned RCT. Participants: (a) reported male sex at birth; (b) were between 14 and 18 years old; (c) self-identified as gay, bisexual, or queer; (d) owned a cell phone and had used text messaging for at least 6 months; (e) planned to maintain their current phone number for the next 6 months; and (f) were enrolled in an unlimited text messaging plan. Male gender identity was added as an inclusion criteria after the FGs.

Focus Groups with Target Population

The FGs aimed to acquire feedback from AGBM about issues related to intervention development (e.g., privacy concerns about receiving HIV prevention messages on a personal cell phone, number and timing of messages). Feedback was also solicited about condom use [25], sexual decision making [26] and the FG experience [25–27], which has been reported elsewhere.

Methods

Four bulletin board-style FGs were conducted, each with a targeted sample size of 18–20 participants per group, over 3 days. One benefit of online FGs is that they can accommodate participants from a wide geographic area because participants can log on when and where it is convenient for them to participate [28]. Participants were thus recruited nationally. In addition to Facebook, advertisements were posted on national LGBT-focused organization websites.

As sexual experience was posited to be the most important difference among youth, two FGs were conducted with sexually inexperienced (i.e., reported never having had vaginal or anal sex) AGBM, and two with sexually experienced (i.e., reported having vaginal or anal sex at least one time in their lifetime) AGBM.

Separate moderator scripts were developed for the sexually inexperienced and sexually experienced FGs and are available online [29]. These scripts were developed based upon questions used in FGs informing the development of an mHealth smoking cessation program [17] and were further iterated upon by the research team. Participants visited the online password-protected FG site two to three times per day and responded to moderators’ questions and comments from other participants. Participants received an incentive of $25 for completing all 3 days of the FGs.

Electronic transcripts of the FG discussions were imported into Dedoose, a mixed-methods analysis software [30]. Qualitative data were analyzed both to increase understanding of sexual health decision making among AGBM, and to develop and revise program content. For example, to examine decision making about condom use, two rounds of constant comparison analysis were applied [25, 31]. Review of transcripts and thematic analyses were used to develop and revise text message content. In the first round of open coding, transcripts were coded based on a priori guiding questions and emerging themes. These themes were then reviewed and refined using techniques aimed at assessing similarities and differences between codes [32].

Results

Eighteen AGBM participated in each of the two sexually inexperienced FGs, and 19 and 20 AGBM respectively, in the two sexually experienced FGs. Participant characteristics are shown in Table 1.

Table 1.

Participant characteristics by development activity

Demographic characteristics	Focus groups (n = 75) % (n)	CAT 1 (n = 17) % (n)	CAT 2 (n = 15) % (n)	Control CAT (n = 4) % (n)	Beta test 1 (n = 18) % (n)
Age n (M:SD)
14–16	n = 46 (15.34:0.77)	n = 9 (15.56:0.73)	n = 5 (14.88:0.84)	n = 2 (15.08:0.00)	n = 13 (15.00:0.71)
17–18	n = 29 (17.44:0.51)	n = 8 (17.37:0.52)	n = 10 (17.20:0.42)	n = 2 (17.58:0.71)	n = 5 (17.20:0.45)
Race
Caucasian	54.67 (41)	64.71 (11)	60.00 (9)	50.00 (2)	66.67 (12)
African American	5.33 (4)	0.0 (0)	13.33 (2)	25.00 (1)	11.11 (2)
Asian	5.33 (4)	5.88 (1)	13.33 (2)	0.00 (0)	0.00 (0)
Mixed racial background	18.67 (14)	23.53 (4)	6.67 (1)	25.00 (1)	11.11 (2)
Native American or Alaskan Native	1.33 (1)	5.88 (1)	6.67 (1)	0.00 (0)	5.56 (1)
Other	14.67 (11)	0.0 (0)	0.0 (0)	0.00 (0)	5.56 (1)
Hispanic ethnicity	25.33 (19)	29.41 (5)	40.00 (6)	25.00 (1)	22.22 (4)
Sexual orientationa
Gay	86.67 (65)	100.00 (17)	80.00 (12)	75.00 (3)	94.44 (17)
Bisexual	18.67 (14)	5.88 (1)	20.00 (3)	25.00 (1)	5.56 (1)
Queer	5.33 (4)	11.76 (2)	13.33 (2)	25.00 (1)	11.11 (2)
Region
North	21.33 (16)	17.65 (3)	20.00 (3)	25.00 (1)	22.22 (4)
South	26.67 (20)	23.53 (4)	33.33 (5)	25.00 (1)	11.11 (2)
Midwest	25.33 (19)	29.41 (5)	20.00 (3)	0.00 (0)	38.89 (7)
West	26.67 (20)	29.41 (5)	26.67 (4)	50.00 (2)	27.78 (5)
Urban–rural residence
Urban	70.67 (53)	58.82 (10)	73.33 (11)	50.00 (2)	83.33 (15)
Rural	29.33 (22)	41.18 (7)	26.67 (4)	50.00 (2)	16.67 (3)
“Out” to parentsb
Yes, to both	42.67 (32)	76.47 (13)	73.33 (11)	100.00 (4)	Not asked
Yes, to my mom	18.67 (14)	17.65 (3)	0.0 (0)	0.0 (0)	Not asked
Yes, to my dad	2.67 (2)	0.0 (0)	0.0 (0)	0.0 (0)	Not asked
Level of “outness” (1 = not out at all, 5 = completely out) (M:SD)b	4.05:1.11	4.18:0.95	4.00:1.00	4.50:0.58	Not asked
Any GBQ friendsb
At school	86.84 (33)	64.71 (11)	80.00 (12)	50.00 (2)	Not asked
In your neighborhood	28.95 (11)	29.41 (5)	66.67 (10)	50.00 (2)	Not asked
School has a GSAb	50.67 (38)	29.41 (5)	66.67 (10)	100.00 (4)	Not asked
Youth accessed GSA at school or town/city	45.33 (34)	11.76 (2)	26.67 (4)	50.00 (2)	Not asked
Ever had sexa
Yes, with a boy	50.67 (38)	47.06 (8)	33.33 (5)	50.00 (2)	50.00 (9)
Yes, with a girl	9.33 (7)	0.0 (0)	13.33 (2)	0.0 (0)	5.56 (1)

GSA Gay Straight Alliance

^aCategories are not mutually exclusive

^bIncluded for the second, but not first, round of FGs

Across the four FGs, participants expressed interest in information about how to meet partners, as well as how to develop and maintain healthy romantic relationships. This was mirrored by an expressed lack of access to educational programs about homosexual sex and sexuality. Participants expressed a preference for messages that were positive and supportive. Both the proposed message intensity (i.e., 5–10 messages per day) and schedule (i.e., outside of school hours on school days and equivalently across the day on weekend days) were acceptable.

The moderators queried the acceptability of RCT participants completing a self-safety assessment: Study staff would talk candidates through various scenarios (e.g., if their parents found out about their participation in G2G) and allow them to determine whether or not they would feel safe to take part in Guy2Guy. FG participants agreed that this would be an appropriate process for their peers to decide their own safety. Participants were also asked about concerns they might have if someone saw program content on their phones and ways these concerns might be mitigated. Many participants reported being out to family, friends, and peers and had little concern about negative consequences that might arise if someone intercepted their text messages. That said, some participants suggested that study staff should help participants password-protect their phone and disable the pop-up text message preview feature, encourage them to delete messages they do not want others to see, and use euphemisms to refer to sensitive topics, such as anal sex.

The moderators also asked about incentive ideas to promote retention. Participants liked a proposed idea for an online store where participants could redeem points earned during the program. Many participants also suggested gift cards, coupons, or discount codes from established online stores (e.g., Target). Participants also supported the idea of receiving free condoms from the program, but some expressed concern about the package being intercepted by their parents or otherwise having to explain the delivery. Participants suggested that the delivery package should be plain with non-descript exterior.

Moderators also presented two program features for acceptability: (a) Text Buddy, which would match two intervention participants to each other so they could send text messages throughout the program, and (b) SOS Tonight, which would send on-demand healthy sexuality advice. Participants well supported the Text Buddy idea; however, some were concerned that it might be “weird” and that conversations could potentially become sexually inappropriate. Participants suggested articulating rules against inappropriate behavior. Participants were less enthusiastic about SOS Tonight, as many felt they would get similar information from their Text Buddy.

Integration of Findings Back into Guy2Guy

During this stage, the program text messages were longer than 160 characters (i.e., the character maximum for a text message), as the aim was to identify appropriate topics, tone, and flow. Quotes from the FGs were used to tailor intervention content. For example, several sexually inexperienced participants expressed their reason for waiting to have sex was wanting to wait until they met the “right guy.” One participant said:

Your body is a temple. Regardless of whether you’re Jewish, Catholic, Buddhist, Hindi, agnostic, or atheistic, your body is a temple. It is the only thing that keeps you alive. Do not defile your own temple. Do not throw away what should be given only to the worthy person(s). Cheesy? Maybe. Too bad.

This was translated into a program message thusly: You have made the choice to wait to have sex. And that’s great! Your body is a temple and you aren’t going to share it with anyone less than the guy who is perfect for you.

Contrary to FG feedback however, we decided to use explicit language in program content where appropriate to avoid potential confusion or misunderstandings about important topics even if it might cause discomfort for some participants. The self-safety assessment and user guides were deemed sufficient to help youth keep embarrassing or dangerous messages away from others’ eyes.

To help regulate Text Buddy messages, the interface was updated to flag or block inappropriate key words. Flagged messages were highlighted for staff attention but were still sent to the buddy. Examples included words that related to drugs, alcohol, and profanity. Messages with blocked words were not sent to the buddy until staff explicitly approved the message. Examples included derogatory slurs. SOS Tonight was reconceived as G2Genie, an on-demand question-and-answer feature that provided sexual health information of interest to AGBM adolescents (e.g., how to know if someone you’re interested in is gay). An Amazon gift card was identified as the incentive for subsequent research activities because of its ease and ubiquitous use.

Content Advisory Team Review

Three sequential CATs were next implemented to determine acceptability and feasibility of each program text message, content flow, and message tone. CAT 1 elicited feedback about the intervention content, then CAT 2 was conducted to confirm whether CAT 1 feedback had been integrated appropriately. Thirdly, a control CAT was conducted to gain feedback about the control group content.

Methods

CAT participants were emailed a Word document that included all of the program text messages. They were asked to provide feedback on each message. To determine the optimal delivery method, half of CAT 1 participants were randomly assigned to receive all messages in one document, and the other half to receive a week’s worth of messages divided over multiple documents and sent over 5 days.

Questions were based upon previously fielded CATs [17] and focused on three areas: (a) Message tone: “Describe the tone of the text messages. Do they make you feel like you’re being lectured or talked down to, or supported?” (b) Message clarity: “Address the structure of the message: Is the message clear? Are you able to read it and understand it easily? Are there changes we could make to make it easier to read?” (c) Message appeal: “Are the messages interesting and informative? Boring or dull? Are there changes we could make to make them more interesting?” Participants were asked to return their feedback within 1 week.

Once feedback was received from each participant, they took part in a moderated, 2-day online discussion to share their experiences with the content and to express additional ideas about improvements. Moderators also asked participants to provide feedback on the ‘gaming’ features of the program, including leveling up (i.e., answering questions correctly to move to the next module) and badges (i.e., earned for achieving sequential goals: [1] getting condoms, [2] carrying condoms, [3] using condoms, [4] getting tested for HIV. Badges 3 and 4 were only asked of sexually experienced participants).

Participant feedback for each text message were collated in an Excel spreadsheet, and findings were integrated into the program text messages. CAT 2 then confirmed whether the resulting changes had appropriately addressed concerns from CAT 1. The online discussion script was iterated further in between the first and second round to further probe issues that had not been well illuminated in CAT 1. Participants received a combined $50 incentive for their time: $30 for their individual feedback and $20 for their participation in the online discussion.

Results

Twenty-four participants each were recruited for CAT 1 and CAT 2. In CAT 1, 17 participants (71 %) completed both the individual text message review and online discussion; in CAT 2, 15 participants (63 %) completed the individual review, 14 of whom also completed the online discussion. Completion rates in both CATs were lower than desired. Those who completed versus did not complete the activities were similar demographically, although trends suggested that those who did not complete participation were more likely to be from a rural community (62.50 %) versus an urban community (37.50 %), X² (1, N = 48) = 3.42, p = 0.064.

CAT 1 Results

CAT 1 completion rates were similar for those who received all the messages in one document versus messages divided by program week sent over 5 days.

Participants expressed a strong preference for “supportive” content. For example, in response to the program message: “Life is wild and wacky. Sometimes one thing leads to another and even if you weren’t planning to have sex, it happens. You got to be ready. For anything. Any time,” a CAT 1 participant said: “Totally right, and the tone is pretty playful here, not stern which is great. The point gets across very clearly.” Although a “lecturing” tone was sometimes seen as necessary, overall, participants expressed a negative opinion of messages that were perceived as “demanding” or “stern.”

Several sexually experienced participants suggested rewording messages that implied AGBM were having sex “all of the time.” Several also expressed embarrassment or surprise at suggestions to practice using condoms. For example, in response to: “There are lots of ways to try: put on a condom when you are masturbating. If you don’t masturbate, try putting a condom on a banana or a cucumber,” a sexually experienced participant said: “OMG this one is so blunt. It just seems really awkward to have to suddenly go and get a banana or cucumber and do that.” Sexually inexperienced participants felt that the beginning of the program was too pro-abstinence. At the same time, some found the messages encouraging them to carry condoms inapplicable because they were not having sex. Participants suggested moving week 4 messages about healthy relationships (i.e., a more comfortable and approachable topic), before week 3 messages about obtaining, carrying, and using condoms (i.e., a more direct and uncomfortable topic for sexually inexperienced participants).

Participants enjoyed messages of testimonials that depicted experiences of AGBM peers, such as: “One guy asked me: ‘If I broke up a month ago, should I get tested immediately? Or wait until after the window period if I know I won’t be having sex?’” The proceeding message explained the window period and advised getting tested immediately if they wanted to, but definitely after the window period as well. Additionally, participants provided suggestions for improving the transitions between topics and reducing repetition. They also found the Text Buddy introduction and game features (i.e., leveling up and earning badges) on Days 1 and 2 to be confusing.

Overall, participants liked the idea of leveling up, although some though it was childish: “Seems to come off just a bit ‘kid-ish’ but I do like the idea.” Regarding badges, a sexually experienced CAT 2 participant shared: “This badge seems fine and seems persistent at getting guys to get condoms, a good reminder over the span of 16 days if they reach the fourth attempt.” Others had more critical feedback: “This badge section really seems a bit pushy to get someone to buy condoms, rather than understanding how they’re used.”

Integrating CAT 1 Findings into the Content

Consistent feedback mentioned by multiple participants was integrated into the content. In some cases, when participants’ opinions about the messages differed, the text messages were rewritten for clarity. More testimonials were added to reinforce program themes. Based on feedback from the sexually inexperienced participants, study staff decided to reorder content flow so that information about sex and using condoms came towards the end of the program.

CAT 2 Results

Given the time required of study staff to separate messages, send multiple documents to participants, and manage the volume of feedback, participants in CAT 2 and control CAT received all messages in one document. Overall, CAT 2 participants were enthusiastic about program content. They felt that naming the program “sender” (e.g., Jamie) made the information easier to internalize and the messages more conversational and engaging. Participants found the tone of the messages to be casual, easy-to-read, and encouraging; and the content flow appropriate.

Integrating CAT 2 Findings into the Content

Only a few of the post-CAT 1 changes made to the content were reconsidered. For example, a CAT 1 participant suggested that more emoticons should be integrated. CAT 2 participants felt, however, that these detracted from the overall messages and should be used only sparingly. Other examples of how feedback from CAT 1 and 2 participants were integrated into program content are shown in Table 2. Messages were further refined. Those that were longer than 160 characters were shortened or split into multiple messages. This resulted in 8–15 messages scheduled per program day.

Table 2.

Example feedback from the Content Advisory Teams

CAT	Program message	P1	P2	P3	P4	P5	P6	P7	P8	P9
CAT 1, sexually inexperienced	There are a lot of good reasons to wait to have anal sex: you can practice other stuff like communication, and also other sex stuff if you want, like hand jobs and blow jobs	Very good	I feel like your giving the guy more opinions. My reaction is i thought that trying different thing are just like sex	“and also other sex activities such as oral sex” but otherwise informative, and clear	Message is on a lecturing/suggesting side, clear and easy to read, not boring. Perfect as is	This message seems abrasive. It just wasn’t smoothly introduced. You are reading through the texts, and BAM you hit this wall that seems uncomfortable. Maybe introducing this text a bit more subtly will help	Clear message that shows other things that you can do that are just as enjoyable and supports your decision in choosing to wait	Tone: This is where it gets a little awkward, at least for me. It works though, and I can’t think of anything else that would work Clarity: Perfect Appeal: Perfect	Whoa. This one was… Blunt, to say the least. Still someone has to be, I suppose. Still, it’s friendly and understandable	Tone: It’s like being talked to from a good friend Clarity: It’s very clear the alternatives for anal Appeal: It’s definitely convincing!
CAT 2, sexually inexperienced	There are a lot of other good reasons to wait: you can practice other stuff like communication. You can also practice other stuff that feels good sexually (like kissing, hand jobs)	Very up-front about information, which I like. The texts don’t stay away from ‘taboo’ subjects	For me, a relationship is build on the communication	The term “other stuff” seems repetitive	Good	This works	This presents them with alternatives that they will feel more comfortable list. It reminds them that actual intercourse isn’t the only way to have sexual experiences and grow closer to their significant other	“Stuff that feels good sexually”. Does this mean anything except “going all the way”?	That’s a good way to stay abstinent
CAT 1, sexually experienced	Or, practice with a partner—see how many times you can put on a new condom before actually doing it—a bit of ‘play’ time, as it were:)	Hm, I am actually going to implement this message!	Make it clear you practice putting condoms on things that aren’t penises…unless that is the recommendation. In that case…why not try putting on condoms then having sex	This message doesn’t really make much sense to me	Good tip Friendly tone	This message seems to have some mixed emotions. First it is helpful in tell you to practice with a partner, but then it includes the phrase “A bit of ‘play’ time” which for most people will come across in a dirty way. I would elaborate more on what you mean by the last phrase in case it isn’t meant to be taken in a dirty way	Sounds a bit strange as an activity, but okay	Gives more tips and has a nice tone	This is good too! We always want to involve our partner
CAT 2, sexually experienced	Or, practice with a partner— see how many times you can put on a new condom as a little foreplay before actually having sex	That actually sounds fun!	Send on the same day as the previous rather than the next day	This one made me laugh. It’s a silly thing to think of but I’m sure it’s a fun way to incorporate a type of foreplay for partners		Needs no improvement	Ehh… not the most financially practical thing. But it would be good practice. Alright message

In anticipation of the beta test, the self-safety assessment was created and included links to several “how-to” guides, including how to password-protect a cell phone. The Text Buddy Code of Conduct was also created to articulate expectations that Buddies would be positive and that the feature is not a dating service [29].

Control CAT Results

The control CAT was conducted to confirm the acceptability of the control group content, which described HIV prevention information publicly available on the Internet, as well as other healthy lifestyle topics such as diet and exercise, substance use, and coping with bullying. Because the aim was a blinded control group, the messages needed to be salient and aligned with what AGBM might expect to learn in an HIV/healthy sexuality program.

Eight participants were recruited for the control CAT. Four (50 %) participants completed the control content text message review. Participants were affirming about all of the program topics and messages and seemed to enjoy especially the messages related to self-esteem and body image. For example, in response to a program message:

Another influence: the media. Often we base our idea of the perfect body off of celebrities and models. We are flooded with images of them everywhere: online, tv, magazines. If feel that you don’t measure up, you’re not the only one who feels this way.

a participant said: “Oh god please. Here comes the unrealistic G.I. Joe and Barbie action figure styles. Please this text is needed in so many ways.”

Integrating Control CAT Findings into the Content

The healthy lifestyle content was deemed acceptable for the control group. Study staff made minor edits and added additional messages about improving self-esteem and having a positive body image into the final control group content.

Internal Team Test

The aim of the internal team test, in which the study staff served as pseudo-participants, was to conduct alpha testing to confirm the enrollment process, ensure the randomization worked properly, interact with the content in the intended delivery mechanism (i.e., as text messages on a phone rather than on a computer screen), and identify any potential bugs in the software.

Methods

Each member of the study staff was randomly assigned to either the intervention or control program. All testers, regardless of study group, were matched with a Text Buddy and had access to G2Genie. Testers were sent the first 5 weeks of their assigned program’s content.

Results

The team was able to register, complete the baseline survey, and be randomized after completing the survey. Testers were also able to utilize G2Genie, Text Buddy, and the “snooze” functionality that pauses messages up to 7 days.

Opportunities to improve the software functionality were identified. For example, one tester experienced an unintended delay in program messages that lasted several days. In another case, testers found a delay in messages to be disruptive: When no response is received to the level-up feature, the program sends reminder messages until a response is given. Content resumes on the day after testers respond—even if a response was given in the morning or early afternoon. Testers found this protocol unnecessarily lengthened the program duration. Many of the error messages were also found to be too vague, making it difficult for testers to identify the causes of the errors. A Text Buddy message that was over 160 characters additionally revealed that the software treated the second half of the message as a G2Genie query, triggering an error message in response.

Integrating Findings into the Program

Technology issues were resolved, and improvements made so that staff could monitor participant progress more closely and identify challenges more quickly (e.g., creating a message log so that staff could easily view messages sent to and received from each participant). The software was also updated to resume program messages the same day that participants respond to level-up prompts. Additional programming allowed the software to treat two messages as a single message if they were received consecutively, as would be the case with some Text Buddy messages.

Beta Test

The last development step was to test the protocol, and program feasibility and acceptability among AGBM youth in a beta test.

Methods

Study staff reviewed the screeners received from interested youth and sequentially contacted those who appeared eligible, while purposefully balancing the sample on race/ethnicity, sexual experience, age, and rural/urban status (e.g., for the white, sexually experienced, rural, 15-year-old “bin”—the first candidate meeting that criteria was contacted, then the second, etc. until that particular bin was filled). Once participants provided verbal assent or consent, they were sent a link to the baseline survey. Participants were randomly assigned to either the intervention or control group, balanced on sexual identity (i.e., gay/queer versus bisexual) and sexual experience (inexperienced versus experienced) as reported in the baseline survey. All study participants received 8–15 text messages a day over 5 weeks. After a 1-week break, participants received a “booster” (i.e., review) week of messages.

To fully test the two program components intended for the intervention arm, both intervention and control participants were matched with a Text Buddy within their study arm and asked to test the G2Genie feature. Buddies were matched based on sexual experience, as well as physical proximity. To ensure messages were sent within similar time zones, Buddies were purposefully assigned so that they lived within one time zone of each other. However, to prevent the possibility that Text Buddies might try to meet in person, pairs lived at least 500 miles apart. Exceptions to the distance requirement were made on a case-by-case basis, but were never closer than 250 miles.

Participants completed a 10-item survey that asked for their feedback at the end of each week. These messages assessed their level of interaction with the program, their opinions of the messages they received, and other feedback. A $30 incentive was provided to participants who completed the program end survey.

Results

Approximately three times as many eligible screeners were received for sexually experienced versus inexperienced youth. Sexually inexperienced youth were also less likely to respond to text messages to set up an enrollment appointment and, after assenting to participate, less likely to complete the baseline survey without extensive follow-up by staff.

Fifty-one candidates were contacted regarding the beta test: two became no longer interested, three were ineligible, and 11 did not respond to contact attempts. Of these non-responders, 36 % were non-White race, 36 % were Hispanic, and 45 % were rural. Thus, urbanicity seemed to be the only differentiating factor. The remaining 15 youth who were contacted but not enrolled were excluded because the bin they represented had already been filled.

Twenty participants assented to take part in the beta test. By design, participants were diverse in terms of race, ethnicity, and urbanicity (Table 1). None of the participants expressed safety concerns during the self-safety assessment. Two youth did not complete the baseline survey and were not randomized: one expressed discomfort with the sexual topics in the survey questions, and the other participant broke his phone. Of the 18 participants who were enrolled in the study, one learned that he was losing his unlimited text messaging plan and withdrew before receiving messages. Two participants withdrew during the field period: one due to personal reasons, and the other reported that he no longer had text messaging capabilities. No participants withdrew because of program content. Fifteen youth completed the beta test.

With few exceptions, program components functioned as intended. That said, participant stop times appeared to be problematic: Several youth chose end times that were later than midnight. This confused how the software program determined which messages to send based upon the day that the participant was in the program. Additionally, the randomization protocol was assigning participants sequentially rather than randomly. This was not immediately apparent because the arms were being correctly balanced on sexual identity and experience.

Response rates (69–87 %) for the weekly survey varied over time, with the highest response rate observed at week 6. Study staff reviewed participants’ comments on the tone, spacing, and content of the text messages in an ongoing manner (Table 3). Most feedback was positive. However, participants recommended clarifying error messages (e.g., specifying which program feature was generating the error) and reducing the number of reminders to respond to the level-up feature to three messages. They also suggested that the weekly survey questions should vary and be specific to the week’s content. Feedback did not differ by participant sexual experience.

Table 3.

Beta test weekly check-in participant feedback

Weekly check-in question	Week 1	Week 2	Week 3	Week 4	Week 5	Week 6
Question 1: what are your thoughts on the number of messages you received per day this past week?	Thought it was good. If fewer messages were sent they would likely need to be longer messages that are sent and he wouldn’t want to read them.—16, Inexperienced Thought it was going to be more messages. It’s a good number though.—15, Experienced	“I think that the number of texts I received each day this week was good. Not too many, not too few; but more would’ve been welcome too.”—15, Inexperienced “The amount of messages was good. There were a lot of messages but they gave me lots of great info and knowledge.”— 14, Inexperienced	“There were a lot more messages but it wasn’t too much in my opinion.”—14, Inexperienced	“There were a lot of messages this week I thought. More than usual. It was kind of hard to keep up with.”—14, Inexperienced	N/A	“There were not nearly enough. Some days it seemed I didn’t get any messages.”— Inexperienced, 16 “It was a good amount.”—15, Inexperienced “Felt like more.”— 15, Experienced
Question 2: what are your thoughts on the spacing of text messages each day this past week?	Some left a bit of a cliff hanger because they were spaced to far apart—15, Inexperienced They come at evenly spaced times so it gives him enough time to think about each one.—14, Inexperienced	“The spacing of the messages were great.”—15, Experienced “The messages could have been a little bit more spaced, giving more time for thought.”—14, Inexperienced	“Better than last week. No more cliff hangers.”— 15, Inexperienced “They were evenly spaced at appropriate times”—15, Experienced	“There were a lot of messages so the spacing was closer. Again, it was a little difficult to stay with.”—14, Inexperienced “Could be spaced better.”— 15, Experienced	N/A	“The spacing was good; good intervals between the messages.”— 15, Inexperienced “Could be better and fix the duplicates.”—15, Experienced
Question 3: what did you think about the overall tone of the messages?	Like how they are written. Not too personal or lecturing. Seems one-on-one. Doesn’t seem like a computer sometimes, seems like it could be an actual person. Likes the wording of message.—15, Experienced They didn’t seem too professional, but also weren’t too like we’re friends. And they weren’t too depressing. When you think about a program about AIDS you think it would be really depressing and negative, but this wasn’t.—16, Inexperienced	“They were friendly and happy- toned.”—14, Experienced “I think that the overall tone of the messages is fine. Most of them aren’t too harsh, although they can be more on a serious topic.”—15, Inexperienced	“The tone was good. It had a lot of scenarios, which I liked.”— 14, Inexperienced “They weren’t too harsh, so overall they were pretty informative.”— 15, Inexperienced	“The tone of the messages was pretty good; not too harsh, but it got the message across.”—15, Inexperienced “Friendly.”—14, Experienced “The tone was very informative, which I liked. It taught me a lot about HIV, safe sex, and testing. I liked learning more.”—14, Inexperienced	N/A	“I felt like they were sent before.”—15, Experienced “Positive. Friendly.”—15, Inexperienced “The tone was good. I liked the informative tone.”—14, Inexperienced
Question 4: what one text message from this week sticks in your mind as particularly helpful or interesting?	Liked the one that said being single isn’t necessarily bad and can help you focus on you and what you want to do.—16, Inexperienced The message about you can get HIV cause of the glands in the anus.—15, Experienced	“Wrap your fingers around the bottom of the condom (at the base of the penis, close to the body) so that it doesn’t come off as you’re (he’s) pulling out.”—14, Experienced “The text messages that stuck in my mind the most from this week were the ones about being with an older guy, like: “No matter how old he is, just b/c he buys you dinner or drives you around in a nice car doesn’t mean he gets to do whatever he wants. Your body, your choice.”—15, Inexperienced	“A few of the text messages that stuck out to me were about communicating with your partner about using condoms.”—15, Inexperienced “[Messages about getting an] HIV test every 3 months.”—15, Experienced	“Several text messages that stuck in my head as helpful were the ABC’s to sex.”—15, Inexperienced “You can tell him what you don’t like with your body too: pulling away, pushing him away, frowning.”—14, Experienced	N/A	“Anything regarding body language.”—14, Experienced “The signs of unhealthy relationships.”— 16, Inexperienced “I liked the text about being assertive.”—14, Inexperienced “Messages regarding obtaining condoms were helpful.”—15, Inexperienced
Question 5: what one text message from this week sticks in your mind as being particularly unhelpful, annoying, or maybe even uncomfortable?	It’s annoying if I forget to include the word buddy, because the program thinks you are trying to text g2genie.—15, Inexperienced No—surprisingly. He thought there would be quite a few messages like that.—15, Experienced	“None were like that.”—15, Experienced “The text about the common sexual fantasy among straight men was maybe a little bit useless.”—14, Inexperienced	“I felt like you were telling me to text my texting partner too much and it got really annoying. Like I get that we’re supposed to text them but you’re suggestions weren’t very god either.”—16, Inexperienced	A response of “none” from all participants who completed the Week 4 checkin	N/A	“All of them. They were a repeat of the week prior to the no message week.”—15, Inexperienced “I thought all of the text messages were great! None were unhelpful.”—14, Inexperienced
Question 6: also, we want to know about how G2Genie is working for you. Were you able to text G2Genie? If so, what did you text?	Yes. Feels like he tried all the topics because he was curious about it.— 16, Inexperienced Yes. Can’t remember exactly what topics he was trying to get info on.—15, Experienced	“It’s very helpful. I texted relationship.”—15, Inexperienced	“Yes, I was able to text g2genie about coming out. It was informative.”—15, Inexperienced	“It’s working well! I texted about community.”—14, Inexperienced	NA	“G1genie worked for me! I texted about relationships and finding a guy.”— 14, Inexperienced “Yes, I was able to text g2genie about condoms.”—15, Inexperienced
Question 7: did you get a response from G2Genie? If so, did it provide you with information that was helpful? That you were expecting?	Yes, got a response. Response was not what he was expecting (the msg said something like I’ve talked to guys and here’s what they said…) but it was helpful.— 15, Inexperienced	“When I used it, the advice was very helpful. But I think the info should be changed sometimes because some people might want new advice.”—14, Inexperienced	“Yes, g2genie provided me with helpful information that I was expecting.”— 15, Inexperienced	“Yes I got a response. The info I got was very helpful. It gave me resources that I can reference in the future.”—14, Inexperienced	“Yes, I was able to text g2genie and get a response.”— 15, Inexperienced	“I did get a response. It gives me the same info every time, which isn’t very helpful. But it did give me inspiration to go on TrevorSpace. I ended up talking to a guy and it was great!”—14, Inexperienced
Question 8: now, more generally: What did you particularly like about G2G this past week?	Like the convenience of being able to access it, really. It was just kinda like a never ending pool of knowledge with information that’s not always easy to find on google.—15, Inexperienced Like how it actually gives options. With his school sexed program it only talks about abstinence which is idiotiac to expect a teenager to do. It’s nice to have all the options (having sex or not) talked about and what each would likely lead to.—15, Experienced	“The text in questions instead of call.” [This reflects the change from phone call to text message weekly check in]—15, Experienced “I love the buddy texting. It’s nice to have someone to talk to when you’re bored, in need of advice, or when you just want to talk to a friend.”—14, Inexperienced	“They gave a little bit of advice on coming out and communication which I think is really important.”—16, Inexperienced “Telling me things that I didn’t know before.”—15, Experienced	“Body language talk”—15, Experienced “The talk about making sex easier.”—15, Inexperienced “I liked the subject of the texts”.—14, Inexperienced	“I really liked all the help for relationships. I feel really prepared for the future.”—15, Experienced “They gave me the tools to get condoms.”—15, Inexperienced	“Attitude, motivation.”—15, Experienced “I liked the interactive messages, even though I wasn’t able to respond to them right away.”—14, Inexperienced
Question 9: from your experience this past week, what is one thing that you think is critical for us to improve?	“This is a hard question. Can’t think of anything bad that happened. Maybe improve the text buddy instruction message.”—16, Inexperienced “Timing of messages—need to be spaced out more.”—15, Experienced	“It’s doing good. I don’t see anything I need to improve yet.”—15, Experienced “More interactive text messages. Kind of like the leveling up but just for fun.”—14, Inexperienced	“Time stamp on buddy messages.”—14, Experienced	“Making sure the spacing is good enough depending on the amount of texts for the certain day.”—14, Inexperienced “Stop sending duplicate.”—15, Experienced	“It was a little too pushy with the whole condoms thing. I felt pressured and uncomfortable.”— 15, Inexperienced “Sometimes it send two texts of the same thing fix that.”—15, Experienced	“Definitely improve the info we get from G2Genie. Make it have more than just one answer for each section. I didn’t like that I got the same exact advice when I asked about finding a guy.”—14, Inexperienced “A way to turn off these messages for a temporary time period, for when someone may be using your phone for a bit, like texting [Stop 5 min].”—15, Experienced
Question 10: is there anything else that we haven’t talked about that you’d like me to know about your experience with the program this past week?	A response of “no” from all participants who completed the Week 1 check-in	A response of “no” from all participants who completed the Week 2 check-in	A response of “no” from all participants who completed the Week 3 check-in	A response of “no” from all participants who completed the Week 4 check-in	N/A	A response of “no” from all participants who completed the Week 5 check-in

Week 1 weekly check-in was conducted over the phone with study staff. All other check-ins were automated and conducted via text messaging

N/A = Not applicable. N/A questions in week 5 were not asked this week because we included an intervention end assessment of recent sexual activity and HIV testing in lieu of these questions

All participants tested the Text Buddy feature to confirm the Text Buddy protocol. The control group Text Buddy functionality was suspended within 2 weeks of beta testing because two Buddy pairs exchanged personal information, despite agreeing to the terms of the Code of Conduct which had explicit instructions against doing so. A review of the conversations suggested this may have occurred because the youth were unsure of what to discuss. Unlike the intervention content, which provided suggestions for conversation topics (e.g., “If you’ve had a boyfriend before, how did you tell him you wanted to wait to have sex? If you haven’t, what would you say? Text your buddy and share ideas”), control participants were not provided with directions.

Between 0 and 359 messages were sent between assigned Text Buddy pairs. Interestingly, sexually experienced participants in the intervention group sent an average of 16.6 messages, while sexually inexperienced intervention participants sent an average of 210.5 messages.

Integrating Findings into the Program

The high program completion rate (83 %) provided optimism that the RCT could be implemented feasibly. The higher message intensity than was proposed in the FGs (i.e., 8–15 messages vs. 5–10 messages per day) was found acceptable, and the Text Buddy matching criteria (i.e., 500 miles apart; within one time zone) also were deemed feasible. Nonetheless, extensive staff time was clearly needed to monitor Text Buddy conversations. Additionally, a more exhaustive list of blocked (e.g., “your number,” “Facebook”) and flagged words (e.g., “get together,” “car”) were identified to better prevent the exchange of personal information.

Based upon user feedback (Table 3), messages for the sexually inexperienced group were further refined to acknowledge that they were not currently having sex and to emphasize the importance of buying and carrying condoms as practice. Reminder messages also became less frequent, such that participants received only two reminders for not responding to level-up questions: one reminder the following morning and another reminder in the evening. If no response was received, the program messages automatically resumed.

The randomization code was corrected. An outer bound for the end time of messages was implemented such that program messages would not be sent after 11:45 p.m. each day. Error messages provided more specific instructions. For example, participants originally received a generic error message if the program did not understand their response: “G2Genie: Sorry, I couldn’t understand what you said. Try rephrasing your message!” This error message was modified to first determine the program component that the participant wanted to use: “Hey! Sorry, I couldn’t understand what you said. Text 1 if you were trying to text your buddy, or 2 if you were trying to text g2genie.” Specific instructions for each program component were then created: “Texting your buddy is easy: Include the word ‘buddy’ anywhere in the text so that the G2G program knows the message is for him” and “Trying to text G2Genie? What topic do you want advice on: sex, condoms, safe decisions, testing, being HIV+, relationships, coming out, feeling down, community.”

Discussion

Refinement of Guy2Guy to ensure its saliency for AGBM was a 2-year, iterative process that revealed important insights about testing a new mHealth intervention. Concrete examples of how to integrate user feedback into the content demonstrate how participatory research designs can ensure the target population has a voice in the program content while still ensuring its adherence with behavior change theory. Important lessons learned include a preference for content that has a positive tone that is friendly and supportive. At the same time, it is important to try to sound neither like a peer nor a professional. Moreover, far from being seen as intrusive or too intense, the volume of messages (i.e., 8–15 messages per day for 5 weeks) was acceptable among the beta test participants who received them. This is particularly important because a minimum level of message saturation is likely needed to affect behavior change. That the control content was just as well received by participants in that arm suggests that text messaging might be used to deliver programs to AGBM about topics other than sex.

The Text Buddy component also was well received by participants but youth clearly needed concrete direction about topics to discuss. Interestingly, the component seemed to be used more intensively by sexually inexperienced compared to sexually experienced youth. Perhaps, youth who are sexually inexperienced have smaller social networks of like-others than sexually experienced youth. It may be that youth who have lower levels of social support or nascent social networks face-to-face will benefit more from having a Text Buddy. This will be examined in the RCT evaluation study. An important minority of youth also required significant monitoring and staff intervention to ensure that they were not deviating into overly sexual or personally identifiable conversations. Indeed, the plan that was developed during the Beta test to ensure appropriate monitoring mandated that at least one staff member was responsible for checking the Buddy messages once every 2 h between 8 a.m. Eastern Standard Time and 8 a.m. Pacific Standard Time (i.e., 11 p.m. Eastern Standard Time) every day when we were in field. Community-based organizations may not necessarily have the staff available to similarly monitor this feature if the intervention was publicly available. Whether the feature is a critical piece of the intervention, or perhaps a component that can be included or excluded depending on an organization’s capability, is something that will be examined in the RCT. This experience highlights the reality however, that automated portions of the intervention are low-cost to implement, while in contrast, the Text Buddy feature required intensive staff time to monitor and facilitate the discussions.

The acceptability of the self-safety assessment among FG participants and the feasibility of its implementation among beta test participants are noteworthy as well. An added benefit of partnering directly with the youth target population—especially those more vulnerable to victimization—is that the self-safety assessment may have even empowered them. It also confirmed that the research team’s trust was appropriately placed in allowing adolescents to make some of their own decisions regarding their personal safety.

Participants were required to have their own cell phone and be enrolled in an unlimited text messaging plan. Samples recruited for each research activity were nonetheless diverse racially, geographically, and by sexual experience. Moreover, non-working phones were not a significant threat to internal validity: Only 15 % of the AGBM youth in the beta test reported phone issues (i.e., broken phone, loss of unlimited text messaging plan) over the 7-week field period. Together, these data suggest that mHealth programs can be targeted to frequent users of text messaging without necessarily resulting in overly privileged samples or high drop-out rates.

It is possible that participants lied about their age or other eligibility criteria. Several steps were taken to reduce this likelihood, however. First, the screener website did not mention the gift card incentive or indicate which fields were the eligibility criteria. Second, to enroll, respondents were required to talk with research staff, who were well familiar with the demeanor of adolescent men based upon their previous work with this population. To prevent against participants taking part in more than one development step, candidates’ cell phone numbers and email addresses were checked with those of previous participants. Although someone could have possibly used a different phone number and email address to enroll a second time, they would have to have known that this was an exclusion criterion. Moreover, the incentive amounts were nominal, reducing the enticement of multiple enrollments by the same person.

Findings should be interpreted within the limitations of the research. Specific program recommendations that emerged from this work with AGBM may not generalize to other populations. Adolescents responding to online recruitment approaches are unlikely to be representative of the larger AGBM population. However, the aim when testing mHealth interventions should not necessarily be to understand how the program works in the general population, but rather in the population most likely to use it—in this case, frequent texters. Furthermore, as noted above, while monetary incentives were used, they were not part of the recruitment advertising, to increase generalizability of the participants. Lastly, confirming that youth actually read the program messages is not possible beyond their self-report.

Conclusion

Text messaging-based programs to affect HIV preventive behavior among AGBM have great potential as a low-cost, scalable approach to intervention delivery. Because face-to-face interaction is not required, those who are less “out” to others can freely participate, as can those in more remote geographical areas without access to traditional in-person prevention programming. As previously noted [11], iterative intervention refinement is time-intensive and can be costly, but also increases the likelihood that the final product is salient to and used by the target population while retaining its adherence to theory. Moreover, the constant iteration of the online interface during the internal team and beta tests was critical to ensuring a usable tool for study staff to monitor participant progress in the RCT. Similarly iterative development is recommended for future mHealth interventions.