Feasibility, Acceptability, and Preliminary Efficacy of an Online HIV Prevention Program for Diverse Young Men who have Sex with Men: The Keep It Up! Intervention

Abstract

Young men who have sex with men (YMSM) are disproportionately infected with HIV/AIDS and there are few prevention programs with published efficacy for this population. This study evaluated the feasibility, acceptability, and preliminary efficacy of an online, interactive, and highly engaging HIV prevention program called Keep It Up! The intervention was designed to be delivered to diverse YMSM upon receiving an HIV negative text result, with the goal for them to “Keep It Up” and stay negative. In a randomized clinical trial, the intervention was compared to an online didactic HIV knowledge condition. The study sample included 102 sexually active YMSM. Participants reported completing online modules in settings that were private and not distracting. Mixed methods data showed intervention participants felt the program was valuable and acceptable. Compared to the control condition, participants in the intervention arm had a 44 % lower rate of unprotected anal sex acts at the 12-week follow-up (p < 0.05).

Introduction

In the U.S., young men who have sex with men (YMSM) are disproportionately infected with HIV/AIDS. Among all U.S. male and female youth ages 13–24 years, YMSM accounted for 53 % of HIV infections in 2006 and this increased to 61 % in 2009 [1]. Disparities are even greater among Black YMSM—by race and risk group Black YMSM were the only population in the U.S. to show a statistically significant increase (48 %) in new HIV infections from 2006 to 2009 [2]. The CDC’s 2008 National HIV Behavioral Surveillance system (NHBS) of MSM in 21 U.S. cities found the HIV prevalence to be 7 % in 18–19 year-olds and 12 % in 20–24 year-olds, and higher among Black young MSM (9 and 20 %, respectively) [3]. These prevalences rival those in the globally hardest hit areas [4]. Given the size of the MSM population and their HIV prevalence, the CDC estimated the rate of new HIV diagnoses as 44 times higher that than of other men [5].

As alarming as these epidemiological data are, there has not been a commensurate prevention response. The vast majority of youth-oriented HIV prevention programs in the published literature are focused on heterosexuals [6]. A recent meta-analysis of HIV behavioral interventions targeting MSM did not report a single RCT where the mean age was <23 [7]. In the CDC’s compendium of evidence-based HIV prevention programs there are seven best evidence interventions for youth, none of which are targeted at YMSM [8]. In fact, of the 69 included programs, only one “promising evidence” intervention focused on YMSM, the community-level Mpowerment Program [9]. This insufficiency of proven interventions led the CDC’s HIV/AIDS Prevention Research Synthesis Team to conclude that there is a significant gap in HIV prevention programs targeting YMSM [10]. In order to address this inequity in HIV prevention services among the highest risk group there is an urgent need to develop scientifically sound prevention programs for YMSM, particularly for youth of color.

HIV prevention programs for YMSM should not be simple modifications of those developed for heterosexual youth, as YMSM are impacted by unique cultural, contextual, and developmental factors [6, 11–14]. In adolescence, MSM are less likely than their heterosexual peers to receive relevant sexuality education in school [15, 16] and therefore may have greater deficits in HIV prevention knowledge. The roles of sociosexual contexts are also dramatically different for YMSM than their heterosexual peers [6, 14, 17]. Similarly, programs designed for adult MSM may not generalize to the developmental context of emerging adulthood, the unique period during which individuals are “too old” to be adolescents (>18 years), but not “old enough” to be adults (~24 years) [18–20]. This represents a period of profound change as varied possibilities with regard to love, work, and worldviews are explored. Psychosocial capacities that are dependent on brain development and which moderate risk taking—such as impulse control and resistance to peer influence—continue to mature through emerging adulthood [21]. These developmental factors are nested within sociocultural and relational factors, like having older partners [22–24]. For example, risk factors, such as alcohol consumption, have been found to have differential effects across developmental stages in MSM [25]. As such, it is critical that interventions targeted at YMSM are contextualized to their experiences and unique risk and protective factors.

While several offline behavioral interventions have proven effective at reducing HIV risk behaviors in adult MSM [10, 26, 27] and heterosexual youth [10, 28–30], the reach of these offline interventions has been restricted by barriers to widespread dissemination and implementation [31–35]. Computer- and internet-based interventions have significant potential for overcoming barriers to cost-effective dissemination and implementation, dramatically increasing potential intervention impact [36, 37]. A recent meta-analysis found significant effects of computer-based programs on HIV risk and protective behaviors that compared favorably to interventions delivered by human facilitators [38].

The Keep It Up! (KIU!)¹ intervention was designed to be an effective online HIV prevention intervention with the potential for reaching large numbers of YMSM of all racial/ethnic groups. It was developed for delivery to YMSM recruited by clinic staff upon receiving a negative HIV test result. The intervention was linked to HIV testing to both increase intervention reach to diverse YMSM and also to build greater HIV prevention into the HIV testing and counseling experience. Data suggest most urban high risk YMSM have recently received HIV testing (77 % in prior 12 months; 58 % in prior 6 months [40]). Furthermore, this face-to-face recruitment approach may increase engagement of racial/ethnic minority YMSM compared to online recruitment. While frequent Internet use is the norm among 18–24 year-olds of all racial/ethnic groups [41], there is evidence that Black YMSM are less likely to seek out HIV information online [42]. With a few exceptions [43, 44], prior internet-based HIV behavioral studies of MSM have substantially under-enrolled racial/ethnic minorities [45–49]. Our experience in conducting research with racial/ethnic minority YMSM suggested that face-to-face recruitment by trusted HIV clinic staff could facilitate enrollment of racial/ethnic minorities, even when the intervention was subsequently delivered online. This hypothesis was supported by analyses of recruitment patterns of a sub-set of participants from the KIU! pilot trial that found no racial/ethnic differences in willingness to participate during face-to-face recruitment by HIV testing staff [50].

The second reason for targeting KIU! to YMSM receiving HIV testing is that further intervention is necessary in this context. Findings from research on the effectiveness of standard voluntary counseling and testing (VCT) in reducing risk behaviors has been mixed, with some support for positive effects among those who test positive or are in discordant heterosexual couples, but more inconclusive results among MSM [51–54]. There is also evidence that quality of prevention counseling is highly variable across settings [53]. This is a missed “teachable moment” [55] for prevention with high risk YMSM who are already convinced that HIV is self-relevant, given their desire to seek testing.

Prior publications have described the formative mixed-methods research that was used to inform the development of the KIU! intervention [42] and racial/ethnic effects on recruitment patterns [50]. The primary aims of this pilot randomized clinical trial (RCT) were to: (1) determine the feasibility of enrolling YMSM into an online HIV prevention program subsequent to obtaining a face-to-face HIV negative test result in a community clinic setting; (2) establish the feasibility of retaining participants in the period of intervention and through a 3 month follow-up; (3) measure the acceptability of the KIU! intervention condition; and (4) obtain a preliminary estimate of the efficacy of the interactive, multimedia KIU! intervention relative to an HIV information-only arm that was also delivered online.

Methods

Procedures

We conducted the RCT between August 2009 and September 2010. As reported in ClinicalTrials.gov, the primary intervention outcome was the count of unprotected sex acts among sexually active participants. Secondary outcomes included: condom errors and problems, HIV knowledge, and attitudes towards HIV risk and prevention. Condom errors and related problems, such as failures, were selected as a secondary outcome based on evidence of high rates among YMSM [56].

Eligibility

Study participants were YMSM who presented for HIV testing and counseling at several community-based organizations in Chicago. Participation eligibility criteria included: (1) being between the ages of 18–24; (2) male birth sex and gender identity; (3) receiving an HIV negative test result from one of the clinics participating in recruitment; (4) had sex with a male in the prior 3 months; (5) had at least one act of unprotected anal sex in the prior 3 months; (6) was not currently in an exclusive/monogamous relationships lasting longer than 12 months; (7) able to read at an 8th grade level and (8) accessed the internet at least several times in the past month.

Recruitment and Retention

Participants were recruited by indigenous clinic HIV testing staff who were trained to provide a form to all males known to be between the ages of 18 and 24 and all males of unknown age. The form described the study, collected demographic information, and asked for contact information from interested individuals. This form served the dual purpose of monitoring demographic differences in enrollment patterns (results already reported [50]) and avoiding recruitment of interested individual who did not meet demographic inclusion criteria (i.e., not male, not age 18–24). Inspection of clinic records indicates excellent compliance in the provision of forms by clinic staff and completion of forms by targeted testing individuals.

If a potential participant was interested and met demographic eligibility requirements based on the clinic-administered form, they were then invited to complete an online questionnaire that established eligibility based on remaining inclusion criteria (e.g., unprotected sex with a male). If eligibility was confirmed in the online screener, participants were invited to participate in the RCT. Participants then completed an informed consent form that indicated that they would be randomized to one of two versions of an online HIV and STI education program. Upon completing the baseline assessment, participants were randomized by a computerized algorithm stratified by race (Black, White, Latino, Other). Participants in both arms completed assessments at baseline, immediate post-test, prior to a 6 week booster session, and at 12 weeks post-intervention completion. In the interest of reducing participant burden, only a sub-set of measures were administered at each assessment time point. Participants were compensated for completing questionnaires in the following amounts: $25 for baseline, $30 for immediate post-test, $35 for 6-week follow-up, and $35 for 12-week follow-up.

In order to assure that participants waited at least 24 h between intervention sessions, an email was sent 24 h subsequent to the completion of a session with a link to the next session. If a session was not completed after 1 week, personalized follow-up emails were sent to the participant. If the participant was unresponsive to these emails, other contact modes (i.e., text messages, phone calls) were used. A similar approach was used to alert and remind participants about completing post-intervention assessments and the booster session. The trial was double blinded so that neither participants nor tracking staff with direct participant contact were informed which arm participants were randomized. Participant confidentially was protected by maintaining participant contact information and tracking data in a password protected database and requiring all staff to sign a confidentiality pledge and to undergo training in data security and human subjects research. Participant privacy was maximized by including a statement on the study login web page reminding participants of the types of content and questions they will be viewing and suggesting they access the material in a private location.

Intervention Description

The KIU! intervention involved 7 modules completed across three sessions that were done at least 24 h apart that in total took ~2 h to complete. The information-motivation-behavioral skills (IMB) model of HIV risk behavior change [57–59] was used to guide the development of intervention content. An innovative aspect of the KIU! intervention was that each module was based on a particular setting or situation relevant to the lives of YMSM, with developmentally appropriate health behavior change content embedded within each of these settings. Module content was designed to be sensitive and responsive to user bandwidth.

Module 1 welcomed participants to the intervention using diverse peer videos discussing connections to family, community, and romantic partners, which set positive peer norms for condom use and obtaining support from families of origin and choice. Module 2 used stylized animation to follow three YMSM chatting online with a focus on identifying triggers for unprotected sex. Embedded content focused on the effects of mood on risk taking [60, 61], negotiating correct condom use and assertive communication, consequences of drug and alcohol abuse on decision making, and information about STI transmission, symptoms, and prevention. Module 3 was a scripted soap opera style video with a racially/ethnically diverse cast of YMSM highlighting the risks in making assumptions about a partner’s HIV status or assuming monogamy. It highlighted the limits of serosorting for HIV negative YMSM when only a minority of HIV positive YMSM know their status [62], the importance of regular testing, skills for negotiating condom use within relationships, and dispelled myths about HIV.

Module 4 was an animated bar/club game built within a virtual reality environment with interactive characters. Through interactive game play, participants identified pros/cons of condom use, steps to correct condom use, consequences of excessive alcohol consumption or drug use, issues with presuming HIV status in others, and effects of sexual arousal on decision making. Module 5 used flash animation to explore the power dynamics between an older and younger man in a dating relationship, and how YMSM can assert healthy behaviors. Embedded in the module was identification of a continuum of safer sex behaviors and strategies for implementing them.

Module 6 was an illustrated story about dating and considered ways to get sexual, emotional, and health needs met in relationships and how ongoing condom use can be an important aspect of that. The module also included a video of a YMSM who receives an HIV diagnosis while in a relationship. Finally, it ended with a video with actors portraying examples of good and bad communication about condom use. In Module 7 participants developed a realistic and practical HIV and STI prevention plan. Suggested goals were tailored to risks reported in participants’ baseline assessment data. The purpose was to plan to prevent previous risky behaviors and to troubleshoot obstacles to successful implementation of the risk reduction goals.

Booster 1 included a chance to revisit goals, receive tailored feedback to troubleshoot obstacles, and set new goals or re-affirm existing ones. Also included was a video follow-up from one of the characters from the Module 3 soap opera who, like the participants, received an HIV negative test result in the recent past and was working to maintain his risk reduction as some of the fear accompanying the test has subsided. Across these modules, the KIU! intervention used diverse delivery methods (e.g., videos, animation, games) to address gaps in HIV knowledge, motivate safer behaviors, teach behavioral skills, and instill self-efficacy for preventive behaviors.

Control Description

An active HIV knowledge control condition was used that contained the same number of modules as the KIU! condition, with the same requirement to participate across three sessions. Using this approach as a control condition helped ensure that both groups had equivalent access to the Internet for HIV-related content, but the total time to complete each control session was not matched to the time for the KIU! arm sessions. The control condition included HIV information that was available at the time on many existing websites; it was didactic, not tailored to user characteristics, non-interactive, and focused exclusively on HIV/STI facts. The modules included general information about condom use, statistics about HIV and STIs among YMSM and general transmission information on STIs including HIV. The control condition was purposely designed to be comparable to existing web-based HIV information so as to understand how the interactive KIU! intervention may improve upon what is already available online.

Measures

General Demographics

Participant characteristics (e.g., age, race/ethnicity, and sexual orientation) were identified using a demographics questionnaire administered at baseline.

Intervention Acceptability

The acceptability and tolerability of the intervention was evaluated using a series of eight items derived from the abbreviated acceptability rating profile developed by Tarnowski and Simonian [63]. Participants used a 5-point Likert scale (1 = strongly disagree to 6 = strongly agree) to rate the extent to which they agreed with each acceptability statement (e.g., “This program was a good way to learn about HIV and STDs and how to prevent them.” and “This program would help others who receive STD or HIV testing.”). The measure was administered at the immediate post-intervention assessment. A mean intervention acceptability score was calculated with higher values indicating greater acceptability (α = 0.85). Open-ended questions were also used to elicit additional feedback about the intervention (e.g., “What aspect of the program did you like least?”).

Adverse Events

The occurrence of adverse events were monitored throughout the study by asking participants at multiple time points whether they had any negative experiences that hurt their health as a result of participating in the study.

Location, Privacy, and Distractions

The extent to which participants were able to complete the measures in a private, quiet setting was assessed during the baseline, post-intervention, and 6-week follow-up assessments. Using a multiple response option, participants indicated their location by answering the following question: “Where are you currently using the computer?” (see Table 2 for options). Using a 4-point Likert scale participants rated how private (1 = not private at all to 4 = completely private) and distracting (1 = not distracting at all to 4 = completely distracting) their current location was (i.e., “How private is the place you are using the computer?”; “How distracting is the place you are using the computer?”).

Table 2.

Characteristics of physical location during survey completion (KIU! arm only)

Outcome	Baseline n = 50a	Post-intervention n = 48a	6-Week follow-up n = 44a
Location of intervention
Home (not in bedroom) (%)	50.0	41.7	52.3
Bedroom (at home) (%)	26.0	43.8	29.5
Library (%)	6.0	0	2.3
Coffee shop (%)	0	4.2	0
Community center (%)	0	0	0
Friend’s house (%)	6.0	6.3	6.8
Other (%)	12.0	4.2	9.1
Location characteristics
Level of privacyb	3.50 (0.79)	3.58 (0.85)	3.48 (0.85)
Distractions presentc	1.29 (0.54)	1.23 (0.59)	1.19 (0.59)

^aMean values are presented with standard deviations in parentheses; all other presented values are percentages

^bBased on the following Likert scale: 1 = not private at all to 4 = completely private

^cBased on the following Likert scale: 1 = not distracting at all to 4 = completely distracting

Sex risk Behaviors

The AIDS-Risk behavior assessment (ARBA [64]) is a structured computerized interview designed to assess HIV/STI risk behaviors among youth populations. The ARBA was administered at baseline and the 12 week follow-up. The measure evaluates general global risk behaviors and also contains partner-by-partner questions that address situational variables associated with recent sexual partnerships. A skip structure is employed to determine the specific questions asked of each participant (i.e., negative responses are not followed by more detailed items). The primary sexual risk outcome used for analysis was the total count of unprotected anal sex acts that occurred 6 weeks prior to the interview date.

HIV Knowledge

Participant knowledge of HIV/AIDS was assessed using a 17-item true–false assessment adapted from Carey and Schroder’s HIV/AIDS Knowledge Questionnaire (HIV-KQ) [65]. The measure was developed to evaluate at-risk populations’ knowledge of sexual behavior transmission and prevention of HIV. The HIV-KQ was administered at baseline and post-intervention to detect changes resulting from the treatment intervention. Items were modified from the original measure to make them relevant for MSM. Correct answers were coded as 1 and incorrect or uncertain responses were coded as 0. Composite scores were calculated to reflect the percentage of correct responses.

Self-Efficacy

The safer-sex self-efficacy questionnaire (SSSE) is a 10-item measure derived from Parsons et al. [66] measure of self-efficacy. The questionnaire was designed to assess participants’ confidence in practicing safer sex behaviors (e.g., correct condom use, discussing safe sex with a partner) and ability to avoid the situational temptation to have unprotected sex (e.g., when condoms are not present, pressure from a partner). The SSSE was administered at baseline and the 6 week follow-up. The internal reliability was low-to-adequate (α = 0.67). Using a 5-point Likert scale (1 = strongly disagree to 5 = strongly agree) participants rated their confidence level of practicing safe sex in various situations. Items were then summed to create a composite score, with higher values indicating greater perceived self-efficacy to practice safe sex.

Decisional Balance

Beliefs about the pros and cons of condom use and unprotected sex were assessed using the 15-item benefits and costs of condom use and unprotected sex questionnaire [66]. The measure assesses participants’ attitudes related to the advantages and disadvantages of condom use (e.g., increased feeling of responsibility vs. lack of spontaneity) and unsafe sex (e.g., greater sexual pleasure vs. risk for HIV/STI). The decisional balance measure was administered at baseline and the 6 week follow-up. Items were modified or removed to ensure their relevancy for MSM. Participants used a 5-point Likert scale (1 = strongly disagree to 5 = strongly agree) to rate their level of agreement with each cost/benefit statement. Mean scores were calculated for the subscales (i.e., pros/cons of condom use and unprotected sex) with higher values indicating greater agreement. Alphas were acceptable for pros of unprotected sex and cons of condom use (0.76 and 0.79 respectively), but were low for cons of unprotected sex and pros of condom use (0.31 and 0.50 respectively).

Condom Errors

An abbreviated version of the Condom Use Errors and Problems Questionnaire [67], previously used with YMSM [56], was used to determine the prevalence of condom errors, failures, and erection problems. A gateway question was used so that only participants who had used condoms in the past 6 weeks were administered the 12-item measure at baseline and again at the 12 week follow-up. Using a 5-point Likert scale (1 = never to 5 = always) participants indicated the degree to which they had experienced a condom error (i.e., using an oil-based lubricant), failure (i.e., breakage during sex), or erection loss (i.e., occurring prior to or during sex). Items were recoded and mean scores were calculated for the subscales (i.e., condom errors, failures, and erection problems) with higher values indicating greater condom problems.

Intentions to Use Condoms

Participants’ intentions to use condoms was assessed using a 5-point response option (1 = I will not use a condom to 5 = I will use a condom every time). The item (“Over the next 6 weeks, how often do you think you will use condoms when you have vaginal or anal sex?”) was administered at baseline and the 12 week follow-up with higher values indicating increased intentions to use condoms. A gateway question was used such that only participants who thought they might have vaginal or anal sex in the next 6 weeks were asked about their intentions to use condoms. The item was adapted from Kalichman et al. [68] and has high face validity.

Data Analyses

Evaluation of intervention acceptability was based on descriptive statistics of the acceptability measure and qualitative analysis of responses to the open ended questions. Intervention feasibility is described in a CONSORT map [69], which illustrates results for enrollment and retention. Intervention efficacy was assessed using an intention-to-treat approach where eligible participants were included in the analysis as originally randomized, regardless of their level of intervention completion [70]. It is of note that gateway questions were frequently employed to avoid asking participants irrelevant questions (e.g., participants who did not report condom use in the past 6 weeks were not asked about condom errors, failures or erection problems). This procedure resulted in differing analytic samples across measures with final samples reflecting the total number of participants who were administered a specific measure at baseline and during the 6 or 12 week follow-up.

Baseline descriptive statistics were calculated to summarize sociodemographic and sexual risk behaviors among YMSM in the two conditions. Differences between conditions were assessed using t tests for continuous variables and Chi square or Fisher’s exact tests for categorical variables. All analyses of intervention effects at follow-up assessments included the baseline measure for the corresponding outcome as a covariate. The general linear model (GLM) was used to compute effects for continuous outcomes, with effect sizes represented as Cohen’s d, calculated as the difference between covariance adjusted means divided by the pooled standard error [71]. Count outcomes (number of sex partners, total unprotected anal sex acts) were analyzed using a negative binomial model fit with a log link function, which provided an estimate of the rate ratio. This model is the recommended approach when analyzing data with overdispersion (i.e., inflated variance relative to mean) [72], which is common in sexual risk behavior data. Participants with missing data were excluded from analyses.

Qualitative responses to the intervention acceptability items were coded based on three main categories: format, content, and take-away. All responses were double coded and reliability was assessed using Cohen’s kappa. After all coding was completed and reliable, lists of excerpts matching a particular theme were generated and exemplar responses were selected by the coders.

Results

Sample Characteristics and Attrition

As shown in Fig. 1, 406 YMSM were tested for HIV at one of the participating clinics and completed the brief demographic assessment and participation interest form. Of these, 321 (79.1 %) were eligible on the clinic screener and completed the more intensive online screener for trial participation eligibility. Based on the online screener, 121 participants were found to be eligible. Of the 200 participants who were excluded based on the eligibility criteria 34 did not complete the screener, 4 reported being older than 24 years of age, 8 reported being transgender or not gay/bisexual, 5 reported being HIV positive, 39 reported not being sexually active, 75 reported only having protected sex in the past 3 months, 9 were in exclusive relationships for more than 12 months, 3 were not able to read at an 8th grade reading level and 1 individual reported using the internet less than once a month. Of invited participants, 102 (84.3 %) consented to participate in the trial, completed a baseline assessment, and were randomized. Engagement in the trial was excellent, with 96.1 % of participants in both arms of the trial completing their respective intervention modules. The rate of completion of follow-up assessments through 3-months post-intervention was high, with 80.3 % of participants in the KIU! intervention condition and 94.2 % of those in the control condition. There was no significant difference in participation rates by condition at either the 6 week (p = 0.16; Fisher’s exact test) or 12 week follow-ups (p = 0.69). Additional attrition analyses indicated no significant differences in demographic characteristics between participants who completed the baseline survey and those who completed the 12 week follow-up.

Fig. 1.

CONSORT map of enrollment into the Keep It Up! randomized clinical trial

The baseline characteristics of the 102 participants who were randomized into the trial are shown in Table 1. Demographic and risk behavior characteristics did not significantly differ between participants randomized to the KIU! intervention or the control group. Participants ranged in age from 18 to 24, with a mean age of slightly older than 21 years in both arms. More than three quarters of participants identified as homosexual or gay, with a minority identifying as bisexual. The sample was racially and ethnically diverse, with approximately three quarters identifying as a minority. Participants had an extensive history of prior HIV testing, with a mean of approximately 5 prior tests during their lifetime (range = 1–32 HIV tests). The sample averaged about one new sexual partner a month, for a mean of >3 partners in the 3 months prior to the baseline assessment. Participants reported frequent internet use with 84.3 % accessing the internet at least once a day.

Table 1.

Baseline comparability of the KIU! intervention and HIV knowledge only control group

Baseline measure	Keep It Up! intervention (n = 50) Valuea	Control group (n = 52) Valuea	χ2/t testb	p
Mean age (years)	21.62 (1.97)	21.04 (1.69)	1.60	0.11
Sexual orientation			1.28	0.26
Gay/homosexual	78.0	86.5
Bisexual/other	22.0	13.5
Race/ethnicity			0.21	0.98
White–Latino	46.0	46.2
White–Non-Latino	24.0	26.9
African American	14.0	11.5
Otherc	16.0	15.4
Education			0.12	0.73
Some high school or graduate	24.0	21.2
Some college or graduate	76.0	78.8
Employed	56.0	73.1	3.26	0.07
Currently a student	46.9	59.6	1.63	0.20
Mean religiosity	1.92 (0.77)	2.20 (0.96)	−1.61	0.11
Mean number of prior HIV tests	5.08 (5.11)	5.19 (6.55)	−0.10	0.92
Mean number of male sex partnersd	3.48 (3.33)	3.31 (3.93)	0.24	0.81

^aMean values are presented with standard deviations in parentheses; all other presented values are percentages

^bStatistics are t test for means and χ² tests for proportions

^cIncludes Native American, Asian, and Mixed-Race

^dFrom the past 3 months

Intervention Process

Table 2 shows the number of participants in the KIU! arm that completed measures in various settings at various time points. The majority of participants were at home when they completed the measures, with a small proportion in public locations like libraries and coffee shops. A small proportion also participated from a friend’s home. Across these locations, participants rated their privacy as moderately to completely private and the location to be “not distracting at all.”

Intervention Acceptability

Regarding the program acceptability, eight questions were asked about the program and a composite variable was created based on those items. The mean acceptability rating was 5.29 (SD = 0.73) for the KIU! condition and 5.31 (SD = 0.67) for the control, which represented strong agreement that both arms of the intervention were valuable and acceptable to participants.

Open ended questions were also asked about what the participants thought of the intervention. For the qualitative coding, the inter-rater reliability was high (κ = 0.86), suggesting good consistency in coding responses between the two coders. For the KIU! condition, comments from participants regarding the format highlighted the interactivity of the modules and booster activities (n = 9) as well as the variety of media used (e.g., videos, games, graphics; n = 18). Regarding intervention content, participants positively commented on the language used (e.g., colloquial language; n = 7) and relevance of the scenarios and examples incorporated in the videos (n = 14). Comments from participants in the control condition focused on the pacing and length of the presentation (n = 12) as well as the readability (n = 13) and STI/HIV information presented (n = 16).

Exemplar responses regarding the KIU! arm include: “I liked that this program provided information to youths in an up-beat manner instead of a boring lecture. I felt very active with the program. I also enjoyed the interviews on the website.”

“I liked that it was split up into three parts. This was useful because you don’t have to dedicate a big chunk of time to one night and because it gives you a chance to absorb the information better. If you made it one long session chances are people would ‘zone out’ and not absorb a lot of it.”

“I thought it was a wonderful program. It made me wish that my high school would have had some form of sexual education program.”

“Videos games. Real life examples. They really helped in sending out the message. Also it kept me thinking about a lot of things regarding my own health, emotional, and sexual needs.”

“I was able to see mistakes that I make in the actions of the characters. I wasn’t completely aware of my behavior until I judged a character’s behavior and then compared the same behavior to my own. As an educational game/tool it is superb in its credibility and fun interactivity. Cool animation and ‘real-world’ mini films. The future of gay sexed.”

Comments about aspects of the intervention or control presentation that participants did not like were more varied, with individuals from the KIU! arm commenting on the length of the intervention (n = 6), technology problems (n = 5), and repetition within the activities (n = 4). Participants in the control commented on the lack of media (n = 4), pacing of the presentation (n = 4) and lack of knowledge gained (n = 5).

Intervention Effects

Table 3 reports the means and standard deviations for the study primary and secondary outcomes at baseline and post-intervention follow-up time points. The sample size is included for each analyses because of changes due to attrition and survey branching (i.e., you cannot report condom errors if no condoms were used). The primary outcome of the study was the count of unprotected anal sex acts, which is at the bottom of the table. As shown, the intervention group had a small decrease from baseline to the 12-week follow-up assessment, whereas the control group showed an increase in the rate of unprotected anal sex. The net effect was a significant KIU! intervention effect of a 44 % lower rate of unprotected anal sex (RR = 0.56, p < 0.05) compared to the HIV knowledge control condition, controlling for baseline rates. Note that this outcome was only measured in those who were sexually active because the intervention focused on consistent condom use rather than reduction in number of partners. As shown in the table, both arms showed a small, non-significant reduction in the number of sex partners. Consistent with this focus, the primary intervention effects were on reduced unprotected sex when sexually active, rather than partner reduction.

Table 3.

Effects of the KIU! intervention compared to the HIV knowledge control condition on primary and secondary outcomes

Outcome	Keep It Up! intervention (n = 50)				Control group (n = 52)
	Baseline		Follow-up		Baseline		Follow-up			Effect sizeb
	Mean	SD	Mean	SD	Mean	SD	Mean	SD	p a		n c
Post-intervention
HIV knowledge	84.44	10.26	90.93	6.97	84.47	11.81	92.24	6.11	0.26	0.12	98
6-Week follow-up
Self-efficacy for safer sex	33.45	3.95	34.89	4.91	34.06	4.10	35.30	3.97	1.00	0.10	94
Benefits of condom use	4.71	0.61	4.34	0.70	4.11	0.65	4.15	0.85	0.28	0.24	94
Costs of condom use	2.77	0.96	2.48	0.89	2.83	0.90	2.68	0.92	0.19	0.16	94
Benefits of unprotected sex	3.66	1.01	3.40	1.01	3.67	0.89	3.63	0.58	0.16	0.23	94
Costs of unprotected sex	4.76	0.45	4.66	0.50	4.71	0.50	4.60	0.60	0.76	0.09	94
12-Week follow-up
Intentions to use condoms	3.85	1.46	4.00	1.44	3.85	1.37	4.03	1.37	0.92	0.02	74
Condom errors	1.88	0.47	1.56	0.58	2.11	0.62	1.80	0.59	0.56	0.19	36
Condom failures	1.50	0.95	1.14	0.48	1.14	0.29	1.25	0.31	0.30	0.22	36
Erection loss	1.86	0.90	1.47	0.70	2.08	1.05	1.61	0.78	0.60	0.14	36
Total sex partners	1.98	2.73	1.15	0.88	1.61	1.02	0.84	0.75	0.32d	1.35e	90
Total unprotected anal sex acts	3.85	5.73	3.70	5.76	3.77	3.88	6.20	12.22	0.04d	0.56e	63

^aAnalysis of covariance results, controlling for baseline levels

^bCohen’s d difference between covariance adjusted means divided by pooled standard error

^cSample sizes differ by outcome at week 6 due to attrition and at week 12 due to the use of gateway questions

^dNegative binomial regression controlling for baseline levels

^eRate ratio

Secondary outcomes of the trial included HIV knowledge, condom use intentions, prevention self-efficacy, and behavioral skills for correct condom use. HIV knowledge was measured immediately post-intervention, which ranged in time from 3 days to 3 weeks from completion of the pre-test assessment depending on how long it took participants to complete the intervention. As shown in Table 3, both arms showed a large increase in HIV knowledge from pre-test to post-intervention (Cohen’s d for KIU! = 0.75; for control = 0.87), which is consistent with both conditions including extensive HIV education. There was no significant difference between arms in the size of change in HIV knowledge.

The 6-week follow-up assessment included measures of self-efficacy and decisional balance about condom use and unprotected sex. The effect sizes were mostly in the direction of better outcomes for those in the KIU! arm, although effects were generally small and not-significant. The 12-week follow-up assessments included secondary outcomes of intentions to use condoms and condom related problems. As shown in Table 3, no effects were observed for intentions to use condoms. Both arms included education about the steps for correct condom use, and both arms generally showed evidence of decrease in condom related problem. Condom errors were reduced in both arms (Cohen’s d for KIU! = −0.61; for control = −0.51), and erection loss when using condoms also decreased in both arms (Cohen’s d for KIU! = −0.49; for control = −0.51). Condom failures declined in the KIU! arm (Cohen’s d = −0.50), whereas they increased in the control arm (Cohen’s d = 0.37). None of the differences between arms were statistically significant, although the effect sizes suggest greater improvements in the KIU! arm.

Discussion

This study assessed the feasibility, acceptability, safety, and pilot efficacy of an online HIV prevention program targeted at YMSM that recently tested HIV negative. The aim of the program was for these young men to “KIU” and maintain their HIV negative status. The intervention was developed in partnership with community-based organizations that provide HIV testing services to the LGBT community and with ongoing engagement of diverse YMSM and was evaluated in the context of a pilot RCT.

Study findings suggest the KIU! intervention can be delivered online safely and with excellent participant engagement. Despite repeated queries throughout the study observation window, no participants indicated that they had a negative experience that harmed their health as a result of participation. The primary concern related to adverse events was loss of privacy, and in this regard participants were successfully able to find a variety of locations to complete the measures that afforded them a high degree of privacy. In most cases this was their own home. Participants also rated these environments as not causing distraction, which would have enhanced their ability to focus on the program content. The majority (96 %) of participants completed the intervention modules, evidencing excellent program engagement.

Attitudes towards the KIU! program were extremely positive based on both quantitative and qualitative responses. Participants self-reported the program helped them change their behaviors to reduce HIV infection risk, that the program is important, would help others like them, and that they were glad they participated. Retention was excellent through the 3 month follow-up visit (>80 % in both arms). This pattern of results further supports Rosser’s [73] contention that it is possible to develop a highly engaging Internet-based HIV prevention program for MSM and that it is possible to conduct an online RCT and retain participants over time. Acceptability was also scored highly in the control condition, which we believe reflects the fact that the HIV education materials in this condition were described in relation to YMSM and there are few such sources of information readily available for this population. As such, even a non-interactive and non-dynamic online education webpage with information specifically targeted at YMSM may be appealing to YMSM.

A variety of intervention outcomes were measured through 3 months from program completion. The primary outcome was the number of unprotected anal sex acts, and relative to control participants, those in the KIU! arm had 44 % fewer risk acts at the 3 month assessment. This significant effect was produced by a small decrease in risk among those in the KIU! arm and an increase in risk among those who were in the control arm. The magnitude of this overall effect exceeds those reported in meta-analyses of past behavioral interventions for MSM [7, 26] and those reported in a large scale RCT of an online HIV prevention program for adult MSM [73]; therefore, we consider this to be a promising intervention effect. Further support for the promise of the KIU! intervention comes from the fact that it produced a large increase in HIV knowledge and reduction in condom errors and failures.

Our study design did not allow us to disentangle what elements of the intervention, based on the Information-Motivation-Behavioral Skills model of HIV behavioral change [57–59], led to the lower rate of unprotected anal sex. To determine this would require a larger factorial experiment. However, we speculate that engagement of diverse YMSM in intervention development led to a product that was both enjoyable and engaging. We believe our use of multiple modalities (e.g., video, animation, games) allowed for individuals with different learning styles to be impacted by the content. The goal setting module allowed participants to set their own goals and receive automated feedback about their success and how to overcome obstacles. Moreover, the use of videos with peers telling stories and role modeling successful risk reduction skills may have instilled a sense that healthy behaviors are normative.

What is to be made of the fact that YMSM in the control arm showed an increase in unprotected anal sex despite learning information about HIV and how it is transmitted? We hypothesize that this increase is consistent with what would be found in a no treatment arm or an observational study following YMSM after a negative HIV test result. Some prior research has found repeat negative HIV test results to be correlated with increased HIV risk behaviors among MSM [74, 75], with the mechanism hypothesized to be perceived invulnerability to HIV enhanced by feelings of “dodging the bullet” after multiple negative results. This pattern of increased risk taking parallels the concept of “risk compensation” that has been a concern with biomedical prevention approaches [59–62]. To be enrolled in the RCT, YMSM had to have engaged in HIV risk behaviors and they reported that on average the HIV test that led them to be enrolled in the trial was their fifth lifetime test. This means that they had a history of HIV risk behaviors and multiple HIV tests telling them those risk behaviors did not lead to infection. Their take away from this experience may be that engagement in transmission risk behaviors does not lead them to become infected. In the KIU! arm, the extensive focus on motivating risk reduction may have been sufficient to overcome this sense of invulnerability. Our study does not allow us determine if this phenomenon is indeed the cause of increased risk in the control group, but given the increasing focus on promotion of testing in high risk groups we believe further research is needed on the behavioral effects of testing HIV negative among YMSM. We also believe that the receipt of an HIV negative test result is a “teachable moment,” a time period where people may be open to reflecting on their behavior and receptive to considering alternatives for the future [55]. The opportunity of MSM reflecting on the risk behaviors that may have led to their HIV test, and experiencing the relief of a negative result, should be capitalized on to prevent future risk taking.

There are other next steps for future research on KIU!. First and foremost, it will be critical to test the efficacy of this intervention in a larger sample and with longer follow-up. Second, it would be valuable to look at effects of the intervention on outcomes that are not self-reported, such as incident STIs. Effects on HIV incidence will be difficult to study because despite high prevalence, annual incidence in this population is such that a very large sample would be needed to have power to detect effects [76]. Third, future studies should explore the efficacy across highly vulnerable sub-groups of YMSM (e.g., Black YMSM). There are dramatic disparities in HIV prevalence by race among YMSM [1] and it has been suggested that interventions tailored to the needs of minority racial groups may be more effective [77]. KIU! was developed in collaboration with diverse YMSM [42] and it will be important to establish how well it functions across racial and ethnic groups. There is also evidence that among YMSM, cognitive and affective processes related to unprotected sex may differ in serious and casual relationships [24], and therefore future studies should test for differential efficacy across types of sexual partners. Further, KIU! was designed with feedback from gay and bisexual identified YMSM, who also comprised the RCT sample. It will be important to test for differential efficacy of the intervention among gay, bisexual, and MSM with other identities (e.g., heterosexual). Finally, with larger samples more sophisticated modeling of intervention effects can be utilized, such as zero-inflated regression models [78], which allow for simultaneous estimation of intervention effects on the count of unprotected sex acts among those who are sexually active and if unprotected sex occurred or not.

Furthermore, we point out that the promise of online interventions lie in their potential for intervention impact [22, 23], defined as the product of efficacy times reach (% of population receiving) [24, 25]. The current arsenal of HIV prevention in the U.S. includes a variety of individual, small group, and community interventions that have proven efficacy for reducing HIV risk behaviors in adult MSM [26–28] and heterosexual youth [28–31] (but not YMSM), but whose reach has been limited by economic and structural barriers to implementation [32–37]. For example, the NHBS found that only 26 % of MSM ages 18-24 report using an HIV prevention service or program in the prior 12 months [79]. And while computer- and Internet-based interventions have significant potential for overcoming barriers to cost-effective implementation [37, 41, 42], the reality is that this potential has not yet been realized by dissemination and implementation [23]. A critically important barrier to dissemination is that fact that most prior online HIV interventions have been built into an IT system that is idiosyncratic to the researcher’s environment, thereby limiting intervention portability, scalability, tailorability, and the ability to update and refresh as new technology emerges [8]. To meet the bar of potential intervention impact, it is critical that online interventions are designed with consideration of reach and future dissemination. And just like with offline interventions, reaching and retaining YMSM can be extremely difficult; it will be important to develop approaches for online interventions that will be effective during implementation outside of the research context. We believe linking to HIV testing is one such approach.

The primary limitation of this study is that participants completed it under highly controlled conditions, including having study staff provide reminder emails and phone calls and providing participant incentives. Therefore it is difficult to generalize to how KIU! would function if implemented outside of a funded clinical trial. Second, all outcomes were measured using self-report, which are prone to recall and social desirability bias. Third, the self-efficacy and decisional balance measures had low internal reliability. Finally, this was a relatively small sample, which was consistent with the aim of conducting a pilot RCT to provide preliminary evidence of efficacy. As such, power was limited to detect the significance of small effect and effect with substantial variability.

This study is the first to show feasibility, acceptability, safety, and preliminary efficacy of an online HIV prevention program for diverse YMSM. The findings of this study support the need for continued effort to develop innovative and engaging online approaches to motivate risk reduction behaviors. Given the high incidence and lack of existing, validated prevention programs for this population, it is a high priority to rapidly establish the effectiveness of KIU! and proceed with dissemination if merited.