Measuring Acceptability and Engagement of the Keep It Up! Internet-based HIV Prevention Randomized Controlled Trial for Young Men who have Sex with Men

Abstract

HIV disproportionately impacts young men of color who have sex with men. Keep It Up! (KIU!) is an online intervention that addresses the needs of this population. The study objective was to examine intervention acceptability and engagement. Outcomes of interests were qualitative and quantitative acceptability and engagement measures, content ratings, and paradata. On average, participants rated content (≥4 out of 5 stars) and acceptability (3.5 out of 4) highly. Compared to White participants, Black participants found KIU! more useful (p=.03), engaging (p<.001), and acceptable (p=.001); Latino participants found KIU! more engaging (p=.03); and “other” non-White participants found KIU! more engaging (p=.008) and acceptable (p=.02). Participants with high school or less education found KIU! more useful, engaging, acceptable, and deserving of five stars than college (p-values = .047, <.001, .002, .01) and graduate degree holders (p-values = .04, .001, <.001, .004). KIU! is a promising prevention tool for highest risk populations.

Introduction

Since 2010, new HIV infections have decreased in each transmission category except that of male-to-male sexual contact, which has remained stable at about 26,000 per year (CDC, 2016b). Men who have sex with men (MSM) continue to account for an estimated 70% of all new HIV infections in the United States (CDC, 2016b). Among MSM, the highest rates of diagnoses are among 25–34 year olds and second highest among 13–24 year olds (CDC, 2016b). Between 2010 and 2014, the number of HIV infections increased by 23% among 25–34 year olds and by 14% among Latino MSM (CDC, 2016b). When looking at new infections at the intersection of age and race/ethnicity, Latino MSM aged 13–24 were the only group to experience an increase in HIV incidence while diagnoses stabilized among Black and White MSM in the same age group (CDC, 2016b).

Despite the burden of HIV among young MSM (YMSM), particularly youth of color, there is a dearth of tailored HIV prevention programs for this population. Our recent analysis of the CDC’s Compendium of Evidence-Based HIV Prevention Programs (CDC, 2017b) found that only two out of 93 programs focused on YMSM (i.e., Mpowerment and Young Men’s Health Project) (Mustanski & Fisher, 2016). Programs designed more broadly for all adult MSM often do not meet the unique prevention needs of YMSM (Mustanski, Newcomb, Du Bois, Garcia, & Grov, 2011). Furthermore, the current arsenal of evidence-based interventions primarily includes face-to-face individual and small-group programs (CDC, 2017b), and their reach has been limited by economic and structural barriers to implementation (Bell et al., 2007; Kelly, Spielberg, & McAuliffe, 2008; Neumann & Sogolow, 2000; Rietmeijer, 2007; Solomon, Card, & Malow, 2006; Swendeman & Rotheram-Borus, 2010). Only 28% of HIV-negative urban YMSM report participating in an HIV prevention program (CDC, 2016a).

eHealth represents an opportunity to deliver effective interventions to large numbers of diverse YMSM. eHealth is most commonly defined as “an emerging field in the intersection of medical informatics, public health and business, referring to health services and information delivered or enhanced through the Internet and related technologies” (Barello et al., 2015; Eysenbach, 2001). eHealth interventions have successfully been used to address a wide range of issues such as mental health, substance use, and health literacy (Estrada et al., 2019; Jacobs, Lou, Ownby, & Caballero, 2016; Stratton et al., 2017). With more than 92% of young adults across all races and income levels going online daily, particularly via smartphones (Lenhart, 2015), eHealth approaches represent a modality for engaging YMSM and delivering intervention content directly to “where they are” while overcoming barriers to access (e.g., geography (Bowen, Horvath, & Williams, 2007; Bowen, Williams, Daniel, & Clayton, 2008)) and circumventing delivery challenges (e.g., fidelity (Mustanski & Fisher, 2016)). Evidence from systematic reviews and meta-analyses indicate that eHealth programs have significant effects on HIV risk and protective behaviors comparable to in-person evidence-based interventions (Bailey et al., 2015; Guse et al., 2012; S. M. Noar, 2011; Seth M. Noar, Pierce, & Black, 2010). The potential for cost-efficient scalability and flexible dissemination is a major boon for eHealth HIV prevention among YMSM (Bailey, Mann, Wayal, Abraham, & Murray, 2015; Bailey et al., 2015; Brown et al., 2013; Gabarron & Wynn, 2016; Guse et al., 2012).

As eHealth interventions are being evaluated it is important to measure how participants engage with the content in order to study what components of the intervention may have led to behavior change or for whom it was most engaging. Literature on measurements of participant engagement in online health interventions is sparse. While online health interventions for HIV and other health conditions continue to grow in popularity, there is a lack of knowledge or agreement on the best ways to measure participant engagement (Baltierra et al., 2016; Strecher et al., 2008). Some measures of engagement use self-report measures collected proximally to the intervention (e.g., star ratings, thumbs up/down) or at the completion of the intervention (e.g., acceptability surveys). Other measures of engagement are drawn from paradata (auxiliary data that capture details about the process of interaction with the eHealth intervention). For example, paradata has been used across multiple studies to estimate duration of time spent on an intervention (Bonett, Connochie, Golinkoff, Horvath, & Bauermeister, 2018; Fredericks, Martorella, & Catallo, 2015; Perski, Blandford, West, & Michie, 2017) and the effect of navigation autonomy on engagement (Crutzen, Cyr, & de Vries, 2012; McClure et al., 2014; McClure et al., 2013). However, there are no standard scales or measures that allow for comparison across online interventions (Baltierra et al., 2016). There is little consensus on how to account for the fact that some users may step away from the intervention while the system continues to count time. There has also been little research on the associations between various metrics of engagement, leaving open questions surrounding time in intervention and its association with acceptability, or surrounding proximal measures of acceptability and their redundancy with survey measures.

Keep It Up! (KIU!) is an online HIV prevention intervention that was created to address the lack of prevention programs for YMSM who are most at risk for HIV infection (Greene, Madkins, Andrews, Dispenza, & Mustanski, 2016; Mustanski, Garofalo, Monahan, Gratzer, & Andrews, 2013; Mustanski et al., 2017b; Mustanski et al., 2018). The current article examines the acceptability and engagement of the KIU! intervention within the context of a randomized controlled trial (RCT) (Mustanski et al., 2017b; Mustanski et al., 2018). There are two primary aims of this paper. The first aim is to explore if there were differences by age, race/ethnicity, and education in acceptability of, and engagement in, the KIU! intervention. The second aim is to advance understanding of how to measure participant engagement in eHealth interventions by examining associations between paradata (time in intervention), proximal measures of acceptability, close ended survey items of acceptability, and qualitative coding of open ended responses.

Methods

Study Design

Data were taken from an RCT of the KIU! intervention. KIU! is an interactive online HIV prevention intervention tailored to ethnically and racially diverse YMSM (Mustanski et al., 2017b; Mustanski et al., 2018). The KIU! intervention was informed by principles of e-learning (Clark & Mayer, 2003), the Information-Motivation-Behavior Skills (IMB) model of HIV risk behavior change (Fisher & Fisher, 2002; Fisher, Fisher, Williams, & Malloy, 1994), and mixed methods research conducted with diverse YMSM (Mustanski, Lyons, & Garcia, 2011).

The intervention includes seven modules that are completed across three sessions at least 24 hours apart. The intervention uses diverse delivery methods such as “man-on-the-street” interviews, scripted videos, animation, and interactive games to provide participants with the knowledge, motivation, behavioral skills, and self-efficacy to prevent HIV and sexually transmitted infections (STI). The use of various content delivery formats allows the intervention to meet the needs of individuals with different educational backgrounds and learning styles (Mustanski et al., 2017a; Mustanski et al., 2018). Each module also focuses on a setting or situation relevant to the lives of YMSM. For example, one module uses the animated storylines of three characters who go online to find “hook-ups” to highlight: (a) the effects of mood on risk behavior, (b) ways to negotiate correct condom use, and (c) the effects of drug and alcohol use on decision-making.

The KIU! 2.0 intervention was integrated into the online Web-based patient reported outcome (PRO) platform, Assessment Center (AC) (Cella et al., 2007; Gershon, Rothrock, Hanrahan, Bass, & Cella, 2010). Assessment Center, like other PRO platforms, is a research management software application that provides a library of PRO instruments, administers surveys and content to participants, and is a central facility for the storage, retrieval, organization, and sharing of study research items and data. Additional information about the intervention content and its effectiveness can be found in previously published manuscripts (Mustanski et al., 2017a; Mustanski et al., 2018).

Recruitment

From May 2013 to December 2015, participants were recruited from a variety of sources that included: (1) HIV testing clinics and mobile testing units of partner community-based organizations (CBOs) in Atlanta, Chicago, and New York; (2) university-based HIV testing at research sites in Atlanta and New York; (3) local health department clinics in Chicago; (4) street outreach in Atlanta, Chicago, and New York; (5) local and national advertising; and (6) research participant registries at the university locations.

Eligibility

Study inclusion criteria included: (1) being between the ages of 18 and 29; (2) assigned male at birth and current male gender identity; (3) receipt of an HIV-negative test result from a study site or remote HIV self-testing; (4) reporting at least one act of condomless anal sex (CAS) with a male partner in the prior six months; (5) not being in a behaviorally monogamous relationship lasting longer than six months; (6) being able to read English at an 8th grade level; and (7) having an email address that could be used for contact and retention purposes.

Eligible participants completed a baseline assessment and self-collected urethral and rectal samples for testing for chlamydia and gonorrhea. STI testing was included in the intervention because the prevalence of STIs, which can increase the risk of HIV transmission, is high among YMSM and can also serve as a biomarker for engaging in HIV risk behaviors (Centers for Disease Control and Prevention, 2018; Pinkerton, Layde, & group, 2002). Upon completion of the baseline assessment and STI testing, participants were enrolled and randomized to the KIU! intervention or the active control arm that contained didactic HIV/STI information. Additional details about the enrollment process can be found in a previously published manuscript (Mustanski et al., 2017a).

Only the baseline assessment, KIU! 2.0 intervention modules, and immediate post-test assessment data collected from the 445 participants randomized to the intervention arm are presented here. Participants were compensated up to $70 for these study activities: $30 for baseline assessment and STI testing, $20 for post-test assessment, and an additional $20 to cover travel costs if baseline activities were completed at a university site or health department clinic. All procedures performed in this study were approved by the Emory University, Hunter College, and Northwestern University Institutional Review Boards, and informed consent obtained from all study participants.

Measures

Demographics.

Participants self-reported their age, race/ethnicity, education, and sexual orientation. Participants’ geographic site was automatically logged by the AC platform during the screening process.

Intervention Acceptability and Tolerability (IAT).

Intervention Acceptability and Tolerability was assessed using an adapted version of the Abbreviated Acceptability Rating Profile (Tarnowski & Simonian, 1992). The measure included open-ended questions (“What did you like about this program?”; “What did you dislike about this program?”) and 15 closed-ended questions with responses on 4-point Likert scales (Cronbach alpha =.87). The questions were adapted from the original measure of eight items (Tarnowski & Simonian, 1992) to be specific to an online HIV intervention for adults. The adaptations were based on the investigators’ experience in the field, as were newly created items such as, “How much did the program draw you in?”

Process measures.

Star ratings.

Following the investigators’ prior eHealth research (Mustanski, Greene, Ryan, & Whitton, 2015), a row of five stars appeared underneath the interactive and video content of each intervention module to assess how much participants liked the content. Participants were required to give a rating on a scale of one to five for each module before they could proceed to the next module. In some cases, participants did not enter a star rating because they either did not finish the module or there was a malfunction in how the page loaded that kept them from being able to record a rating.

Time spent in intervention.

The amount of time that it took participants to complete each module of intervention content was automatically assessed by the AC platform. Unlike some prior work by our team that used a content management system that assessed time on each page (Mustanski et al., 2013; Mustanski et al., 2015), the AC platform recorded an “Instrument Start” date and timestamp when participants began a module. The AC platform did not assess breaks or time away from the keyboard. The timer continued to run until an “Instrument End” date and timestamp was created when participants completed the module. A list of each module and its start and end times was exported from AC.

Statistical analyses

Confirmatory factor analysis using oblimin rotation was conducted to explore the dimensionality of the 15-item IAT scale. The variance accounted for by the solution, the variance accounted for by each individual factor, and the interpretability of the factors were all evaluated to determine the plausibility of the factor structure. Moreover, an interfactor correlation was specified between the latent variables. To further confirm the factor structure, a parallel analysis was used, which compared eigenvalues from the factors in the target data to eigenvalues from randomly generated data. Factors in the target data with eigenvalues greater than the eigenvalues in the randomly generated data were kept.

Adjusted analysis of variance (ANOVA) with Tukey post-hoc tests were used to compare differences in the continuous variables of interest (time to complete intervention, factor scores, IAT scale score) across race and educational attainment. Due to the distribution of the star ratings, demographic comparisons were made using the binary outcomes of the likelihood of endorsing an average of four or greater, and a separate model for five. These analyses employed logistic regression. All models adjusted for race, education, sexual orientation, age, and intervention site. To determine whether race moderated the education/outcome relationship, we assessed race by education interactions and included these interactions where significant.

For the qualitative response data, content analysis (Krippendorff, 2004) was used to identify themes around intervention acceptability. The qualitative coding was organized based on participant responses to two IAT questions assessing intervention acceptability: a) “What did you like about this program?” and b) “What did you dislike about this program?” The three themes that emerged from participant responses were “format,” “content,” and “takeaway.” Two independent raters coded responses to the open-ended questions and reliability was assessed using Cohen’s kappa (ranging from 0.71 to 1.00). After coding was complete, lists of excerpts by theme were generated and exemplar responses were identified by the coders.

To analyze the qualitative codes statistically, a numerical composite score of a participant’s qualitative codes was created. This was done by taking the sum of number of separate “like” codes a participant endorsed, then subtracting the number of “dislike” codes they endorsed. This provided an aggregate of overall favorability of the intervention as higher scores indicated a generally positive view of the intervention, and lower scores indicated a generally negative view of the intervention. Mean composite scores were compared by race and education in the same way as the continuous process variables. All analyses were conducted in R version 3.4 using the Psych package (R Core Team, 2017; Revelle, 2017).

Results

Demographics and basic IAT and process measure scores of the sample (n = 445) are presented in Table 1. The mean age of the sample was 24.33 years. Nearly 63% (280/445) of participants identified as a racial or ethnic minority, 86.5% (385/445) identified as gay, and 84.3% (375/445) reported at least some college. Overall, participants rated the intervention highly, with the majority of participants giving the intervention content four stars or higher (on a 5-star scale) and giving a mean IAT score of 3.4 out of four. A total of 70 participants (15.7%) did not complete the intervention. These participants were excluded from analyses.

Table 1.

Descriptive characteristics of the analytic sample (N = 445)

		Mean (SD)	% (N)
Race
	White	-	37.1 (165)
	Black	-	23.8 (106)
	Latino	-	30.3 (135)
	Other Race	-	8.8 (39)
Education
	High School or Less	-	15.7 (70)
	Some College	-	26.1 (116)
	College Education	-	45.6 (203)
	Graduate Degree	-	12.6 (56)
Age		24.33 (3.00)	-
Sexual Orientation
	Gay/Homosexual	-	86.5 (385)
	Bisexual/Other	-	13.5 (60)
Location
	Chicago	-	39.1 (174)
	Atlanta	-	18.0 (80)
	New York	-	34.2 (152)
	Other location	-	8.8 (39)
Process Measures a
	Total Time (minutes)a	65.73 (22.1)
	Star mean a	4.30 (0.73)
	Star mean 4+ a		70.1 (263)
	Star mean 5 a		24.3 (91)
	IAT Mean Score a	3.41 (0.37)

^aProcess measures among those who completed KIU (N = 375).

Factor Analysis

A four-factor model of intervention acceptability was tested using confirmatory factor analysis. The four selected factors (impact, usefulness, engagement, usability) are based on attributes commonly described in the literature on flow theory and participant engagement (O’Brien & Toms, 2008). One of the IAT items was dropped due to poor factor loading. The four-factor model fit adequately statistically given the size of the sample (χ² [41, N = 445] = 60.54, p = .03) and descriptively (RMSEA = .033, TLI = .98). In addition, a parallel analysis indicated that a four-factor solution best represented the data when eigenvalues from the target data set were compared to eigenvalues from randomly generated data. The eigenvalue for factor one was 4.24 in the target data compared to 0.49 in the random data, 0.91 versus 0.26 for factor two, 0.48 versus 0.20 for factor three, 0.23 versus 0.15 for factor four. A five-factor solution was not considered because the eigenvalue was 0.04 in the target data, compared to 0.11 for factor 5. The four latent acceptability variables were indicated by 14 observed variables: three variables on the Impact factor, four variables on the Usefulness factor, five factors on the Engagement factor, and two variables on the Usability factor.

Table 2 presents the factor loadings for IAT items in the four-factor solution. All standardized factor loadings were generally large for the Impact factor (values ranged from .44-.91), Usefulness factor (values ranged from .48-.90), Engagement factor (values ranged from .31-.83), and Usability factor (values ranged from .57 to .66). The items that loaded onto the Impact factor were the program’s acceptability, its helpfulness in changing behavior, and its importance. The items that loaded onto the Usefulness factor were the program’s likelihood of helping others who receive HIV/STI testing, liking the program, whether it was a good way to learn about HIV/STIs, and its overall helpfulness. The items that loaded onto the Engagement factor were the program’s interactivity, how much it drew the participant in, how much fun it was, how up to date it was, and how comparable it was to other well-liked websites. The items that loaded onto the Usability factor were the program’s convenience and ease of use. The interfactor correlations were also moderate and statistically significant (r ranged from .26 to .65, p-values <.001), indicating that participants were likely to report consistently higher or lower scores across IAT items.

Table 2.

IAT Scale factor loadings.

		Factor Loadings
IAT Item No.	IAT Item	Impact	Usefulness	Engagement	Usability
1	This program was acceptable to me.	0.57	-	-
2	This program has helped me change my behavior to keep myself safe from HIV and other STDs	0.44	-	-	-
3	The issue of HIV and STDs is important enough to have this program.	0.91	-	-	-
4	This program would help others who receive STD or HIV testing.	-	0.48
6	I liked this program.	-	0.60	-	-
7	This program was a good way to learn about HIV and STDs and how to prevent them.	-	0.79	-	-
8	Overall, I found this program helpful.	-	0.90	-	-
12	How interactive did you find the program?	-	-	0.40	-
13	How much did the program draw you in?	-	-	0.70	-
14	How fun did you find the program?	-	-	0.83	-
15	How up to date did you find the program?	-	-	0.31	-
17	How comparable is the program to other websites you like?	-	-	0.38	-
16	How convenient did you find the program?	-	-	-	0.57
18	How easy to use is the program?	-	-	-	0.66
Removed			-	-	-
5	This program was not good for the participants.	-	0.24	-	-

Relationships between Intervention Acceptability and Process Measures

Pearson correlation coefficients among the process measures, intervention acceptability measures, and qualitative composite scores were all highly correlated. Average star rating was positively correlated with time spent in intervention (r = 0.11, p = .03), and all of the IAT factors and the IAT mean score (r from 0.29 to 0.53, p-values < .001). Average star rating was also positively correlated with the Format qualitative theme (r = 0.18, p < .001). The qualitative Format theme composite score was weakly correlated with the Engagement and Usability factors (r = 0.19, p < .001, r = 0.26, p < .001, respectively) and the IAT mean (r = 0.20, p < .001). The qualitative Content theme composite score was correlated with the Usefulness factor (r = 0.17, p = .002), Engagement factor (r = 0.15, p = .006), and IAT mean (r = 0.16, p = .003). The Takeaway theme composite score was only correlated with the Usefulness factor (r = 0.15, p = .004). Time spent in the intervention was correlated with the Engagement factor (r = 0.12, p = .02). The qualitative themes were all inter-correlated (r from −0.10 to 0.27, p-values from < .001 to .04).

Demographic Comparisons of Intervention Acceptability and Process Measures

Measures of intervention acceptability were compared by race and educational attainment (Table 3). Black participants found the intervention more useful (p = .03), more engaging (p < .001), and reported higher mean IAT scores (p = .001) than White participants. Latino participants found the intervention more engaging than White participants (p = .03). Latino participants also regarded the intervention content more highly than other non-White participants (p = .045). This was the only demographic difference in the qualitative theme composite scores. Other non-White participants also found the intervention more engaging than White participants (p = .008), and also reported higher IAT scores (p = .02). Participants who had completed high school or less found the intervention more useful, more engaging, reported higher IAT scores, and were more likely to rate the program with five stars than participants who were college educated (p-values = .047, <.001, .002, .01 respectively) or had graduate degrees (p-values = .04, .001, <.001, .004 respectively).

Table 3.

Unadjusted mean engagement variables by race and education

	RACE				Difference a,b (P-value)	EDUCATION				Differences a,c (P-value)
Process Measures	White	Black	Latino	Other		High School or Less	Some College	College	Graduate Degree
Star Rating 4+	65.6%	76.7%	68.4%	82.8%		86.2%	74.2%	64.4%	64.7%	HvC (0.0496)
Star Rating 5	19.5%	30.2%	23.7%	27.6%		37.9%	27.8%	20.3%	11.8%	HvC (.01), HvG (.004), SvG (.04)
Impact Factor	0.01 (0.70)	0.05 (0.95)	−0.13 (1.21)	0.07 (0.75)	Interaction	0.03 (0.97)	−0.02 (1.16)	0.02 (0.74)	−0.19 (1.00)	Interaction
Usefulness Factor	−0.22 (0.87)	0.26 (0.80)	−0.01 (1.12)	0.16 (0.75)	WvB (.03)	0.36 (0.73)	0.17 (0.93)	−0.18 (0.96)	−0.30 (0.94)	HvC (0.047), HvG (0.04)
Engagement Factor	−0.29 (0.92)	0.34 (0.77)	0.04 (0.93)	0.30 (0.66)	WvB (< .001), WvL (.03), WvO (.008)	0.49 (0.82)	0.13 (0.87)	−0.17 (0.90)	−0.27 (0.87)	HvC (< .001), HvG (.001), SvG (.0494)
Usability Factor	−0.17 (0.88)	0.16 (0.74)	−0.03 (0.74)	0.31 (0.56)	Interaction	0.31 (0.66)	0.07 (0.76)	−0.13 (0.84)	−0.15 (0.79)	Interaction
Factor Sum	−0.67 (2.64)	0.80 (2.33)	−0.13 (3.16)	0.84 (2.24)	Interaction	1.20 (2.38)	0.35 (2.84)	−0.47 (2.73)	−0.90 (2.66)	Interaction
IAT Mean	3.32 (0.36)	3.53 (0.32)	3.40 (0.41)	3.55 (0.28)	WvB (.001), WvO (.02)	3.60 (0.32)	3.47 (0.37)	3.35 (0.37)	3.29 (0.35)	HvC (.002), HvG (< .001), SvG (.02)
Minutes in intervention	64.29 (19.33)	73.05 (26.49)	62.94 (20.66)	62.32 (23.05)	Interaction	64.16 (20.27)	66.63 (20.87)	65.64 (22.72)	66.04 (24.56)	Interaction
Qualitative Measures
Content	0.26 (0.83)	0.19 (0.71)	0.33 (0.67)	0.00 (0.65)	LvO (.045)	0.31 (0.60)	0.23 (0.69)	0.19 (0.79)	0.36 (0.82)
Format	0.00 (0.86)	0.23 (0.80)	0.02 (0.76)	0.23 (0.74)		0.07 (0.57)	0.21 (0.80)	0.02 (0.89)	0.04 (0.76)
Takeaway	0.10 (0.86)	0.16 (0.80)	0.16 (0.76)	0.08 (0.74)		0.21 (0.57)	0.09 (0.80)	0.14 (0.89)	0.07 (0.76)

Note.

^aDifferences report statistical differences after adjusting for education, age, and city.

^bWvB = White versus Black, WvL = White versus Latino, WvO = White versus Other, LvO = Latino versus Other, Interaction = education by race interaction effects explained in text.

^cHvC = High School versus College, HvG = High School versus Graduate Degree, SvG = Some College versus Graduate Degree, Interaction = education by race interaction effects explained in text.

Race by education interactions were significant in the models for the Usability factor, Impact factor, factor sum, and time to complete intervention. Race by education interactions were non-significant in Star Ratings 4+, Star Ratings 5, Usefulness factor, Engagement factor, and mean IAT score. Among Black participants, graduate degree-earning participants spent significantly more time on the intervention than high-school or less educated participants (p = .02). Otherwise, Black participants did not reflect any differences in measures by educational status. Among White participants, however, high school or less participants exhibited greater acceptability than college educated participants (Usability p < .001, factor sum p = .003) and graduate-school educated participants (factor sum p = .005). Additionally, within White participants, graduate degree-earning participants reported lower Impact scores than participants with some college education (p = .03) and trended towards lower scores than participants with a high school diploma or less (p = .051). Latino and “other” race participants did not exhibit any differences by education for any of these acceptability measures.

The only outcome in which age was a significant covariate was in the probability of having an average star rating of five (p = .03). Older participants were more likely to rate the intervention modules with five stars. Measures of intervention acceptability were also compared by geographic site, where the only difference was in the probability of having an average star rating of five. Atlanta participants were more likely to have an average rating of five stars than New York participants (p = .02).

Qualitative IAT Responses

Inter-rater reliability between the two coders was moderate to high across variable domains (kappa = 0.71 to 1.00), suggesting good consistency in coding the responses. For “intervention likes,” Table 4 includes descriptions of the three thematic categories of interest (e.g., format, content, and takeaway), the 19 axial codes that fell within these categories, and the number of excerpts that were coded with each axial code. Across the main thematic categories, participants most frequently referenced the intervention content when describing what they liked about KIU!. The axial codes most often applied were “general information,” “relevance,” and “tone.” The general information code referenced non-specific information and facts delivered in the intervention, such as, “It gave me interesting facts” (Black, 26, High School). Excerpts coded with the relevance code captured descriptions of how realistic and relatable the intervention content was. This code is reflected in the following quote, “The online component… was geared towards gay men and it understood how we operate and how dating works in the contemporary moment. I received examples of situations that were realistic and I got advice on how to protect myself while dating” (White, 27, College). The intervention tone was also frequently discussed by participants with references to the content being non-judgmental, sex-positive, humorous, honest, and straight-forward. Representative quotes include, “It was fun casual and obviously young gay men were involved in writing it. It was realistic and not condescending or out of touch” (White, 26, Grad School) and “I like how things are explicit and get straight to the point. Things seemed real and things weren’t hush-hush (drug use and alcohol use)” (White, 23, College).

Table 4.

Themes and axial codes for intervention acceptability: Keep It Up! 2.0 intervention comments by valence

	Definition	Interv. Likes n (% of main theme)a	Interv. Dislikes n (% of main theme)a
Main Theme: Format		171	162
Media	References to different ways of delivering information, including videos and games; Descriptions of the intervention design	91 (53.2)	70 (43.2)
Interaction	Descriptions of active engagement and interaction with the intervention	54 (31.6)	5 (3.1)
Delivery	Descriptions of delivery modality; References to convenience and flexibility, including references to the Internet and privacy	28 (16.4)	12 (7.4)
Usability	Descriptions of the extent the format is user-friendly, including references to program features and functionality	19 (11.1)	9 (5.6)
Length/Pace	References to length of the modules, time spent completing the intervention, and assessment timeframes	16 (9.4)	42 (25.9)
Personalization	Descriptions of the capacity for the intervention to deliver personalized content	2 (1.2)	2 (1.2)
Repetition	Descriptions of information being repeated throughout the intervention	1 (0.6)	9 (5.6)
Technology	References to the intervention platform, including browser compatibility and audio or sound quality	0 (0.0)	26 (16.1)
Main Theme: Content		177	85
General information	References to non-specific information and facts delivered in the intervention	67 (37.9)	22 (25.9)
Relevance	Descriptions of how realistic and relatable the content is in videos and other scenarios	57 (32.2)	0 (0.0)
Tone	References to the extent that the content is non-judgmental, sex-positive, humorous, honest, and straight-forward	37 (20.9)	21 (24.7)
STI/HIV information	References to STI/HIV, sexual risks, and risk reduction options, including condom use and PrEP	31 (17.5)	13 (15.3)
Difficulty	Descriptions of content difficulty; Reference to content depth and breadth; References to target audience age or sexual health knowledge	5 (2.8)	7 (8.2)
Language	Descriptions of the readability of the text and use of terms that are understandable; References to tailored content for youth	4 (2.3)	1 (1.2)
Actors/Acting	Descriptions of actor attributes and acting quality	2 (1.1)	22 (25.9)
Main Theme: Takeaway		63	7
Knowledge	References to learning, including descriptions of the intervention as informative or helpful	42 (66.7)	5 (71.4)
Benefits to community	Descriptions of the intervention as a source of information and education for others, including intentions to share with friends	10 (15.9)	1 (14.3)
Introspection/ Enlightenment	Descriptions of reflections on individual behaviors and experiences	7 (11.1)	0 (0.0)
Motivation	Descriptions of motivation to engage in behavior change	6 (9.5)	1 (14.3)

^aBolded theme counts (i.e., Format, Content, and Takeaway) represent the number of full quotes that fell into those themes. Axial counts (those numbers below each theme count) represent the number of times the axial code was counted within the theme. This means that the number of axial code counts can be greater than the number of quotes, particularly in cases where multiple axial codes might have applied to a single quote.

Closely following content, participants frequently referenced the intervention format when describing what they liked about KIU!. The axial codes most often applied were “media” and “interaction.” Examples of excerpts categorized under the media code (i.e., references to different ways of delivering information, including videos and games) include the following, “The variety of videos and activities keeping it interesting” (White, 28, College). Another participant liked “the way [KIU!] presented the information, the simplicity of it, and the real life examples” and “the approach that [KIU!] used for informing us through interactive games” (Latino, 22, High School). The interaction code was applied to excerpts describing engagement and interaction with the intervention, such as, “I liked that this program was very interactive. You were required to click on things and drag them places in order to get an answer correct. I feel like this allowed for more enhanced learning and retention of crucial information” (White, 20, Some College).

Intervention takeaway was the least frequently addressed theme in reasons for liking KIU!. Four axial codes emerged within this theme: “knowledge,” “benefits to community,” “introspection/enlightenment,” and “motivation.” Knowledge acquisition was the most common takeaway for participants and this code is represented by the following quote, “It taught me new things common in young MSM populations (i.e., facts on drugs/alcohol STI transmission HIV myths) even though I have a Bachelors in Nursing!” (White, 23, College). Benefits to community and introspection/enlightenment were mentioned less frequently, but suggest that future implementations of the intervention would be beneficial. One participant stated that KIU! “can help men out there who may be afraid of looking for assistance such as places to get tested and just overall being knowledgeable about safer sex practices” (Latino, 24, College) while another said that KIU! provided “plenty to think about in regards to my sexual behavior…lots to consider & change for the future” (White, 28, College).

For “intervention dislikes,” Table 4 includes descriptions of the same thematic categories and axial codes as the “intervention likes,” and the number of excerpts that were coded with each axial code. Across the main thematic categories, participants most frequently referenced the intervention format when describing what they disliked about KIU!. Under the format category, the most disliked intervention features were “media,” “length/pace,” and “technology.” Overwhelmingly, the most commonly cited dislike across all themes was media, as participants often complained about the KIU! video content: “the poor quality sound and dialogue in the videos” (White, 29, College), “some of the videos are too long and pointless” (Asian, 26, Grad School), and “videos were a bit cheesy but relatable” (Latino, 20, College). Length/pace of the intervention was the second most commonly cited dislike of KIU!. The coded excerpts included references to length of the modules, time spent completing the intervention, and assessment timeframes. Representative quotes include the following: “the modules were too long and I don’t think an average person would take this much time to learn about STI/HIV online” (Latino, 29, College) and “would’ve loved to get all 3 sessions done at one time” (Latino, 26, College). The third most discussed dislike was technology (i.e., references to the intervention platform, including browser compatibility, and audio or sound quality). One participant noted that “the interface was pretty buggy/couldn’t complete on phone or tablet” (White, 24, College) and another participant “couldn’t get it to always work on my mac safari” (Latino, 22, College).

Of the numerical composite scores of the qualitative codes, only content differed by race. Latino respondents reported greater acceptability to the intervention content than participants of “other” race. Format trended towards significance in the difference between White and Black (p = .08) with Black participants reporting higher composite scores. There were no significant differences by educational status, however the difference in format between participants who completed “Some College” and “College” trended towards significance (p = .08) with “Some College” participants reporting higher scores.

Discussion

Our exploratory analysis into the best approaches to measure participant engagement in and acceptability of the study revealed mixed results. While some of the qualitative and quantitative measures were correlated (e.g. IAT Usability factor and qualitative Format theme), others were not (e.g. IAT Impact factor and qualitative Takeaway theme). Participants’ star ratings of intervention content were strongly correlated with IAT factors and the mean IAT score. This internal consistency between star ratings and the IAT measure suggests that using both to collect data may be redundant. Instead, one of the two measures can be used to learn how participants regard intervention content. For example, researchers concerned with the length of post-test assessments could embed star ratings within their intervention content instead of using the IAT measure. Star ratings of the intervention were not as strongly correlated with the qualitative themes. This suggests that in the future greater care can be taken when writing the prompts used for open-ended responses, or that it may be useful to continue to collect qualitative data along with star ratings for a fuller understanding of participant opinions on the intervention. The measure of minutes in the intervention was not strongly correlated with any of the acceptability data. Time spent in the intervention was only weakly correlated with star ratings and the IAT Engagement factor. This suggests that more work still needs to be done in creating a time variable that better captures participant engagement in an intervention.

Triangulation of our mixed-methods data reveal a positive impression of the intervention, particularly among populations that are most at-risk for HIV infection. While all participants gave the intervention relatively high acceptability ratings, overall it was found most acceptable by participants who were younger, identified as racial and ethnic minorities, and lived in the South—the demographic groups at greatest risk for HIV (CDC, 2016b). Specifically, we found that Black, Latino, and “other” racial group participants perceived the intervention as useful, engaging, and having good content. We found that participants with less education gave it higher star ratings and also thought it more useful and engaging relative to participants with college or graduate educations. Educational attainment is an important individual characteristic to consider in terms of risk of HIV infection. In a prospective cohort study conducted by Sullivan and colleagues, lower education was a predictor of HIV infection among Black and White MSM in Atlanta (Sullivan et al., 2015). Compared to participants who completed college, participants with some college or only a high school diploma were two to three times more likely to be HIV-positive (Sullivan et al., 2015). A 2013 national study of MSM also found that HIV prevalence was highest amongst men without a college or advanced degree (Wejnert et al., 2013). Considering these previous studies and that KIU! was rated higher among men with lower educational attainment, this intervention may be an effective way to provide HIV prevention messaging to a group that needs it most.

In looking at the significant interaction findings, White participants showed the most consistent associations between education levels and reporting positive attitudes towards the intervention. Compared to participants of color, White participants with greater educational attainment felt the intervention had less utility for them. Although White participants’ ratings were lower in comparison to participants of color, their ratings of the intervention were still high in general. It is also of note that participants did not differ significantly in acceptability across the three different study cities (Chicago, New York, and Atlanta). This speaks towards the ability to implement KIU! across geographic locations, without having to extensively adapt it to the social or cultural idiosyncrasies of a particular town or city, and its population. Minimal geographic adaptation was done by different versions of the Module 1 videos that provide a welcome and peer discussion of the broad definition and importance of community, family, and sexual health.

The qualitative follow-up provided insight into the quantitative main and interaction effects. Participants expressed that the content reflected many of the issues they dealt with on a daily basis with respect to relationships, risk, and HIV/STI testing. For the White group that also reported more education, there was less resonance with the program, which may be a function of that group already being educated about safer sex practices and having access to resources (e.g. HIV/STI testing) that can protect one’s sexual health, or that much of the content features people of color. This conclusion is supported by some participants from this group describing the content as being “cheesy” or “corny” and wanting more nuance and complexity regarding the realities of low, medium, and high HIV risk behaviors.

The survey and open-ended data are very promising and project future success of this program for populations that are most in need of HIV/STI prevention, chiefly racial/ethnic minority groups and those with less education. KIU! was crafted to be representative of all racial/ethnic minorities and special care was given to using a diverse group of actors and situations that actually might arise for MSM of all racial/ethnic backgrounds. The program also counteracted the information deficits that contribute to HIV health disparities by normalizing regular HIV testing, making partner infidelity more conceivable, and educating about biomedical interventions. It is for these reasons that we believe a wider scale implementation of KIU! is appropriate and will be well received by high risk populations across the nation.

Limitations

Our acceptability study of the KIU! intervention was not without its limitations. One limitation was in the way that time spent in the intervention was measured. The AC platform’s recording of module start and stop timestamps was unable to account for time when a participant was away from their keyboard or screen. As such, there were instances of a participant taking over a week to complete a module. The platform was unable to provide any information about whether the participant was actively engaging in the content or logged in during that time. To address this, a future iteration of KIU! will incorporate timestamps of each participant interaction with the platform. This includes all clicks/taps to move to the next page, as well as all interactions with video content. Each of these interactions will be logged with a timestamp and accompanying attributes such as the page that the interaction occurred on and the module the module the page belongs to. Additionally, the new KIU! implementation will include a “time out” feature which will log a participant out of the platform after 15 minutes of idle time. This feature will allow for the removal of away-from-keyboard time in computing time to completion. More broadly, however, completion time of an intervention is not well understood (Donkin et al., 2013). The interplay of individual differences to complete modules, internet speed and lag, avoidable and unavoidable distractions experienced by participants, and environmental factors such as the physical location where viewing the intervention all could have interacted to create noise in the measure of engagement and attention. It was for this reason we measured engagement and attention in other more dynamic ways (i.e., with the IAT scale, star ratings, and free response questions).

Another limitation was the voluntary nature of responding to the open-ended questions. As with most free response data, there was variation in the depth and breadth of response to the questions assessing what the participants liked or did not like. It was impossible to know if participants who did not respond to those questions felt the intervention was positive or negative. It could have been that they were ambivalent to it, did not feel like providing qualitative data, or merely had survey fatigue. Given how enlightening the qualitative data was in detailing the quantitative findings, future iterations of KIU! might require participants to respond to process questions about the intervention.

An additional criticism of the intervention was that it was not easily accessible through smartphones since nearly half of the KIU! intervention content was programmed for Adobe Flash. The current standard for compatibility across devices and web browsers without the need for additional plugins is through HTML5. Considering the fast-paced turnover of devices, especially within the population to which KIU! was geared (Holloway et al., 2017), updating the program to accommodate smartphones, tablets, and other internet-enabled mobile devices is paramount. Therefore, a future implementation of KIU! should be rewritten in HTML5 to play on any internet-ready medium. This might provide even stronger reach within subgroups of MSM who may only have access to one internet-ready device—their phone.

Finally, implementation of KIU! through a community-based organization or non-academic organization would be instrumental to see if its success could exist without a high degree of methodological attention. KIU! was hosted and administered through Northwestern University. Participants were fastidiously managed by a robust staff with plentiful resources necessary to keep participants moving through the intervention. It is necessary to see a more autonomous application of KIU! to truly measure its success and effectiveness as a plug-and-play program to curtail new HIV infections. Our results from this study and previous iterations of the KIU! intervention (Mustanski et al., 2013) support that such a rollout would be successful and well tolerated by MSM within any community. A future direction might be to identify key stakeholders, in any size US city, that are positioned to adopt such an online intervention for their clientele.

Conclusions

An aim of our current research was to assess the acceptability of, and participant engagement in, the KIU! intervention. Our findings are supportive of future implementation of the program across other cities. HIV incidence continues to be most prominent in racial and ethnic minorities in the US (CDC, 2017a), those with less education (World Food Programme, 2006), and those under the age of 30 (CDC, 2016b). Creating an online intervention that can be easily provided to these highest-risk populations, that also has empirical evidence that they will use it, engage with it, and even find it entertaining, is important towards primary HIV prevention. Moreover, the portions of the program negatively rated by these populations (i.e., the evaluation and assessment portions required to prove efficacy) could easily be separated from the base intervention, making it even more approachable and tolerable to participants. Thus, future directions for KIU! should include a trial of the intervention without the heavy battery of behavioral and attitudinal questions.

In the final analysis, KIU! represents the future of primary HIV prevention for MSM. Given it is perceived as being useful, engaging, and impactful among participants, the only remaining barrier is finding community and academic outlets to promote it.