The Usability and Acceptability of an Adolescent mHealth HIV/STI and Drug Abuse Preventive Intervention in Primary Care

Abstract

HIV/STI risk behaviors among adolescents remain significant public health concerns. Shifts in policy and advances in technology provide opportunities for researchers and clinicians to deliver and evaluate m(obile)Health prevention programs in primary care, however, research is limited. This study assessed the usability and acceptability of Storytelling 4 Empowerment —a mHealth HIV/STI and drug abuse preventive intervention app— among adolescents in primary care. Informed by principles of community-based participatory research, we recruited a purposive sample of 30 adolescents from a youth-centered community health care clinic in Southeast Michigan. The study sample is primarily African American and female. Adolescents who participated in the Storytelling 4 Empowerment intervention assessed its usability and acceptability, and self-reported their HIV/STI risk behaviors. We used a multiple-methods approach. Adolescents’ reported high acceptability of the content, process, and format of Storytelling 4 Empowerment, as evidenced by qualitative data and mean scores from the Session Evaluation Form for the HIV/STI and Alcohol/Drug content, overall Storytelling 4 Empowerment intervention, and Client Satisfaction Questionnaire-8. Findings indicate that Storytelling 4 Empowerment is acceptable among adolescents in primary care. A next step is to examine the effect of Storytelling 4 Empowerment on adolescent sexual risk and drug use behaviors and HIV/STI testing.

Introduction

HIV/STI infections remain significant public health concerns.¹ Two prominent adolescent risk behaviors—sexual risk and drug use behaviors^2,3—play a significant role in HIV/STI acquisition.^4,5 Furthermore, although HIV/STI testing is a federal prevention priority,⁶ few adolescents report having ever been tested for HIV/STI. In fact, national surveillance data suggest that only 17.2% of all 12^th grade adolescents report having ever been tested for HIV.² mobile-health (mHealth) interventions —the practice and dissemination of public health through mobile technology⁷— provide opportunities for prevention scientists and clinicians to develop, test and implement interventions in primary care aimed at assessing and targeting multiple HIV/STI adolescent risk behaviors, and increasing HIV/STI testing.^8–11 Assessing the usability and acceptability, including content, format and process, of mHealth interventions in the targeted community is an essential step in working toward efficacious prevention programs, yet research on mHealth HIV/STI interventions for adolescents in primary care settings is scarce.¹² Therefore, the primary purpose of this study was to examine the usability and acceptability of Storytelling 4 Empowerment (S4E), an HIV/STI and drug abuse mHealth preventive intervention app,¹³ in a primary care sample of Southeast Michigan, adolescents.

A face-to-face intervention, Storytelling for Empowerment (SFE) intervention is efficacious in preventing and reducing adolescent HIV/STI risk, including drug use and sexual risk.^14,15 Guided by the empowerment¹⁶ and ecodevelopmental¹⁷ frameworks, SFE aims to increase condom use and drug resistance self-efficacy, HIV/STI and drug use knowledge and communication.

Employing the principles of community-based participatory research¹⁸ and integrating the National Institute on Drug Abuse recommended prevention principles,¹⁹ SFE was adapted into a mHealth app for primary care (hereon referred to as S4E).¹³ Adapting SFE into a mHealth platform was ideal for several reasons. First, SFE has demonstrated efficacy in preventing and reducing sexual risk and drug use in other adolescent populations,^14,15,20 and thereby provides a research-based foundation to develop a mHealth app. Second, the use of culturally specific stories fosters a highly flexible and easily transportable intervention into a brief mHealth app for the targeted community. Third, 73% of youth ages 13–17²¹ and 86% of young adults ages 18–29²² have or have access to a smartphone. Relative to non-Hispanic whites, Black youth are more likely to report smartphone ownership.²¹ Furthermore, 58% of youth ages 13–17 have or have access to a tablet.²¹ Given its widespread use, mobile technology may provide an effective outlet to disseminate research-based programs.²³ Now that the S4E mHealth preventive intervention has been developed, a next important step is to examine the usability and acceptability of S4E among adolescents in primary care¹³ (Fig. 1).

Figure 1.

Screenshots of the S4E preventive intervention app.

Methods

Design

The present study consisted of both quantitative and qualitative data collection and was informed by the principles of community-based participatory research.¹⁸ We collaborated with a Youth Leadership Council (YLC) to inform the development of the S4E mHealth preventive intervention¹³ and present study design. S4E is a theory-driven, culturally specific mHealth app designed to prevent and reduce drug use and condomless sex and increase HIV/STI testing among adolescents.¹³ Guided by empowerment¹⁶ and ecodevelopmental²⁴ theories, S4E aims to improve drug use resistance and condom use self-efficacy and clinician-adolescent HIV/STI risk communication. Data collection occurred between July and September 2015, and was approved by the University of Michigan’s Institutional Review Board.

Participants

We recruited a purposive sample of 30 adolescents in the clinic waiting room of a youth-centered community health care clinic located in Southeast Michigan. To be included in this study, male or female adolescents had to: (1) be between 13 to 22 years old, (2) live in Southeast Michigan, (3) provide consent, or assent and parental consent (for participants aged 13 to 17 years), and (4) have no prior psychiatric hospitalization. The study sample was primarily female (n=21, 70%) with a mean age of 18 years (SD: 2, Range: 13 to 22 years). Participants self-identified as Black/African American (n=12, 40%), followed by non-Hispanic white (n=9, 30%), Hispanic (n=4, 13.3%), more than one race (n=3, 10%), and Other (n=2, 6.7%).

Procedures

Adolescents completed a demographics questionnaire and HIV/STI risk behaviors and testing assessment on iPads using Research electronic data capture (REDCap).²⁵ Adolescents then participated in the S4E intervention via iPads in a reserved room located in the clinic. Using both quantitative and qualitative methods, we assessed participants’ usability and acceptability after completing each module of the S4E intervention, as well as the overall user experience and user interface of the S4E intervention. Therefore, immediately following the adolescents’ participation in each S4E module, youth participated in either a focus group (n=8) or individual (n=10) interview to provide in-depth qualitative data on their experiences with regard to the usability and acceptability of S4E. Focus groups ranged in size from 2–4 participants. Although focus groups have been shown to be empowering for disenfranchised populations because they provide a space for shared experience, not all youth may feel comfortable with discussing sensitive topics, including sexual risk and drug use behaviors, in a group format. Therefore, we also used individual interviews in our approach.¹³ We provided adolescents with a $20 incentive to help defray the costs of participating in the study.

Qualitative Measure

The interview guide consisted of open-ended grand tour questions, including, “What did you think about the HIV/STI module as a whole (or in general)?”, “What aspects of the HIV/STI module did you like the most or found the most useful and why?”, “What aspects of the HIV/STI module did you find the least useful or liked the least and why?”, and “What things would you change to improve the HIV/STI module?” We asked the same question for the Risk Assessment, Alcohol and Drugs module, as well as the overall S4E intervention for both individual and focus group participants. Trained research assistants conducted the interviews which ranged in total time from 33 to 97 minutes.

Qualitative data analysis

The audio recordings of each interview were transcribed verbatim and reviewed for accuracy. We transferred the data to Nvivo10 for storage, organization, and data analysis.²⁶ Data analysis followed the tenets of content analysis, systematically condensing the data into related sub-categories, categories and overarching themes.²⁷ First, three members of the research team, including the lead author, read each transcript to familiarize themselves with the data. Second, the team conducted open coding with respect to the overarching research question (i.e., Is the S4E intervention usable and acceptable to adolescents?). Third, the codes were collated into emerging sub-categories. Fourth, the team merged related sub-categories to major categories and themes. We established trustworthiness of data through credibility, dependability, and transferability.²⁸

Quantitative Measures

Demographics.

Participants completed a demographics questionnaire, including age, gender, race and ethnicity.

HIV/STI risk behaviors.

We assessed adolescents’ lifetime and past 90-days substance use and sexual risk behaviors using items extracted from the Monitoring the Future study³ and the Youth Risk Behavior Surveillance survey.² An example question is, “In the past 90-days, have you had oral, vaginal and/or anal sexual intercourse without a condom?” (0=no, 1=yes).

HIV/STI testing.

Adolescents were asked whether they have been tested for HIV and STIs in their lifetime (0=no, 1=yes).

STI.

We assessed adolescents’ lifetime and past 90-days STIs. An example statement is, “Has a doctor ever told you that you had a sexually transmitted infection?” (0=no, 1=yes).

Acceptability.

We assessed the adolescent’s acceptability of S4E using a modified Session Evaluation Form (SEF). ²⁹ The SEF is a brief 13-item questionnaire given to participants at the end of each module, and overall intervention. Responses range from, “1=strongly agree,” to, “4=strongly disagree.” An example statement is, “I will be able to apply what I learned from the HIV/STI session in my life.” Responses were reverse coded so that higher scores indicate higher levels of acceptability among participants. Cronbach alphas for the HIV/STI, Alcohol/Drug, and overall S4E intervention were 0.87, 0.92, and 0.89, respectively.

Participants’ acceptability was also assessed using an adapted Client Satisfaction Questionnaire-8 (CSQ; α=0.87).³⁰ The CSQ-8 consists of 8 items and assesses participants’ overall satisfaction with the intervention. An example question is, “How would you rate the app you have received?” Response options range from, “1=poor,” to, “4=excellent.”

Quantitative Data Analysis.

We conducted a descriptive statistics analysis for demographic and HIV/STI risk behaviors variables. To evaluate whether and to what extent adolescents found S4E acceptable, means and standard deviation for the SEF (HIV/STI module, Drugs/Alcohol module, and overall intervention), and CSQ-8 were computed. Data were analyzed using SPSS version 23.³¹

Results

Quantitative Findings

As can be seen in Table 1, descriptive statistics indicate that adolescent HIV/STI risk behaviors are prominent. For example, among participants who report having had sexual intercourse in the prior 90 days (71%), over three-quarters reported engaging in condomless sex. Approximately 59% and 55% of adolescents report lifetime HIV and STI testing, respectively. Furthermore, 47% and 35.7% of adolescents report lifetime and past 90-day illicit drug use, respectively.

Table 1.

Adolescent HIV/STI risk behaviors

Variable	f a.	%	SD
Lifetime vaginal, anal or oral sex (n=28)	21	70.0	0.44
Lifetime condomless vaginal, anal or oral sex (n=18)	17	94.4	0.24
Lifetime alcohol use prior to sex (n=20)	7	35.0	0.49
Lifetime drug use while having sex (n=21)	6	28.6	0.46
Past 90 days vaginal, anal, or oral sex (n=21)	15	71.4	0.46
Past 90 days condomless vaginal, anal, or oral sex (n=14)	11	78.6	0.43
Past 90 days alcohol use prior to sex (n=15)	3	20.0	0.59
Past 90 days drug use while having sex (n=15)	3	20.0	0.59
Condomless sex at last sexual intercourse (n=19)	11	57.9	0.51
Alcohol or drug use prior to last sexual intercourse (n=21)	1	4.8	0.22
Ever tested for HIV (n=29)	17	58.6	0.50
Ever tested for an STI (n=29)	16	55.2	0.51
Lifetime STI (n=28)	4	14.3	0.36
Past 90 day STI (n=28)	1	3.6	0.19
Lifetime alcohol use (n=28)	20	71.4	0.46
Past 90 day alcohol use (n=20)	16	80.0	0.41
Binge drinking in the past 90 days (n=15)	8	53.3	0.52
Lifetime drug use (n=30)	14	46.7	0.51
Past 90 days drug use (n=14)	5	35.7	0.50

^a.Frequency of participants who responded yes to each item.

Acceptability.

As can be seen in Table 2, SEF mean scores for the HIV/STI, Alcohol/Drug, and overall S4E intervention indicate high levels of acceptability among adolescents. Furthermore, CSQ-8 mean score for the overall intervention also indicates adolescents’ overall satisfaction with S4E.

Table 2.

Adolescents’ acceptability of S4E (n=30) on a 4-point scale

Scale	Mean	Standard deviation	Range
HIV/STI session evaluation form	3.45	0.41	2.78–4.00
Alcohol/Drugs session evaluation form	3.42	0.47	2.44–4.00
Overall S4E session evaluation form	3.51	0.44	2.44–4.00
Client satisfaction questionnaire	3.40	0.44	2.50–4.00

Qualitative Findings

Themes are organized according to the major goal of this study: to elucidate the usability and acceptability, including (1) content; (2) format; and (3) process of the S4E preventive intervention in primary care. Additionally, we provide a negative case analysis (Table 3).

Table 3.

Illustrative quotes

Theme and Subthemes (n)	Illustrative Quotes
A. Content The risk assessment will have great utility to facilitate clinician-adolescent communication (n=27)     • Feeling more comfortable talking to doctor and asking questions based on common behaviors (n=23)     • Doctor will know your answers beforehand to facilitate conversation (n=18)     • Doctor can provide appropriate care based on answers shared (n=12)     • Youth more comfortable using technology to share info with doctor (n=13)	Because, yeah definitely, because then there’s like something to start the conversation. Like if you need to talk to your doctor about like HIV, getting tested, and you just, it’s hard to bring that up out of the blue. But if you watch a video and then your doctor asks “oh did you have any questions about the, like did you watch the video, did you have any questions? Like, comments, concerns, anything relevant to you?” Then you can kind of say “oh yeah, like it said get HIV tested. I should get tested.” Kind of thing. Or like oh I,that got me thinking (18, Male, White). And like it’s, I like it because it’s helpful so it sort of guides the doctor’s appointment. And like it gives you guys like the subject matter to talk about. It’s, I don’t know it just gives them like a, sort of like a background to like, they won’t just like talk, be talking about like random stuff, cause like that irritates me. Like when you’re at the doctor’s office and then they’re going on about stuff and I’m like well I mean, like, ok (laughs). But it’s not like really beneficial to you, so I like the questions (15, Female, Black). Um, I will feel comfortable because like I did went to this after one time and it wasn’t electronic you have like this questionnaire like you have to fill it out, so it’s kind of like this but like not as complex and high tech and everything but I felt comfortable. I mean don’t see how you would not feel comfortable. I mean it’s just going to stay between you and your doctor (19, Male, Black).
The HIV/STI and drug use modules are youth-focused and culturally specific (n=27)     • Language was age-appropriate and easy to understand (n=19)     • Would recommend app to friends (n=27)     • More engaging than paper/what youth learn in school (n= 12)	This is actually a lot easier because I, one we’re just reading it off paper you know, some people drift off and you can tell that they get bored and stuff like that. So then it got to the point where we got like visual images and we noticed that they were more interested and they’re like the next week ‘oh well can we do more visual things’, so this would be something that would be really helpful. Especially for like the younger group of kids that come because some kids they can start at um eleven if their maturity level is high enough. So, for them like you can tell that they don’t understand some of the words, but if we had this for them I think that they would really understand it a lot better (15, Female, Black). I mean, clearly the job is to educate young people about sex, drugs, and diseases and stuff like that, so like it’s doing what it needs to do and I think the assessment was straight to the point, like he said. Like I think, you know, “hey are these kids having sex?” “Hey are they doing drugs?” “What can we do to inform them to be safe so they don’t infect anyone else or promote anything else that could endanger someone’s life.” And it’s doing the job! (17, Female, Other). Um I thought that it had a very clear vision ‘cause sometimes when people make uh like videos, they don’t make it really clear. Sometimes they use like for example, big words and most people don’t know like long big words. And they were actually going step-by-step. And like… what you’re supposed to do and what you’re not supposed to do. Like pros and cons and stuff like that. And I thought it was very useful (16, Female, Hispanic). I would [recommend to a friend]…cause I know too many people who, like does stuff like this [HIV risk behaviors]. But then they don’t think of the consequences that come after until that happens to them. Too many of my friends are either locked up or experiencing some of the things that happened (17, Female, Mixed Race). You know how we used to be in high school and you watch the video with everybody else? And everybody don’t take it serious cause they think it’s funny because they wanna sit through it and giggle, laugh with they friends. But if you have it on the app and they just doing it theyself, I think they would take it more serious (15, Female, Black).
S4E content promotes HIV/STI and drug use knowledge development and self-efficacy (n=27)     • App is empowering/gives you information to make decisions for yourself (n= 20)     • Statistics were informational and engaging (n= 23)     • Explaining the cause/effect of HIV/STI risk behaviors and drug and alcohol use was helpful (n= 26)	Yeah, I think if, like, someone who doesn’t have refusal skills sees it, it would kind of help them maybe, like, build them a little bit (19, Female, White). I think it’s highly useful, especially for teens. And it’s not like an embarrassing useful, it’s more like a personal let me think about what I’m doing with my life right now useful (17, Female, Mixed) I would have to say [I like] the video about the brain and how it functions after you do alcohol and drugs and stuff like that. Because no one ever really thinks about their brain when they’re taking drinking alcohol or smoking marijuana. They never really think about their brain it just shows you like the levels of how many brain cells you lose (21, Female, Black). The cause and effect of it. I was just mainly talking about the videos and how the videos just tells you what alcohol and drugs do to your brain and what’s the effect of taking it and how can you treat people by taking it and stuff like that (15, Female, Black).
Negative case analysis (n=3)     • Some videos should be modified or omitted (n=3) B. Format Interactive activities will engage youth and increase program uptake (n=25)     • Q&As reinforce app content (n= 22)     • Accessible/no challenges for adolescent population (n= 20)     • App keeps you occupied while waiting for doctor (n= 11)	The first one [video should be omitted]. Although it was funny, it took a while for me to understand the point and I was kinda confused as to what was going on for a while. (19, Female, Black) And then I feel like, it’s good because when people like wanna be doing something, cuz it’s boring just to sit there [in the waiting room]. And so they’ll be like, getting information (15, Female, Mixed). I think that with those little questions they can also help them remind them at least one more time.… I just feel like it was a good way to make sure the person listened to like they got the message (17, Female, Hispanic). It helped me remember things. It helped me remember like the videos and stuff. I missed some questions, but I got most of ‘em. It didn’t feel like I was taking a test. It wa- Like I said, it was just to ensure that I learned something from the videos (14, Female, Black).
Videos highlighting various aspects of HIV/STI and drug use are funny and engaging, yet deliver a serious message (n=25)     • Funny and animated videos keep you engaged (n= 19)     • Videos are informative and educational (n= 22) Voice audio in all aspects of the intervention helped user to follow along and digest the information (n=25)     • Voice over was engaging (n= 19)     • Repetition of information helps to reinforce content (n= 14)	It’s [animations] more understandable than somebody speaking to you about it and- It-, it’s, it jumped over your head, but if you actually see it. Like me, I had to see it to understand. So when I was seeing that, I understood it all-, I understood everything. I understood more than what everybody was telling me (21, Female, Black). I, it was excellent - I really really like it, I really really liked it um, you said the videos were definitely attention-grabbing, I love the fact that there were cartoons and then there was real-life videos so that definitely grabbed my attention (17, Female, Black). I feel like it was great information you know how like, to see how many kids have HIV and stuff like that. And see how many kids like don’t wear condoms during having sex. It was like, it was very informational for some people and I think some kids might like to know that (13, Male, Black). Yeah. That was com-like, the voice it was like, calming type voice but, like informing at the same time. And so, I… it… it made me like, focus in… it make me like, you know, want to, like, listen… and some people… a lot of like teens don’t really like to read (21, Female, Black). Uh audio was definitely a plus it was clear it was not hard to understand and it gave you a sense of like somebody talking to you obviously, but like that is a good thing when you are listening to a presentation. And it wasn’t just, it wasn’t just repeating information that was already on the screen. It was other stuff too so I liked the audio part (16, Male, White). Because it go directly straight to the point of what’s actually going on and was I comfortable, and I would say “yeah” you know, it was straight to the point, I liked that (21, Female, Black)
The graphics and bright colors are visually stimulating (n=25)     • Eye-catching colors (n= 14)     • User interface is engaging (n= 15) Negative case analysis (n=5)     • Interactive activities should be integrated throughout the intervention, not only at the end (n=4)     • You should be able to navigate throughout the app (n=5)	Um, yeah. But I do like the colors, the colors is a big thing. If you have bright colors, kids are gonna be like “oh what’s this?” and if you have neutral colors they’re not going to be like, they’re not going to be as an interested as you would like them to be and in terms of like, you really if you’re going to take this out into the community or anywhere you have to be able to keep people’s attention (19, Male, Black). Yeah. I think it’s really um …it just catches your interest. All the colors are great. They’re vibrant and just exciting. I love color (20, Female, White). Um, I felt like when talking about the colors, I kept mentioning it being neutral so that it’s not just like all girly colors or all guy colors. That’s why the colors were very interesting for anybody to look at. Like you could just looking at it’s like ‘oh well this looks really interesting and it’s very colorful, very bright’. So I really enjoyed that (15, Female, Black). After that video, I was kind of bored with that portion of the app. I was like, there were no [interactive activities] for me to answer. There was no additional information. It was just videos and that kind of got boring after a while…So, I was expecting that [interactive activity], and I didn’t get that. So, I was kinda let down. Plus, it just seemed longer than [the first module] because there was nothing interactive to do (15, Female, Black). I like the information and stuff and some of the videos were really helpful. But, you should have a little bit more freedom when navigating. Like the STD videos, I feel like that it should be like a chart and then you can pick which ones you wanted to watch and stuff. Instead of having to watch like all of them…So yeah, I kind of want some more freedom and navigating around (15, Female, Black).
C. Process Completing S4E in the waiting room prior to seeing the clinician may be most effective for capturing youths’ attention (n=25)     • Completing app before seeing doctor (n= 18)     • Complete app in waiting room (n= 13)     • Have the app during the entire doctor’s appointment (n= 11)	It could, I mean it all depends on how fast the doctor will come out and get the person or will they just actually set a time frame for them to … well it really depends because like what if they got something else to do after the appointment and, so it’s really, I feel like it’s just should be out in the waiting room and also you know when you go to get your weight checked and then they tell you to sit in the room until after the nurse get done until the doctor comes back, then you could continue finishing just in case you don’t finish it in the waiting are (21, Female, Black). I think if, like, you-- like, in order to actually finish it-- if you got it right after you checked in and then, like, took it with you when the nurse took you back and then you could finish it then (19, Female, White). Giving out at check in would be the most simple way to do it and probably most effective way of doing it (17, Male, Hispanic).
The length of the S4E intervention is important to consider (n=25)     • Ability to complete entire app in doctor’s office depends on length (n= 23)     • Making app accessible outside of doctor’s office (n= 11)	In the classrooms in health they always talk about sex and drugs and all that so it would be a good idea to have this app in the classrooms and in the waiting room as well cuz, because when I am in the waiting room in hospital it takes like an hour to be seen (14, Male, Hispanic). I just think that it’d be good if you had like a menu so that people can go to different parts of the module, because like, it’s kinda long and if like, someone leaves, and comes back to it, then, like, I wouldn’t want to sit through all of it, all over again. Ummmm, other than that I thought it was pretty good (20, Female, White). I feel like sitting in a waiting room could take anywhere from 15 minutes to an hour, so I feel like if you had the option to do what you wanted to do, if you’re shorter on time you can get to some things that are more important but at the same, like if you’re an hour, you might be able to get to all of it. So I feel like 15 to an hour is the most or least for anyone to wait in the waiting room (20, Female, White).
Ensuring confidentiality is essential to adolescents’ disclosure of HIV/STI risk behaviors (n=25)     • Increased privacy needed for app data intake (n= 10)     • Giving consent to share assessment with physician (n= 17)	You could get, like, those screen things for the iPads, like they have for the computers where you type in all this stuff (clears throat), like when you initially have your first visit. ‘Cos I mean, its, like, not really any information and it differs (18, Female, White). I think it should be optional because not every teen is comfortable with it and they might get a little bit upset, you know? I don’t know if I’m the only one who thinks that, but that’s-that’s just where I think, I think (20, Female, White). Um, well, I think it should be an option - because with the survey - it’s just - a checked box, an unchecked box, it’s not actually talking to your doctor and telling them like, being honest with them? It’s just filling out a survey and sometimes people don’t fill out the survey correctly or they misunderstand it but with a conversation about it (20, Female, White).
The flow of S4E maintains adolescents’ engagement in the intervention (n=25)     • Interactive (n= 20)     • Good flow of content (n= 20)     • Two modules complement each other (n= 17) Negative case analysis (n=5)     • The need to shorten the length of the intervention (n=3)     • Rearrange the order of the videos (n=3)     • Additional recommendations to improve the app (n=5)	Oh, yeah. The layout is great. Everything about that, even the way the words are, like, flowing up on the screen, the way he’s talking with the words coming up; I think that’s all a good pace and it’s all presented really well (20, Female, White). Yeah, especially how they incorporated the SODAS acronym with both of them. That’s what tied it in for me. Cause they were both like, um, both kind of, they showed it in the HIV and STD, STI ones. Like in the beginning kind of, and then at the end as well. And I liked how they did at the end for the drugs and alcohol one, just to sort of just be like this is finished. And I liked that. Cause that’s what made it seem like that everything was correlating with each other (15, Female, Black). It’s a really great app and… like I like how you can from one thing to the next, but it’s still like, you’re still like learning things, you’re still keeping your like your brain flowing and thinking. I just really like it as a whole because it has flow (19, Female, Mixed). It was kind of long and dragging out. So after a while you lose interest and then you don’t even remember half the stuff that was in there because it was so long and dragged out. (21, Female, Black) Basically, just what I said before about like the first video away and like the last one being the first one (19, Female, White). Well I do think it should be a good idea to like, at the end of a page from part of the app, is different contact to different clinics so that someone who don’t have a doctor or someone who don’t know where to find one they can go to that number and contact that person and say “hey I’m a new patient, I don’t know what to but can you help me” and then they’ll prescribe them to come into the clinic or so (21, Female, Other).

Content.

The risk assessment will have great utility to facilitate clinician-adolescent communication.

Adolescents affirmed that the risk assessment was acceptable and may be helpful for clinician-initiated HIV/STI risk and testing communication. Specifically, participants expressed that completing the risk assessment on a tablet may ameliorate stressors associated with adolescent-initiated, face-to-face conversations with clinicians. One adolescent stated, “I feel it would be easier to do it on the tablet than to actually talk to the doctor…Because there’s stigma around STIs and I feel it’s kinda hard for people to want to talk about it” (15, Female, Mixed Race). Participants expressed that completing the risk assessment on a tablet prior to their visit may prevent non-disclosure of HIV/STI risk behaviors because the clinician would already have access to their responses:

[Youth] be more comfortable answering on the app compared to talking to the doctor. Cause they probably scared, nervous. Instead, [youth] could just walk in the room and then you just calmly get straight to it [HIV/STI risk communication] instead of [the clinician] like, “Oh, what’s your concern?” And [youth] might just, “Oh, I don’t wanna ask no more” (21, Female, Black).

The HIV/STI and drug use modules are youth-focused and culturally specific.

Given that CBPR principles were employed in the development of S4E content, it is not surprising that adolescents found the HIV/STI and drug use content to be engaging, youth-focused and culturally relevant for the targeted community. One adolescent described, “It kept you engaged and to the point where it’s not boring” (17, Female, Mixed Race). Participants went on to describe their experience:

It [S4E] was actually pretty good. I like that there was a spread of things throughout the video. It went from animated videos or real life videos to PowerPoints with facts and graphs and somebody narrating. It had a really widespread of the ways that the information was delivered. That would really help people understand things. It’s easier when information is given to you different ways (16, Male, White).

S4E content promotes HIV/STI and drug use knowledge development and self-efficacy.

Adolescents described that the S4E content, grounded in prevention science, was informative and interesting. One participant mentioned, “It was great information to see how many kids have HIV. And see how many kids don’t wear condoms during sex. It was very informational and some kids might like to know that (13, Male, Black).” Adolescents indicated S4E may increase self-efficacy refusal skills and decision making. One youth shared, “The idea of SODAS [Stop, Options, Decide, Ask, Self-praise] was extremely helpful…It will not only help them choose [safe decisions], but also helps build self-esteem” (18, Female, Mixed Race).

Format.

Interactive activities will engage youth and increase program uptake.

Participants reflected on the ways in which interactive activities were useful in engaging them and completing the S4E intervention, as well as tested their knowledge. One participant shared, “It [interactive activity] was good. It was useful because it makes you have to actually pay attention and think back on what you just watched (21, Female, Black).”

Videos highlighting various aspects of HIV/STI and drug use are funny and engaging, yet deliver serious messaging.

Adolescents reported that the videos in the app disseminated serious messaging with respect to HIV/STI and drug abuse in a manner that was culturally syntonic to adolescents and thereby engaging. One youth shared, “The videos were so funny and weird, I think more people can relate to them and try to get a laugh out of it…try to get that relief and then you’re also learning about something at the same time (15, Female, Black).” Adolescents expressed that the videos provided a balance between funny and serious. One youth indicated, “I liked the videos because I think they’re funny but they also show the importance of issues. So, it’s entertaining but you’re also learning at the same time (21, Female, White).”

Voice audio in all aspects of the intervention is helpful to follow and digest the information.

An important aspect of the S4E intervention is the voice audio incorporated throughout the program. Participants shared that including voice audio provides them opportunities to engage visually, as well as auditory in the intervention. One participant reflected, “The audio was really good and he spoke with inflection to the point where it was conversational” (18, Male, White). Similarly another adolescent stated, “He sounds like a figure of authority, but not too much that makes you draw away from them or pull back, so I like that” (15, Female, Black).

The graphics and bright colors are visually stimulating.

Adolescents noted that the colors and visuals of S4E were aesthetically acceptable. One participant mentioned, “It was very eye catching. The colors are popping, but not screaming at you” (17, Female, Black). Another adolescent stated:

It was captivating! The colors, they catch your attention. It’s not just one or two colors, that’s very bland. We have blue, orange, and red and then the collage around the entire screen makes it more complex and it grabs your attention (17, Male, Hispanic).

Process.

Completing S4E in the waiting room prior to seeing the clinician may be most effective for capturing youths’ attention.

Adolescents described that participating in the S4E intervention prior to their clinician visit may be the most opportunistic time. An adolescent expressed, “When I go to the doctor’s, I end up sitting in the waiting room for like 20 minutes…I’d rather be occupied with something else instead of boring paperwork” (15, Female, Black). One youth said:

I would do S4E in the waiting room…being in the waiting room all quiet and just terrified of what the results might be, I could go to this app and be like, “OK I’m calm, oh this could happen? Well I didn’t know that.” Especially information that you didn’t even know and then when the doctor show back up, you’ll be more calmer (21, Female, Black).

The length of the S4E intervention is important to consider.

Participants highlighted the importance of considering the length of S4E if the intention is complete the intervention in the waiting room of a primary care clinic. An adolescent stated, “In order to actually finish it, if you got it right after you checked in, and then took it with you when the nurse took you back and then you could finish it then (19, Female, White). Another adolescent shared:

I think it’s about the perfect length. It’s not gonna be quick that your doctor sees you ‘cause he has other people to see. So while you’re waiting, usually there’s quite a bit of wait. I think it’s a really good length. I think it’s pretty much perfect length (19, Female, Mixed).

Ensuring confidentiality is essential to adolescents’ disclosure of HIV//STI risk behaviors.

Adolescents emphasized the significance of maintaining confidentiality of their responses to the risk assessment. One participant mentioned, “Maybe having the privacy cover over the iPad? The only thing I can see is sitting next to somebody and they’re peeping over and seeing your answers” (18, Female, Black). Similarly another participated expressed:

I know there’s screen protectors you can get for iPhones and iPads. Make it so only the person looking directly at it can see. And then people from directly at the side can’t look at an angle. That would probably be pretty important (18, Male, White).

The flow of S4E maintains adolescents’ engagement in the intervention.

Adolescents shared that the S4E process, including the risk assessment, use of videos, delivery of HIV/STI and drug use information, and interactive activities, is helpful for engaging and retaining adolescents in the prevention program. An adolescent expressed, “I liked the way it was organized. It kept you like, ‘Oh, what’s goin’ happen next?’” (21, Female, Black). One youth shared:

It wasn’t long. It was short. It was simple. It was learning and it was interesting. It had good vision, you could actually see everything really clear. It was going step-by-step. I thought it was pretty good (16, Female, Hispanic).

Discussion

Usability and acceptability of S4E, a mHealth HIV/STI and drug abuse preventive intervention,¹³ among adolescents in primary care was high. Specifically, participants described favorable views on both the usability and acceptability of S4E, including content, format and process. Notably, adolescents indicated that S4E may have great utility in facilitating clinician-initiated HIV/STI risk behaviors communication. Future research should examine the effects of S4E on clinician-adolescent HIV/STI risk communication, a potential mechanism by which S4E may work.

Given that mHealth, including apps, are ubiquitous in many adolescents’ lives,³¹ they offer innovative approaches to deliver prevention programs^10,33 that are culturally specific and youth-driven.^34,35 Few researchers, however, have capitalized on the widespread use of apps³³ in curbing the tide of the HIV/STI epidemic.³⁶ In the present study, we provided iPads to participants in the clinic. However, given the significant proportion of youth who report having access to smartphones and tablets²¹, future research should examine the feasibility of transitioning S4E from clinic-based iPads to participant-owned devices outside of the clinic. Equally important, the healthcare landscape is significantly and rapidly changing as a result of the Patient Protection and Affordable Care Act, including increases in primary care visits,³⁷ which provide researchers and clinicians opportunities to deliver HIV/STI and drug abuse prevention programs. In fact, recommendations issued by the United States Preventive Services Task Force underscore the need for clinicians to provide HIV/STI preventative services, including screening for HIV/STI risk behaviors and status.³⁸

Parallel to these federal recommendations, researchers have highlighted the urgent need to increase translational HIV/STI and drug use intervention research in primary care settings.³⁹ Despite the facts that (a) mHealth apps are burgeoning as a result of the technology boom, (b) primary care settings offer a ready-made context to engage youth in preventative services, and (c) federal and professional recommendations highlight the urgent need for HIV/STI preventative services in primary care, research on the usability and acceptability of mHealth HIV/STI and drug abuse preventive interventions in the target population remains limited.^40,41 Present findings work to address this critical gap in research by providing data on the usability and acceptability of S4E among adolescents.

Noteworthy, relative to the general U.S. adolescent population, participants in this study appear to be at increased risk of engaging in HIV/STI risk behaviors. When compared to the general 12^th grade adolescent population (47%),² 58% of participants reported condomless sex at last sexual intercourse, underscoring the need for preventive interventions in this setting. Although the prevalence of HIV/STI risk behaviors in this sample is alarming, the majority of adolescents report HIV and STI testing. In fact, relative to national data indicating that 17.2% of all 12^th grade adolescents report having ever been tested for HIV,² 58.6% of adolescents in this study reported lifetime HIV testing. Findings suggest that youth-centered healthcare clinics may be opportunistic venues for adolescent HIV/STI testing—research and practice goals highlighted by federal agencies.⁶

Participants were recruited from one youth-centered community-based health care clinic, hence, the usability and acceptability of S4E may not be generalizable to other adolescent populations. Future research should examine the extent to which S4E is acceptable among other adolescent populations. Additionally, participants were primarily female, racial and ethnic minority adolescents. However, this is a population disproportionately burdened by HIV/STI, and we sought to recruit a sample that was representative of the clinic population. Furthermore, HIV/STI risk behaviors and testing data are self-reported, and adolescents may under or over report. Future research should collect biomarker data to complement self-report data. Finally, the acceptability of S4E was assessed using measures with a 4-point scale, which does not allow for participants to choose a midpoint response. However, researchers have indicated that 4-point scales may minimize social desirability bias, relative to 5-point scales with a midpoint.⁴²

Conclusions

Notwithstanding these limitations, few researchers have assessed the usability and acceptability of a mHealth HIV/STI and drug abuse prevention app in primary care. This study addresses this gap in research and aligns with federal high priority research goals of assessing and targeting HIV/STI risk behaviors and increasing HIV/STI testing aimed at reducing HIV/STI and drug use health inequities. Now that S4E has been shown to be acceptable among a clinic sample of predominately racial and ethnic minority youth, a next important step is to examine the effects of S4E on adolescent HIV/STI risk behaviors, including sexual risk and drug use^{43, 44}, and HIV/STI testing⁶ in a randomized clinical trial. This program of research has the potential to enhance the reach and impact of clinicians in delivering preventive HIV/STI services to adolescent populations via mHealth platforms.