Skip to main content
NCBI home page
European Journal of Psychotraumatology logoLink to European Journal of Psychotraumatology
. 2023 Jan 10;14(1):2157933. doi: 10.1080/20008066.2022.2157933

Teen Well Check: an e-health prevention program for substance use, sexual assault, and sexual risk behaviors for adolescents in primary care

Chequeo de Bienestar del Adolescente: Un programa electrónico de salud para el uso de substancias, agresión sexual, y conductas sexuales de riesgo para adolescentes en atención primaria

青少年健康检查：针对初级医疗中青少年物质使用、性侵犯和危险性行为的电子健康预防计划

Amanda K Gilmore a,CONTACT, Elizabeth A Mosley a,b, Daniel W Oesterle c, Leigh E Ridings d, Idara Umo a, Anna Hutchins a, Holly C Gooding e, Elizabeth Wallis f, Sharon Levy g,h, Kenneth Ruggiero c, Debra Kaysen i, Carla Kmett Danielson j, Shannon Self-Brown a
PMCID: PMC9848361  PMID: 37052109

ABSTRACT

Objective: Adolescents are at risk for substance use, sexual assault, and sexual risk behaviours; however, to date no integrated prevention programmes address all three risk behaviours. The goal of this study was to evaluate the usability and acceptability of Teen Well Check, an e-health prevention programme targeting substance use, sexual assault, and sexual risk among adolescents in primary care settings.

Methods: The current study included content analysis of interviews with adolescents in primary care (aged 14–18; n = 25) in the intervention development process, followed by usability and acceptability testing with qualitative interviews among adolescents in primary care (aged 14–18; n = 10) and pediatric primary care providers (n = 11) in the intervention refinement process. All data were collected in the Southeastern U.S.

Results: Feedback on Teen Well Check addressed content, engagement and interaction, language and tone, aesthetics, logistics, inclusivity, parent/guardian-related topics, and the application of personal stories. Overall, providers reported they would be likely to use this intervention (5.1 out of 7.0) and recommend it to adolescents (5.4 out of 7.0).

Conclusions: These findings suggest preliminary usability and acceptability of Teen Well Check. A randomized clinical trial is needed to assess efficacy.

KEYWORDS: Prevention, adolescents, e-health, drug use, sexual assault, sexual health

Highlights

  • Adolescents are at risk for substance use, sexual assault, and sexual risk behaviours.

  • The goal of this study was to evaluate the usability and acceptability of Teen Well Check, an e-health prevention programme targeting substance use, sexual assault, and sexual risk among adolescents in primary care settings.

  • Providers and adolescents rated Teen Well Check as usable and acceptable, and providers indicated that they would recommend it to their adolescent patients.

Substance use, sexual assault, and sexual risk behaviours among adolescents are common and interrelated (Johnston et al., 2021; Scott-Sheldon et al., 2016; Smith et al., 2018). Among adolescents surveyed in the United States during 2018, past year substance use was 38.3% for alcohol, 24.6% for cannabis, 30.7% for any vaping, and 9.2% for illicit drugs other than cannabis (Johnston et al., 2021). Substance use and sexual behaviour often co-occur, with 22.4% of sexually active adolescents aged 14–18 years using substances before their most recent sexual encounter (Johnston et al., 2016). Impairments in sexual decision-making due to substance use can lead to a range of consequences, including sexual risk behaviours and non-consensual sex. Adolescents are disproportionally affected by the consequences of sexual assault and sexual risk behaviours, being more likely to contract sexually transmitted infections (STIs; Kreisel et al., 2021) be sexually victimized (Black et al., 2011), and experience hardship due to unwanted pregnancy (Noll et al., 2019). Prevalence of these risk behaviours among youth has increased since the onset of the COVID-19 pandemic (Dumas et al., 2020; Masonbrink et al., 2022). Prevention of these three interrelated major public health concerns would significantly reduce mental and medical health burden.

Perceptions of peer behaviour and attitudes, or perceived social norms, are associated with alcohol use, drug use, sexual assault, and sexual risk behaviours among adolescents and young adults (Hoxmeier et al., 2018; Kantawong et al., 2021; Pedersen et al., 2017). Communication skills including how to talk about sex and sexual consent, as well as communication with parents and providers about risk behaviours, are associated with decreased alcohol use, drug use, sexual assault, and sexual risk behaviours (Hops et al., 2011). Personalized normative feedback and communication skills practice may be an effective cross-cutting strategy to reduce and prevent risk for alcohol and drug use, sexual assault, and sexual risk behaviours among adolescents.

Many adolescents seek routine preventative healthcare in primary care settings. Recent health reforms have improved population health outcomes and primary care services (Hofer et al., 2011). e-Health screening and brief interventions allow standardized screening for adolescents prior to meeting with the primary care provider. Adolescents also prefer technology-based screening (Gibson et al., 2021) compared to screening conducted by a provider. A systematic review and meta-analysis of eHealth interventions with adolescents to prevent risk behaviours found beneficial effects across a range of behaviours (Champion et al., 2019). This manuscript describes the development and acceptability testing of an e-Health integrated prevention programme for adolescent substance use, sexual assault, and sexual risk behaviours in primary care settings using the Intervention Mapping framework (Bartholomew Eldredge et al., 2016). The aim of this project was to develop an intervention that could be disseminated into primary care settings without adding burden to primary care providers.

1. Methods

1.2. Participants and procedures

All study procedures (see Figure 1) were approved by the university IRB and were completed between April 2018 and October 2021. All data were collected in the Southeastern USA.

Figure 1.

Figure 1.

Open in a new tab

Study flow.

The intervention was developed using the Intervention Mapping framework (Bartholomew Eldredge et al., 2016) and followed practices consistent with the Template for Intervention Description and Replication (TIDieR) checklist (see Table 1). For a description of the target outcomes and theoretical components, see Table 1. Theory-based processes and qualitative interviews with adolescents (Intervention Development Phase described below) were used to develop the intervention. Using the Intervention Mapping Framework, we translated methods and strategies into an organized programme, consulted with intended participants and stakeholders, prepared programme materials and piloted programme materials for feedback on the acceptability of the programme from providers and adolescent patients (Intervention Refinement Phase described below). In the intervention development process, we prioritized using theory-based methods and content that were acceptable to providers and adolescent patients that would not add burden to primary care providers and disrupt clinic flow. Stakeholders in this process were primary care providers and they gave in-depth feedback on Teen Well Check during the Intervention Refinement phase.

Table 1.

Template for Intervention Description and Replication (TIDieR) checklist.

  Teen Well Check TIDierR checklist
Why: A brief, technology-based integrated prevention for adolescents focused on substance use, sexual assault, and sexual risk in primary care settings has the potential to prevent related risk behaviours in a brief format that can be delivered as part of a wellness visit that does not add burden to primary care providers.
What (material): See Table 2 for a description of intervention components and Figure 2 for examples of the content included in the technology-based intervention.
What (procedures): Teen Well Check can be delivered to adolescents aged 14–18 visiting a primary care clinic and can be delivered in the waiting room, while waiting in a clinic room, or after the visit.
Who provided: Teen well check is a self-led programme and does not require any particular training or person to deliver the programme. However, it was developed to be provided in the context of a primary care visit.
How (mode of delivery): Teen Well Check is a technology-based programme that is delivered through a web-app. The web-app can be accessed via a tablet, smart phone, or computer.
Where: Pediatric primary care clinics. If delivered in the waiting room or a clinic room, the clinics would need a tablet, smart phone, or computer that is connected to internet. If delivered after the primary care visit, the patient would need access to a tablet, smart phone, or computer that is connected to internet.
When and how much: Teen Well Check is a single session programme. It takes approximately 5–15 min to complete.
Tailoring: The current iteration of Teen Well Check is tailored by age, gender, and participant report of substance use. Participants are provided with age and gender tailored normative data throughout the intervention. Further, if participants report no substance use, they are able to choose which substance they want to learn about. If they report using one substance, they are given information about that substance. If they report using multiple substances, they are able to choose which substance they want to learn about out of those they have used. Given that it is a technology-based programme it could be tailored more specifically in future iterations.
Modification: Yes, the purpose of this study was to modify the intervention based on feedback from providers and adolescents. Please see Figure 2 for examples of how the sample content changed after feedback.
How well (planned): In the current study, participants reviewed the entire programme because an interview took place after viewing the intervention content. Future work needs to assess fidelity to the programme when delivered as part of a feasibility trial. Therefore, adherence was not assessed as an outcome in the current study.
How well (actual): Adherence was not assessed as an outcome in the current study.
Open in a new tab

1.2.1. Intervention development phase

Adolescents aged 14–18 years old (n = 25), attending a wellness visit at their primary care doctor were approached in the waiting room or patient room by research staff to complete an eligibility screen for the study. See Table 2 for participant information. All procedures were conducted at academic medical centres that primarily served low-income families insured by Medicaid. The eligibility screen assessed if adolescents met the following eligibility criteria: between 14 and 18 years old, could read and understand English, did not have a developmental disability that would prevent them from understanding the programme contents, and reported knowing a peer who has ever used substances. Eligible individuals then consented (parents) and assented (adolescents) to the study procedures. Adolescents completed a questionnaire on a tablet, interview to provide feedback on the intervention development phase of Teen Well Check (see Figure 2 and below for more information) using screenshot mockups shown to adolescents on a tablet by research staff, and questionnaire about the intervention on a tablet. All study procedures after the consent process were conducted in a private room in the clinic without the parent present. Participants received a $20 gift card for participation.

Table 2.

Theory- and evidence-based components in Teen Well Check.

Target outcome Theoretical framework Prevention components
Substance use Motivational Interviewing (Naar-King & Suarez, 2011) Screening, reinforcement for non-use, self-identified consequences of use (Dimeff et al., 1999; Naar-King & Suarez, 2011; National Institute on Alcohol Abuse and Alcoholism, 2011)
Social Norms Theory (Borsari & Carey, 2001; Collins et al., 1985; Lewis & Neighbors, 2004; Perkins, 2002) Correcting misperceptions of perceived and actual peer norms of alcohol and drug use
Ecological Framework (Bronfenbrenner, 1979) Increase communication skills with peers (including romantic and sexual partners), parents, and medical providers regarding use
Negative Reinforcement Theory (Skinner, 1958) Psychoeducation on traumatic stress symptoms and use of substances to cope with traumatic stress symptoms, encouraging active coping strategies
Substance use before sex Sex-Related Substance Use Expectancies (e.g. Alcohol and Marijuana Expectancy Theory) (Hendershot et al., 2010) Interactive feedback regarding sex-related substance use expectancies (Gilmore et al., 2022)
Impairment Theories (e.g. Alcohol Myopia Theory) (Steele & Josephs, 1990) Interactive feedback applying alcohol myopia theory to sexual decision-making (Gilmore et al., 2022)
Sexual risk behaviours Social Norms Theory Correcting misperceptions of perceived and actual peer norms of sexual risk behaviours (Lewis et al., 2014)
Skills Deficit Model Interactive education to increase condom use, condom use self-efficacy, and sexual communication skills
Sexual assault victimization Ecological Framework/Cognitive Mediational Model (Gidycz et al., 2006; Nurius & Norris, 1996; Ullman, 2014) Increase sexual communication skills, sexual assault risk perception skills, and effective use of resistance strategies through scenarios (Gilmore et al., 2022)
Sexual assault bystander intervention Social Norms Theory/Bystander Education (Fischer et al., 2011; Jouriles et al., 2018; Labhardt et al., 2017; Latané & Darley, 1970; Latané & Nida, 1981; Latané & Rodin, 1969) Interactive bystander education, psychoeducation on substance use and bystander intervention (Gilmore et al., 2022)
Open in a new tab
Figure 2.

Figure 2.

Open in a new tab

Sample content from Teen Well Check across each phase of the study.

1.2.2. Intervention refinement phase

Adolescents aged 14–18 years old (n = 10), completed the same procedure process as the Intervention Development phase but viewed the entire programmed Teen Well Check programme (see Figure 2 and below for more information) that was programmed in a tablet-based application. See Table 2 for participant information. There was no overlap in adolescents in the samples.

Further, we completed usability and acceptability testing via interviews among pediatric primary care providers in community clinics (n = 11). Providers were shown screenshots of the programme during the interview and were compensated via a $100 gift card drawing.

1.3. Teen Well Check development using iterative, end-user focused design

Teen Well Check contains three modules created for this intervention specific to substance use, sexual risk behaviour, and sexual assault, presented in a linear fashion using a motivational interviewing approach. The content of the intervention was designed based on existing literature and published theory as described below and outlined in Table 2.

1.3.1. Substance use module

Participants received personalized feedback on perceived and actual age- and gender-specific substance use, psychoeducation regarding the effect of substance use on brain development, their own consequences experienced, and psychoeducation related to the substance of choice (1. Vaping/cigarettes/Juul; 2. Alcohol; 3. Cannabis/marijuana; 4. Prescription opioids; 5. Other drugs). If participants only used one substance, they learned about that substance. If they used multiple substances, they chose which substance they had used that they wanted to learn about. If participants had not used substances, they were provided an option to learn about any substance. Psychoeducation also included a short video developed by NIDA about brain anatomy, physiology, and effects of substance use (National Institute on Drug Abuse [NIDA/NIH], 2019).

1.3.2. Sexual risk module

Participants were asked questions and provided feedback about how many adolescents their age had been tested for STIs and how many used substances prior to engaging in sexual behaviour. Psychoeducation focused on condom use and how substances can impact sexual communication.

1.3.3. Sexual assault module

Scenarios of potential sexual assault situations were provided where participants could learn how to respond as a potential victim, perpetrator, or bystander. Information regarding sexual consent was also provided, including information about how substances can impact sexual consent.

1.3.4. Teen Well Check intervention development

The intervention development phase (see Figure 2) included ten mock-up designs that were displayed to participants on a tablet. Mock-up designs were developed using theory-based cross-cutting targets identified by the research team. After receiving feedback on the intervention development phase mock-ups, interviews were analyzed. Once results were identified and thematic saturation was achieved, adaptations were made to the content by the research team.

1.3.5. Teen Well Check intervention refinement

Teen Well Check was optimized for use on a tablet to be delivered in primary care waiting rooms. It was presented to teens and providers, where interviews were determined to be completed after reaching thematic saturation. Adolescent feedback was used to make adaptations to the content by the research team during the final iteration, which was programmed into a web-based prevention programme (see Figure 2). In each stage, all adaptations requested were made unless there was disagreement among participants or if it was not feasible within the scope of the study.

1.3.6. Teen Well Check logistics

Before the COVID-19 pandemic, teen wellness visits usually lasted 30 min and intake paperwork (including substance use screening forms) was completed before arrival or in the waiting room before the appointment started. The initial concept of Teen Well Check was for it to be delivered in the waiting room in a single session of 5–15 min. Due to the clinic flow shift that occurred during the pandemic, patients waited in their cars or outside of the clinic rather than in waiting rooms as was typical pre-COVID. For this reason, combined with the concern about sanitizing requirements for tablets, Teen Well Check was programmed to be accessible remotely via a personal device.

1.4. Measures

Interview. Adolescents and providers completed a semi-structured interview ranging from 30 to 60 min to provide feedback about Teen Well Check, including aesthetics, content, and interest in using the intervention.

Acceptability and usability were assessed among adolescents and providers using the Post Study System Usability Questionnaire (PSSUQ) (Lewis, 2002) and questions developed by the research team. The PSSUQ is a 19-item instrument that measures satisfaction with a computer system (Bartholomew Eldredge et al., 2016) with answer choices ranging from ‘Strongly Agree’ (1) to ‘Strongly Disagree’ (7). Lower scores indicate more acceptability and usability. Providers were also asked the likelihood of using and recommending Teen Well Check ranging from ‘Very Unlikely’ (1) to ‘Very Likely’ (7). Mean scores will be used. (Table 3).

Table 3.

Characteristics of primary care providers (n = 11) and adolescents 14–18 years (n = 35).

Characteristic Providers Adolescents (Intervention Development) Adolescents (Intervention Refinement)
N % N % N %
Gender
 Female 11 100% 14 56% 8 80%
 Male 0 0%  9  36% 2 20%
Other (Genderfluid, Two Spirit) 0 0% 2 8% 0 0%
Age     16.3 (M)  1.37 (SD) 16.7 (M) 1.49 (SD)
Race
 White/Caucasian      9 36% 6 60%
 Black/African American     11  44% 1 10%
 Asian American/Pacific Islander      0 0% 1 10%
 Biracial      2  8% 1 10%
 Other/Mixed      3  12% 1 10%
Ethnicity
 Hispanic/Latinx      2 8% 1 10%
 Not Hispanic/Latinx     23  92% 9 90%
Sexuality
 Straight/Heterosexual     15 60% 5 50%
 Bisexual      4 16% 3 40%
 Gay/Lesbian      4 16% 0 0%
 Questioning         2 20%
 Other (Pansexual)     2 8% 0 0%
Tablet Access     24  96% 9 90%
Past year alcohol use     3* 12%* 4 40%
Past year tobacco use     N/A*  N/A* 5 50%
Past year drug use     1*  4%* 7 70%
Open in a new tab

*Note: During Aim One, participants were not asked about smoking behaviours or past year of alcohol or drug use. These results for Aim One reflect past month alcohol and drug use.

1.5. Qualitative data analysis

Interviews were content analyzed using an iterative team-based approach led by a PhD-level researcher. The researcher reviewed each transcript and developed a list of topics that emerged both deductively from the interview guide and inductively from participants’ responses. Using those topic lists as a preliminary codebook and NVivo (QSR International Pty Ltd, 2018), two research staff members separately coded (White et al., 2022) the transcripts, compared and refined codes for each transcript, and reconciled their coding until they had 100% agreement (i.e. reviewed coding together in real-time and came to agreement). Coders developed analytic memos for each code, noting major findings and important quotations. The researcher used those analytic memos, reports from NVivo (ex: code co-occurrence, transcript coverage), and the coded quotations to identify the most salient results.

2. Results

2.1. Intervention development phase of Teen Well Check: adolescent feedback

2.1.1. Aesthetics

Participants provided feedback on how to improve aesthetics of pictures, animations, and videos (see Quote 1 in Table 4). Formatting feedback was provided to improve aesthetic acceptability of the intervention, including commenting on the colour scheme and font. Participants suggested adding more visual components and images to enhance usability throughout the programme. Participants commented on the importance of using images and language that were inclusive and representative of diverse identities (see Table 4 for sample quotes).

Table 4.

Sample feedback quotes on phase of Teen Well Check.

Adolescent interviews Provider interviews
Intervention development results Sample quotes Intervention refinement results Sample quotes Intervention refinement results Sample quotes
Aesthetics (1) ‘Well, color-wise, there are like, or from what I’ve heard, there are colors that attract the human eye more. So, like color-wise, making it, a little more colorful, maybe even the font wise. It's some, it's trying to reach each teenage group, so it's like, if I see color, like if see the love right there and it's in red, it's like a pop out.’ Engagement and Interaction (5) ‘I actually really liked that because it actually helps people learn about your brain. And I didn't even know that some things are certain things. So, I think that's really cool … I like how you added a video too. That's like, I like videos.’ Engagement and Interaction (6) ‘I like the way you guys asked questions that they had to kind of guess on and then you gave them the correct response. So I like that it was interactive.’
Alcohol and Substance Use (2) ‘Some people don't really know about the brain but, this just tells you about it basically and how you could stop doing that because this is how it could affect you like you’re just thinking about it, like oh people told me about lungs, smoking and that but then your brain is also slowly functioning because of you doing that. Yeah, I like that.’ Language and Tone (7) ‘I feel like it's good. I don't think it's very criticizing of people. So, it's very open-ended. And I like how you already included the alcohol at the party. So, it's not like, oh, you should just avoid all risky situations to begin with and like – I like how it automatically – I mean, they’ll understands that we do have – people have sex. And it's not just, oh, you just use abstinence, like it welcomes the idea that you are allowed to do things that like you’re allowed to drink and you’re allowed to, like, have fun and like, engage in sexual activity as long as you’re safe. So, it's kind of acknowledging maybe like, yes, people do instead of ignoring like, oh, you should just not do it until later.’ Language and Tone (8) ‘It sounds harsh. I’m having trouble articulating but like almost like judgy … So anything that has somewhat like all or nothing language, I think unconsciously can turn some kids off and … sort of missed an opportunity. So just subtle things like the word “forever,” which is true, but … You know, maybe that's too darkening … When you do these kind of screening things. The more that you feel you can be asked without judgment. I think the more it lands.’
Healthy Relationships (Sexual Risk) (3) ‘When I took sex ed and like, I think it was eighth grade maybe, I saw this video about like how to put on a condom and it was like this cartoon thingy –– it was teaching us how to put condoms on and he was like putting it on himself and like sliding it down, it was kind of weird but it was like funny and education at the same time. So, maybe [include] something like that.’ Inclusivity (9) ‘There's a lot more identities coming out now. So, just to be a little more inclusive, because a lot of people [are] like non-binary.’ Inclusivity (10) ‘I feel like you guys have tried to have graphics that are like representing diverse individuals. But I think you could just, like, make sure you’re looking at those again because like this page, it looks like a lot of Caucasian. Like five of the six people, at least from the view that I can see what Caucasian and teams. Want to see themselves represented … yeah but being sure to have like diverse body types and skin colors, etc … Not same sex couples, but I think all those are important considerations.’
Consent (Sexual Assault) (4) ‘Like, I think with a girl you could say … Like, I don't want you to do that or like, no you can't do that, sorry. Or don't even apologize because it's your body.
You could say something like, that's making me uncomfortable, can you stop that. I like you, but I don't want to do that.’		 Aesthetics and Images (11) ‘Don't use people pictures.’ Aesthetics and Images (12) ‘You know, thinking teenager mindset, “hokey” stock pictures are their least favorite in the world. There's no way to avoid that.’
    Personal Background and Use of Stories (13) ‘I think I know most about drugs because, my mom did drugs with my dad, so that's why I live with my grandparents. And I’ve seen what it does to you, so I stay away … I’ve seen what other kinds don't and they don't know what actually happens. And they can't feel what I’ve gone through … it's not real to them, they just don't understand … If they’re doing it, they aren't going to stop because we’ve already been taught not to do it. Unless something dramatic happens … I don't know maybe put in some scary pictures … what it looks like after you use this type of stuff. And what I think really gets to people are like having other people's stories be heard. Like in middle school we had this guy come in and he was telling us about his son and like I don't know – I feel like you can't really get personal with just some slides and stuff.’ Personal Background and Use of Stories (14) ‘Some of this content is like a lot of fact-based content. I don't know if there's any role of like … More of a like personal story or anecdote. Like that's good content, maybe just might feel a little more applicable if it wasn't Just numbers and facts, but like maybe what happened to someone who use this or something like that. So it felt like maybe a little bit more relatable.’
        Logistics and Information Sharing (15) ‘If you were to maybe expand into take-home because I do understand time is an issue so that would be very good. But if it was like a take-it-home and do it separately, I would probably include all of them [modules] just for educational purposes.’
    Parent/guardian-related Topics (16) ‘So having some important resources like this, we just don't want it to be something that's going to cause the person holding this tablet to like, click out of it or shut down and be like, “I don't want to see this” or even something that would make them uncomfortable with it if they’re in the waiting room … Especially like, if it's an underage person, their parent is going to probably be like, right there and watching … So that's also a big issue, especially for me. So like, if I was a kid and say, I’m like, 14, and my mom's right there, so many questions I’m not going to … I wouldn't answer honestly if my parents in, they are sitting right there.’ Parent/guardian-related Topics (17) ‘I think most of our parents will be pretty positive about it. I think a lot of my parents are not naive to think that their kids are exposed or engaging and some of these behaviors … We have a smaller section of patients and families that would be very uncomfortable. They’re very uncomfortable with just alone time with us in the room. And they get very nervous and want to know exactly what we’re talking about. So I think those families would have difficulty with it … But I think for the majority of our families … they would welcome any chance to educate their kids and try and save them from risky behavior.’
Open in a new tab

2.1.2. Alcohol & substance use

Participants also provided feedback on content-specific aspects of the intervention. Specific to substance use, participants liked the alcohol-related content and responded positively to brain-related substance use content (see Quote 2 in Table 4). Participants also suggested expanding information about substance use. Participants requested more content on the relation between substance use and the brain, wanted definitions or content to be clarified, and wanted more information on the consequences of substance use (see Table 4).

2.1.3. Sexual risk behaviours

Overall, participants had positive reactions to the sexual risk behaviour content. Participants suggested including content on the prevalence of STIs and the consequences of sexual risk behaviours without protection or with multiple partners. Participants reported that it would be useful to include a video on using sexual protection such as a condom properly (see Quote 3 in Table 4).

2.1.4. Sexual assault

Participants desired additional strategies to prevent sexual assault. Participants also suggested adding more scenarios from the perspectives of the victim, perpetrator, and bystander. Participants provided feedback on sections related to consent and wanted consent definitions with examples of consensual and non-consensual sexual activity. Participants also expressed the need for language around how to ask for consent, sexual refusal, and the importance of ongoing consent (see Quote 4 in Table 4).

2.2. Intervention refinement phase of Teen Well Check: adolescent and provider feedback

2.2.1. Engagement and interaction

Providers and adolescents indicated that Teen Well Check was engaging and interactive through its inclusion of videos, interactive questions and scenarios, and personalized tailoring to adolescents’ pre-survey answers and preferences (see Quote 5 in Table 4). Providers and teens universally had positive comments about including the short video developed by NIDA about brain anatomy, physiology, and effects of substance use (National Institute on Drug Abuse [NIDA/NIH], 2019).

Providers and adolescents wanted more interactive components of Teen Well Check. Specifically, responses indicated that they would like to see more interactive bystander scenarios (e.g. more ‘clicking,’ more videos, and more ‘quizzes’ to test their knowledge; see Quote 6 in Table 4).

2.2.2. Language and tone

Providers and adolescents emphasized the need for brevity, clarity, and not including too much information or text on the screen. Providers and adolescents both appreciated clear definitions and intentional diction. Generally, providers and adolescents agreed the app did and should continue to use a motivational interviewing approach rather than ‘scare tactics’ and should ‘meet people where they are’ rather than ‘being judgy’ or presenting ‘all or nothing’ scenarios (see Quotes 7 and 8 in Table 4).

2.2.3. Inclusivity

Providers and adolescents encouraged more inclusivity throughout the Teen Well Check app, particularly for trans, non-binary, or those with gender non-conforming identities, as well as inclusion of racial and ethnic minorities. Participants suggested the inclusion of same-sex couples, images of non-binary people, and avoiding binary language such as ‘boy–girl’ or ‘he-she.’ Some providers and adolescents pointed out that the sexual assault scenarios all positioned boys as perpetrators and girls as victims and suggested having scenarios with boys as victims (see Quotes 9 and 10 in Table 4).

2.2.4. Aesthetics and images

Providers and adolescents mostly disliked the stock photos used throughout Teen Well Check, preferring the comic book style images used within the sexual assault module. One participant noted, ‘The circle full of teenagers, it just— it's like a little cliché,’ as did another, stating, ‘Don't use people pictures.’ In fact, some providers predicted that adolescents would find the photos of teens to be ‘hokey’ while also suggesting adolescents would respond well to the comic style (see Quotes 11 and 12 in Table 4). Providers liked Teen Well Check's bright colours, and most adolescents said they enjoy the ‘very bright’ colours ‘which really attracts people's attention’. They noted that they wanted the programme to be ‘aesthetically pleasing’ and ‘cute’.

2.2.5. Personal background and the use of stories

Several providers and adolescents mentioned the importance of using stories and anecdotes to help users understand and engage more deeply with the material (see Quotes 13 and 14 in Table 4).

2.2.6. Logistics and information sharing

Providers expressed concerns about how the 15-minute Teen Well Check intervention would be delivered in the waiting room, including lack of sufficient time and privacy (e.g. from guardians and other patients), difficulty ‘keeping a teenager's focus’ in the context of the pandemic clinic flow. Several participants suggested the intervention should be made available online, allowing participants to complete in their own time from a convenient and private location (see Quote 15 in Table 4). Providers also emphasized the importance of sharing resources with adolescents, connecting them to necessary referrals, and sharing information with the parents to offer appropriate care.

2.2.7. Provider impressions of parent/guardian-related topics

When researchers asked how guardians would respond to Teen Well Check, providers described a variety of expected responses, anticipating that some guardians would be supportive, with others being uncomfortable, particularly with ‘the sex part.’ Providers emphasized they ask their adolescent patients drug, alcohol, and sex questions, but it is done during the confidential history portion of the exam, as allowed by state laws. Adolescents similarly emphasized the importance of resources like Teen Well Check but expressed concerns about confidentiality and privacy (see Table 4).

2.3. Acceptability and usability: adolescents and providers

Most providers were very interested in using a prevention programme like Teen Well Check in their practice, especially if the necessary technology is provided. This high level of interest was reflected in our survey data (see Table 5). When asked how likely they would be to (1) use this intervention and (2) recommend it to 14–18-year-olds, average provider responses indicated that providers would be somewhat likely to use the intervention in their clinical practice and recommend that adolescents use it. Providers were somewhat concerned about using a tablet-based intervention in the waiting room given the new clinic flow due to the pandemic, and recommended an online format. Therefore, the final version of Teen Well Check was adapted to an online format to be used at the clinic or after teens’ appointments based on clinic flow.

Table 5.

Usability and acceptability of the Teen Well Check tablet-based intervention among primary care providers (n = 11) of adolescents 14–18 years in the Intervention Refinement process (n = 10).

Survey Item Provider Average Score; Mean (SD) Adolescent Average Score; Mean (SD) Scale
Post-Study System Usability Questionnaire 1.84 (1.10) 1.24 (0.35) 1=Strongly Agree (Usable) 5=Strongly Disagree (Not Usable)
How likely would you be to use this tablet-based intervention in your clinical practice? 5.09 (1.76)   1=Very Unlikely 7 =Very Likely
How likely would you be to recommend that teens aged 14–18 use this tablet-based intervention? 5.36 (1.50)   1=Very Unlikely 7 =Very Likely
Open in a new tab

In the interviews, most providers indicated that Teen Well Check was appropriate for 14–18-year-olds. Other providers felt the topics needed to be addressed at a younger age (e.g. 12 years old). Providers and adolescents ‘strongly agreed’ or ‘agreed’ that Teen Well Check was highly usable as assessed by the PSSUQ (Mprovider = 1.84; Madolescents = 1.24; see Table 5).

3. Discussion

The current study assessed the usability and acceptability of Teen Well Check, an e-Health prevention programme targeting substance use, sexual assault, and sexual risk among teens in primary care settings, which provides an ideal window of opportunity for identification and prevention education for adolescents. Though the scientific premise for integrating prevention draws from previous work which outlines the interrelated nature of these health risk behaviours (George & Stoner, 2000; Hendershot & George, 2007; Testa & Livingston, 2009), no preventive work has been conducted to date targeting and integrating these three co-occurring risk behaviours among adolescents. Overall, teens and providers found the programme to be acceptable and useable, and providers indicated that they would be somewhat likely to use it in their clinical practice and recommend to adolescents. Teens and providers gave suggestions throughout the intervention's development, which were integrated into the final version of the programme.

When assessing feasibility of implementing Teen Well Check within a primary care setting, providers indicated that changes in clinic flow due to the direct result of the COVID-19 pandemic, a tablet-based programme delivered and completed in clinic waiting rooms may not be feasible. Therefore, the final version was accessible remotely via a personal device to maximize the connectivity of the prevention programme to the healthcare setting, given that is where adolescents learn about the programme, without taking away from routine healthcare. This approach could eventually allow for a very scalable implementation strategy if the programme is found to be efficacious because it can be made broadly accessible nationwide rather than based in specific clinics.

Adolescent participants indicated that improvements in usability and engagement may be achieved through gamifying Teen Well Check. Although this approach to intervention programming would be innovative and potentially useful, it was not feasible to use gamification due to development budget limitations during this study. Future research should consider including gamification in prevention programmes, as well as videos, comics, or quotes to integrate personal stories to enhance prevention, a strategy participant in the present study also suggested.

3.1. Limitations

To protect anonymity, the current interface does not integrate with medical records; however, to implement with pediatric primary care clinics nationwide, it may be useful for clinic buy-in to create a provider dashboard and to integrate with an electronic medical record. This would require significant funds due to the complexity of software needed to integrate with electronic medical record systems. Further, participant interactions with the programme were not recorded due to programming limitations in initial versions, which was addressed in the final version of the programme. A feasibility trial is needed to understand other field notes within one or more primary care settings to learn more about use, process, and how the providers would interface with the programme. If Teen Well Check is found to be feasible and efficacious, there is potential to expand this intervention to be tailored for specific groups of adolescents including tailoring based on race/ethnicity, sexual orientation, and age. Given that this is a prevention programme, it may be possible to tailor to a younger age as well.

4. Conclusions

The current study presents findings about the development and acceptability of Teen Well Check. Future research is required to understand if Teen Well Check effectively prevents substance use, sexual assault, and sexual risk among teens. Overall, Teen Well Check was acceptable to teens and providers, and prevention programmes targeting substance use, sexual assault, and sexual risk may be useful within primary care settings. More work is needed to assess the implementation feasibility and implementation barriers of Teen Well Check. Roll outs of clinical interventions in primary care settings can be a feasible way to address public health concerns.

Acknowledgements

We would like to acknowledge and thank all participants in this research study as well as Dr. Hollie Granato for the initial development of the sexual assault scenarios and Bea Stephens for the comic design.

Funding Statement

Data collection and manuscript preparation was supported by the National Institute on Drug Abuse (K23DA042935; PI: Gilmore). Manuscript preparation also was supported in part by K24DA039783 (PI: Danielson) and the National Institute of Child Health and Human Development (K23HD098325; PI: Ridings).

Disclosure statement

Dr. Levy serves as an expert consultant for the case against JUUL. All other authors have no potential conflicts of interest to report.

Data availability statement

Due to the nature of this research, participants of this study did not agree for their data to be shared publicly, so supporting data is not available.

References

  1. Bartholomew Eldredge, L. K., Markham, C. M., Ruiter, R. A. C., Fernández, M. E., Kok, G., & Parcel, G. S. (2016). Planning health promotion programs: An intervention mapping approach. Jossey-Bass. [Google Scholar]
  2. Black, M. C., Basile, K. C., Breiding, M. J., Smith, S. G., Walters, M. L., Merrick, M. T., Chen, J., & Stevens, M. R. (2011). The National Intimate Partner and Sexual Violence Survey (NISVS): 2010 summary report. National Center for Injury Prevention and Control, Centers for Disease Control and Prevention. Available from https://www.cdc.gov/violenceprevention/pdf/nisvs_report2010-a.pdf [Google Scholar]
  3. Borsari, B., & Carey, K. B. (2001). Peer influences on college drinking: A review of the research. Journal of Substance Abuse, 13(4), 391–424. 10.1016/s0899-3289(01)00098-0 [DOI] [PubMed] [Google Scholar]
  4. Bronfenbrenner, U. (1979). The ecology of human development: Experiments by nature and design. Harvard University Press. [Google Scholar]
  5. Champion, K. E., Paramenter, B., McGowan, C., Spring, B., Wafford, Q. E., Gardner, L. A., Thornton, L., McBride, N., Barrett, E. L., Teesson, M., Newton, N. C., Chapman, C., Slade, T., Sunderland, M., Bauer, J., Allsop, S., Hides, L., Stapinksi, L., Birrell, L., & Mewton, L. (2019). Effectiveness of school-based eHealth interventions to prevent multiple lifestyle risk behaviours among adolescents: A systematic review and meta-analysis. The Lancet Digital Health, 1(5), e206–e221. 10.1016/S2589-7500(19)30088-3 [DOI] [PubMed] [Google Scholar]
  6. Collins, R. L., Parks, G. A., & Marlatt, G. A. (1985). Social determinants of alcohol consumption: The effects of social interaction and model status on the self-administration of alcohol. Journal of Consulting and Clinical Psychology, 53(2), 189–200. 10.1037//0022-006x.53.2.189 [DOI] [PubMed] [Google Scholar]
  7. Dimeff, L. A., Baer, J. S., Kivlahan, D. R., & Marlatt, G. A. (1999). Brief alcohol screening and intervention for college students: A harm reduction approach. Guilford Press. [Google Scholar]
  8. Dumas, T. M., Ellis, W., & Litt, D. M. (2020). What does adolescent substance use look like during the COVID-19 pandemic? Examining changes in frequency, social contexts, and pandemic-related predictors. Journal of Adolescent Health, 67(3), 354–361. 10.1016/j.jadohealth.2020.06.018 [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Fischer, P., Krueger, J. I., Greitemeyer, T., Vogrincic, C., Kastenmüller, A., Frey, D., Heene, M., Wicher, M., & Kainbacher, M. (2011). The bystander-effect: A meta-analytic review on bystander intervention in dangerous and non-dangerous emergencies. Psychological Bulletin, 137(4), 517–537. 10.1037/a0023304 [DOI] [PubMed] [Google Scholar]
  10. George, W. H., & Stoner, S. A. (2000). Understanding acute alcohol effects on sexual behavior. Annual Review of Sex Research, 11, 92–124. 10.1080/10532528.2000.10559785 [DOI] [PubMed] [Google Scholar]
  11. Gibson, E. B., Knight, J. R., Levinson, J. A., Sherritt, L., & Harris, S. K. (2021). Pediatric primary care provider perspectives on a computer-facilitated screening and brief intervention system for adolescent substance use. Journal of Adolescent Health, 69(1), 157–161. 10.1016/j.jadohealth.2020.09.037 [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Gidycz, C. A., McNamara, J. R., & Edwards, K. M. (2006). Women’s risk perception and sexual victimization: A review of the literature. Aggression and Violent Behavior, 11(5), 441–456. 10.1016/j.avb.2006.01.004 [DOI] [Google Scholar]
  13. Gilmore, A. K., Leone, R. M., Oesterle, D., Davis, K. C., Orchowski, L. M., Ramakrishnan, V., & Kaysen, D. (2022). Web-based alcohol and sexual assault prevention program with tailored content based on gender and sexual orientation: Preliminary outcomes and usability of Positive Change (+change). JMIR Formative Research, 6(7), e23823. 10.2196/23823 [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Hendershot, C. S., & George, W. H. (2007). Alcohol and sexuality research in the AIDS era: Trends in publication activity, target populations and research design. AIDS and Behavior, 11(2), 217–226. 10.1007/s10461-006-9130-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Hendershot, C. S., Magnan, R. E., & Bryan, A. D. (2010). Associations of marijuana use and sex-related marijuana expectancies with HIV/STD risk behavior in high-risk adolescents. Psychology of Addictive Behaviors, 24(3), 404–414. 10.1037/a0019844 [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Hofer, A. N., Abraham, J. M., & Moscovice, I. (2011). Expansion of coverage under the Patient Protection and Affordable Care Act and primary care utilization. Milbank Quarterly, 89(1), 69–89. 10.1111/j.1468-0009.2011.00620.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Hops, H., Ozechowski, T. J., Waldron, H. B., Davis, B., Turner, C. W., Brody, J. L., & Barrera, M. (2011). Adolescent health-risk sexual behaviors: Effects of a drug abuse intervention. AIDS and Behavior, 15(8), 1664–1676. 10.1007/s10461-011-0019-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Hoxmeier, J. C., Flay, B. R., & Acock, A. C. (2018). Control, norms, and attitudes: Differences between students who Do and do not intervene as bystanders to sexual assault. Journal of Interpersonal Violence, 33(15), 2379–2401. 10.1177/0886260515625503 [DOI] [PubMed] [Google Scholar]
  19. Johnston, L. D., Miech, R. A., O’Malley, P. M., Bachman, J. G., Schulenberg, J. E., & Patrick, M. E. (2021). Monitoring the future national survey results on drug use 1975–2020: Overview, key findings on adolescent drug use. University of Michigan, Institute for Social Research. Available from: https://files.eric.ed.gov/fulltext/ED611736.pdf [Google Scholar]
  20. Johnston, L. D., O’Malley, P. M., Miech, R. A., Bachman, J. G., & Schulenberg, J. E. (2016). Monitoring the future national survey results on drug use, 1975–2015: Overview, key findings on adolescent drug use. University of Michigan, Institute for Social Research. https://files.eric.ed.gov/fulltext/ED578539.pdf [Google Scholar]
  21. Jouriles, E. N., Krauss, A., Vu, N. L., Banyard, V. L., & McDonald, R. (2018). Bystander programs addressing sexual violence on college campuses: A systematic review and meta-analysis of program outcomes and delivery methods. Journal of American College Health, 66(6), 457–466. 10.1080/07448481.2018.1431906 [DOI] [PubMed] [Google Scholar]
  22. Kantawong, E., Kao, T. A., Robbins, L. B., Ling, J., & Anderson-Carpenter, K. D. (2021). Adolescents’ perceived drinking norms toward alcohol misuse: An integrative review. Western Journal of Nursing Research, 44(5), 477–492. 10.1177/0193945921998376 [DOI] [PubMed] [Google Scholar]
  23. Kreisel, K. M., Spicknall, I. H., Gargano, J. W., Lewis, F. M. T., Lewis, R. M., Markowitz, L. E., Roberts, H., Johnson, A. S., Song, R., St. Cyr, S. B., Weston, E. J., Torrone, E. A., & Weinstock, H. S. (2021). Sexually transmitted infections among US women and men: Prevalence and incidence estimates, 2018. Sexually Transmitted Diseases, 48(4), 208–214. 10.1097/OLQ.0000000000001355 [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Labhardt, D., Holdsworth, E., Brown, S., & Howat, D. (2017). You see but you do not observe: A review of bystander intervention and sexual assault on university campuses. Aggression and Violent Behavior, 35, 13–25. 10.1016/j.avb.2017.05.005 [DOI] [Google Scholar]
  25. Latané, B., & Darley, J. (1970). The unresponsive bystander: Why doesn’t he help? Appleton-Century Crofts. [Google Scholar]
  26. Latané, B., & Nida, S. (1981). Ten years of research on group size and helping. Psychological Bulletin, 89(2), 308–324. 10.1037/0033-2909.89.2.308 [DOI] [Google Scholar]
  27. Latané, B., & Rodin, J. (1969). A lady in distress: Inhibiting effects of friends and strangers on bystander intervention. Journal of Experimental Social Psychology, 5(2), 189–202. 10.1016/0022-1031(69)90046-8 [DOI] [Google Scholar]
  28. Lewis, J. R. (2002). Psychometric evaluation of the PSSUQ using data from five years of usability studies. International Journal of Human-Computer Interaction, 14(3-4), 463–488. 10.1080/10447318.2002.9669130 [DOI] [Google Scholar]
  29. Lewis, M. A., Litt, D., Cronce, J., Blayney, J. A., & Gilmore, A. K. (2014). Underestimating protection and overestimating risk: Examining descriptive normative perceptions and their association with drinking and sexual behaviors. The Journal of Sex Research, 51(1), 86–96. 10.1080/00224499.2012.710664 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Lewis, M. A., & Neighbors, C. (2004). Gender-specific misperceptions of college student drinking norms. Psychology of Addictive Behaviors, 18(4), 334–339. 10.1037/0893-164X.18.4.334 [DOI] [PubMed] [Google Scholar]
  31. Masonbrink, A. R., Middlebrooks, L., Gooding, H. C., Abella, M., Hall, M., Burger, R. K., & Goyal, M. K. (2022). Substance use disorder visits among adolescents at children’s hospitals during COVID-19. Journal of Adolescent Health, 70(4), 673–676. 10.1016/j.jadohealth.2021.12.024 [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Naar-King, S., & Suarez, M. (2011). Motivational interviewing with adolescents and young adults. Guilford Press. [Google Scholar]
  33. National Institute on Alcohol Abuse and Alcoholism . (2011). Alcohol screening and brief intervention: A practitioner’s guide. http://pubs.niaaa.nih.gov/publications/Practitioner/YouthGuide/YouthGuide.pdf
  34. National Institute on Drug Abuse (NIDA/NIH) . (2019). Teen Brain Development [video on the Internet]. https://www.youtube.com/watch?v=EpfnDijz2d8
  35. Noll, J. G., Guastaferro, K., Beal, S. J., Schreier, H. M. C., Barnes, J., Reader, J. M., & Font, S. A. (2019). Is sexual abuse a unique predictor of sexual risk behaviors, pregnancy, and motherhood in adolescence? Journal of Research on Adolescence, 29(4), 967–983. 10.1111/jora.12436 [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Nurius, P. S., & Norris, J. (1996). A cognitive ecological model of women’s response to male sexual coercion in dating. Journal of Psychology & Human Sexuality, 8(1-2), 117–139. 10.1300/J056v08n01_09 [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Pedersen, E. R., Osilla, K. C., Miles, J. N., Tucker, J. S., Ewing, B. A., Shih, R. A., & D’Amico, E. J. (2017). The role of perceived injunctive alcohol norms in adolescent drinking behavior. Addictive Behaviors, 67, 1–7. 10.1016/j.addbeh.2016.11.022 [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Perkins, H. W. (2002). Social norms and the prevention of alcohol misuse in collegiate contexts. Journal of Studies on Alcohol, Supplement, 14(s14), 164–172. 10.15288/jsas.2002.s14.164 [DOI] [PubMed] [Google Scholar]
  39. QSR International Pty Ltd . (2018). NVivo (Version 12). https://www.qsrinternational.com/nvivo-qualitative-data-analysis-software/home
  40. Scott-Sheldon, L. A., Carey, K. B., Cunningham, K., Johnson, B. T., Carey, M. P., & The MASH Research Team . (2016). Alcohol use predicts sexual decision-making: A systematic review and meta-analysis of the experimental literature. AIDS and Behavior, 20(Suppl 1(0-1)), 19–39. 10.1007/s10461-015-1108-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Skinner, B. F. (1958). Reinforcement today. American Psychologist, 13(3), 94–99. 10.1037/h0049039 [DOI] [Google Scholar]
  42. Smith, S. G., Zhang, X., Basile, K. C., Merrick, M.T., Wang, J., Kresnow, M., & Chen, J. (2018). The National Intimate Partner and Sexual Violence Survey (NISVS): 2015 Data brief – Updated release. National Center for Injury Prevention and Control, Centers for Disease Control and Prevention. https://www.cdc.gov/violenceprevention/pdf/2015data-brief508.pdf [Google Scholar]
  43. Steele, C. S., & Josephs, R. A. (1990). Alcohol myopia: Its prized and dangerous effects. American Psychologist, 45(8), 921–933. 10.1037/0003-066X.45.8.921 [DOI] [PubMed] [Google Scholar]
  44. Testa, M., & Livingston, J. A. (2009). Alcohol consumption and women’s vulnerability to sexual victimization: Can reducing women’s drinking prevent rape? Substance Use & Misuse, 44(9-10), 1349–1376. 10.1080/10826080902961468 [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Ullman, S. E. (2014). Reflections on researching rape resistance. Violence Against Women, 20(3), 343–350. 10.1177/1077801214526308 [DOI] [PubMed] [Google Scholar]
  46. White, K., Narasimhan, S. A., Hartwig, S., Carroll, E., McBrayer, A., Hubbard, S., Rebouché, R., Kottke, M., & Hall, K. S. (2022). Parental involvement policies for minors seeking abortion in the southeast and quality of care. Sexuality Research and Social Policy, 19(1), 264–272. 10.1007/s13178-021-00539-0 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

Due to the nature of this research, participants of this study did not agree for their data to be shared publicly, so supporting data is not available.

Articles from European Journal of Psychotraumatology are provided here courtesy of Taylor & Francis

RESOURCES