Using Cognitive Interviews to Adapt Interpersonal Violence Measures for Use With Middle School Youth

Abstract

Experiences of interpersonal violence are common among youth. Starting prevention programming early (e.g., middle school) may be beneficial for primary prevention. Evaluating whether such programs are effective often requires collecting self-report data from youth, but many existing measures have been developed for high school and college-aged youth. This study aimed to assess adolescents’ comprehension of self-report survey items on interpersonal violence with middle school youth. We conducted virtual cognitive interviews with 15 youth in grades 6 to 8. A content analysis was used to identify patterns and to classify the nature and type of comprehension issues youth experienced. Nearly all students found most questions clear and understandable. We identified the following comprehension issues: (1) uncertainty with how the intent of a perpetrator factored into a victim’s experience (e.g., distinguishing the difference between joking and bullying, or intentional versus unintentional behavior); (2) lack of familiarity with certain expressions of sexualized violence (e.g., “sexual looks”) or sex-related terminology (e.g., intercourse); and (3) narrow interpretations of question prompts (e.g., interpreting “forced” as physically forced, not psychologically coerced). Students suggested including language describing dating relationships, types of social media platforms where cyber abuse takes place, and additional examples alongside items to enhance relevance and clarity. Survey questions to measure interpersonal violence may need to be adapted for use among middle school youth. Our findings highlight potential considerations for improving the measurement of interpersonal violence in this age group.

Introduction

Experiences of interpersonal violence (e.g., bullying, cyber abuse, sexual violence, and dating violence) are common among adolescents. In the United States, nationally representative data from 2019 showed that, in the past year, 22% of students reported being bullied (being teased, threatened, spread rumors about, hit, shoved, or hurt on school property or electronically), 11% reported sexual violence by anyone (being forced to do sexual things they did not want to), and 9% of students who dated in the past year reported physical dating violence (being physically hurt on purpose by a dating partner) (Clayton et al., 2023). Experiences of violence are associated with adverse consequences, such as depression, substance use, sexual risk behaviors, and suicide (Exner-Cortens et al., 2013).

School-based policies and programs are an important way to address interpersonal violence (Basile et al., 2016; De La Rue et al., 2017; Gaffney et al., 2019). There has been increasing recognition that addressing violence earlier in the life course may be beneficial for primary prevention. Several programs have been developed for use in middle schools to build foundational skills before youth begin dating and/or experiencing severe violence, with the potential to reduce longer-term outcomes like adult intimate partner violence (Tharp, 2012). Relatedly, substantial evidence supports sex education that scaffolds learning in a developmentally appropriate way, beginning as early as elementary school (Goldfarb & Lieberman, 2021). Middle school programs, including Second Step, Shifting Boundaries, Start Strong, Dating Matters, Fourth R, and Coaching Boys Into Men, have indeed shown effectiveness in reducing cyber abuse, dating violence, sexual violence, and sexual harassment (DeGue et al., 2021; Espelage et al., 2015; E. Miller et al., 2020; S. Miller et al., 2015; Niolon et al., 2019; Taylor et al., 2013; Temple et al., 2021).

Understanding the effectiveness of these programs requires collecting self-report data from students via surveys. However, many common measures to assess interpersonal violence have been developed for high school and college-aged youth and have not been validated for middle school youth (American Association of University Women, 2011; Basile et al., 2020; Wolfe et al., 2001). There is a need to understand if interpersonal violence measures are understandable and clear to younger adolescents, particularly as primary prevention efforts shift to middle school grade levels. Our study aimed to assess adolescents’ comprehension of existing self-report surveys on interpersonal violence (i.e., cyber abuse, sexual violence, and dating violence) through cognitive interviews with middle school youth. We provide considerations designed to ensure the acceptability and appropriateness of survey questions in this age group.

Methods

Sample

The evaluation team conducted a series of individual interviews with 15 middle school-aged youth in 6th through 8th grades. Participants were drawn from one large, urban area through the professional and personal networks of the research team. Parents were contacted with initial information about the study, and if interested, asked to sign an electronic parental consent form. In total, we contacted 18 potential interview participants. Two did not respond to our communications and one declined to participate, for a total of 15 assenting middle school youth.

Data Collection

All procedures were approved by the University of Washington Institutional Review Board and Office of the Youth Protection Coordinator, which develops guidelines, policies, and best practices to promote safety for youth engaged in research. Interviews were conducted over Zoom and lasted approximately 1 hour. Two research team members attended each interview, with one as the lead interviewer while the second team member took notes and listened or added follow-up clarifying questions. Interviews were recorded with parent and youth permission. Participating youth were provided information about the study purpose, example questions, voluntary nature of the interview, and national resources (e.g., sexual assault hotline). Youth verbally assented prior to starting the interview and were provided a $25 Tango card (where they could choose a gift card from a number of online stores) for participating. Demographic data on sex, grade, age, and race/ethnicity were collected from each participant’s consenting parent and validated with the participant. The interview began with a “thinking aloud” exercise to get the youth comfortable describing their thoughts out loud.

We designed the interview guide to assess student understanding of key interpersonal violence questions. We shared specific survey questions and asked the youth: “Please explain what this question means to you, and if it could be worded differently so it is easy for other students to answer.” We had additional probes such as if they would be able to answer the question on a survey or if items were unnecessary. For the first set of items, we also provided two sets of response options related to frequency (one included specific numbers of times something occurred and one had more general frequency categories) and asked which they preferred. The survey questions were sourced and adapted from prior studies and program evaluations for interpersonal violence and previously used in studies with middle school youth (e.g., American Association of University Women, 2011; Miller et al., 2020; Straus et al., 1996; Vivolo-Kantor et al., 2021). Exact questions are provided in Supplemental Table 1. We showed youth survey questions and response options on PowerPoint slides (Supplemental Figure 1) as they would appear on an actual survey by sharing our screen via Zoom. This gave youth an auditory and visual experience for providing reactions and feedback.

Analysis

We used content analysis to classify the type of comprehension issues youth experienced (Hsieh & Shannon, 2005). Two researchers independently coded interview transcripts, coding excerpts according to the type of reported comprehension (clear or unclear), and suggestions for question improvement or clarity. Codes and coding were refined during regular debriefing meetings with the research team. During these meetings, we determined the appropriate end of recruitment by consensus when we reached saturation (i.e., no new themes were identified; Hennink & Kaiser, 2022). After coding, we used an iterative, consensus-based approach to develop themes and interpret findings. We used Dedoose qualitative analysis software for coding and analysis and employed multiple strategies to ensure trustworthiness, including multiple independent coders, consensus-based theme development, iterating between analysis and data collection, and audit trails about the codebook, analysis, and decision points (Nowell et al., 2017).

Results

In our sample of 15 participants, 46.7% (n = 7) identified as female. Approximately half (46.7%, n = 7) were enrolled in 7th grade, 33.3% (n = 5) in 8th grade, and 20.0% (n = 3) in 6th grade (all between ages 12 and 14). Our sample largely identified as White (66.7%, n = 10), with two participants identifying as Asian or Asian American and one as Black or African American.

Themes

Most questions were easily comprehended and perceived as relevant

Most questions were clear and understandable across grade levels, including the most sensitive questions about sexual violence victimization and perpetration. All but two participants expressed a preference for the response options that provided specific numbers when answering questions about the frequency of an event (e.g., A few times [1–3 times], Sometimes [4–9 times], Often [10+ times] as in CA-V #1 vs. options without specific numbers as in CA-V #2 [Supplemental Table 1]).

Participants across grade levels understood the survey questions presented and affirmed their relevance. Participants were able to successfully call several examples to mind for how each behavior might be demonstrated. For example, when asked about receiving a sexual picture when they did not want to (CA-V #1e), participants explained that this might happen through AirDrop or direct messages on social media. When asked about their understanding of the question about being asked to do something sexual online that they did not want to do (CA-V #1f), one participant expressed an example: “At first, I was like, wait, how did that happen, and then I thought of like if they asked you to send them photos or videos online or like on FaceTime or something. Or I guess since you also said social media like Instagram or TikTok, it could be like ask[ing] you to post something, but I don’t think I’ve really ever heard of that” (participant 12, 6th-grade female).

Participants understood questions about seeing someone their age being pressured into doing sexual things (SV-V #1) or someone they were in a relationship with making them do sexual things they did not want to (DV-V #2c). Participants found including a definition of “sex” (“Sex includes oral, vaginal, and/or anal sex”) helpful for their understanding. One student said, “I feel like it is [helpful] just because I feel like certain people think certain things or, just from like conception or misconceptions of what sex is” (participant 7, 7th-grade female). Similarly, participants found having examples listed helpful because individuals may interpret “sexual things” to have different meanings. Younger participants, including those in sixth grade, particularly shared this view. One participant shared, “With kissing to intercourse, those are pretty different. Someone might not take being forced to kiss, compared to intercourse, that seriously. They might not write it down as an answer, but since you’re [listing] all of it, it’s easier to understand” (participant 4, 8th-grade male).

About half of the participants experienced at least one comprehension challenge or suggested an area for improvement. The identified comprehension issues included the following: (1) uncertainty regarding the relevance of perpetrator intent, (2) lack of familiarity with certain expressions of sexualized violence or sex-related terminology, and (3) narrow interpretations of question prompts (Supplemental Table 2).

Uncertainty regarding the relevance of perpetrator intent

A handful of participants expressed uncertainty regarding whether to “count” an experience of violence if the perpetrator committed it “by accident” or did not intend to cause harm. One participant (participant 3, 7th-grade male) noted they would prefer another response option to indicate having perpetrated a certain act and regretting it later, rather than grouping this experience with intentional perpetration. Some participants also shared that they might not consider an act that was perpetrated in jest in their response. For example, referring to “destroyed something that belonged to me” (DV-V #2d), one participant stated that they might not take this seriously, depending on the perpetrator: “This just makes me think of my brother. I just think of any time where someone takes your thing and they don’t give it back to you. This seems more jokingly. I don’t really take anything seriously. I don’t really know anyone that takes your stuff and then chooses to break it, just to bother you” (participant 4, 8th-grade male).

One participant (participant 12, 6th-grade female) shared they were unsure of how to respond to a question about whether they sent a sexual picture when someone did not want to receive it (CA-V #1e). They asked how they should respond if they were unsure of whether the recipient wanted to receive a sexual picture. Some also expressed that it might be difficult to tell whether a comment might be considered a compliment. One participant said, “When I think of that, I think about two different things, whether if it’s a compliment or an insult. I would be confused about what to put down. I’d probably just end up putting if it was an insulting joke” (participant 4, 8th-grade male).

Lack of familiarity with certain expressions of sexualized violence or sex-related terminology

Two participants (participant 1, 6th-grade male and participant 6, 7th-grade female) shared that they were not familiar with the word intercourse, which contributed to their hesitation regarding how to answer the question prompt. A few participants had trouble identifying examples of “unwelcome sexual comments, jokes, gestures or looks” (SV-V #2a)—particularly sexual gestures and looks. Some, however, articulated that even when they could not identify a clear example of a behavior, they could identify when they were experiencing a feeling of discomfort in response to a look or gesture. One participant said, “I’m thinking about like at school or on the bus because, on the bus, it’s not like one class. . .it’s not one grade. There are sixth graders, but there’s also eighth graders on the bus and they might be saying something that you don’t know, like something sexual, and that might make me feel really uncomfortable. That’s what this question makes me think about, especially on the bus. . . I realized a lot of the older kids, they’re talking about this stuff and I don’t really understand all of it” (participant 1, 6th-grade male).

Narrow interpretations of question prompts

Some participants explained that they conceptualized certain parameters around some questions. For example, a few participants understood “spread sexual rumors” (SV-V #2e) to refer to spreading lies about someone’s sexual history, while others interpreted this as spreading true but private information. One participant shared that “forced you to do something sexual” (SV-V #2f) should be clarified to articulate whether someone experienced physical or psychological force: “I guess my only kind of, I want to say objection, but I’m not really objecting to anything is like forced . . . What if you were coerced? I don’t know if that would specifically fit under forced. I feel like maybe pressured, in a sense, maybe would be better, because I think pressured might fit forced and coerced and anything in between” (participant 7, 7th-grade female). Another participant (participant 8, 6th-grade male) interpreted the question about seeing someone their age being pressured into doing sexual things (SV-V #1) as referring only to instances someone witnessed with their own eyes. The participant wondered whether to “count” instances they had only heard about.

Youth suggestions for improving scale items

Many students expressed familiarity with and understanding of violence concepts and terminology and offered suggestions to enhance relevance and clarity for their peers (Supplemental Table 3). Suggestions included language describing dating relationships (e.g., clarifying when a relationship is a dating or romantic relationship, as “relationship” on its own was perceived as also including platonic friendships), expanding social media platforms where cyber abuse occurs (e.g., Instagram, Snapchat, Discord, Twitch), and including additional examples alongside items (e.g., “something sexual”). Some also expressed interest in adding gender-neutral terms (e.g., partner) to the definition of dating relationships, as well as language that reflects more casual forms of dating (e.g., going out without being supervised, someone they are a “thing” with) rather than formally dating.

Discussion

These cognitive interviews highlight that our sample of middle school youth found questions about interpersonal violence, including cyber abuse, sexual violence, and dating violence, broadly understandable and relevant. Violence prevention programs are increasingly being implemented in middle school settings, and evaluating these programs is critically important to understanding if and how these programs are effective (E. Miller et al., 2020; Vivolo-Kantor et al., 2021). Because of the low rate of formal reporting of many types of violence (Bundock et al., 2020), self-report surveys are the optimal way to assess experiences of violence and thus evaluate programs. While school administrators, teachers, and parents may be hesitant to ask sensitive questions about violence to youth, our findings underscore that questions can be adapted to be understood by those as young as 6th grade. In large-scale surveys, questions about sexual or dating violence are often not included for middle school students. For example, the CDC’s Middle School Youth Risk Behavior Survey does not include those questions (which are included for high school students), and Washington State’s Healthy Youth Survey includes only limited questions for 8th-grade students and none for 6th-grade students (CDC, 2023; Healthy Youth Survey: Form B, 2021). However, youth are having these experiences early in middle school, and knowing the extent can help schools allocate resources to address these incidents (Espelage et al., 2015).

Our findings point to several recommendations and strategies, some suggested by the youth themselves, to ensure questions are understandable. One frequent suggestion was to add examples and definitions to make it clear what behaviors were included. For example, youth suggested that “doing something sexual” was somewhat vague so to add or retain specifics (e.g., “like kissing or sexual touch”). In addition, being inclusive of the ways that violence can occur and the means used to perpetrate violence was important. For example, when asking about violence occurring online, youth suggested having a long list of what that might mean including cell phones, e-mails, text messages, or social media like Instagram, TikTok, Snapchat, Discord, and Twitch. Survey questions would also benefit from clarity on intention and impact since some participants reported being unsure whether to count an experience if the perpetrator did not intend to harm. Indeed, prior research has examined how the measurement of violence may be affected by including behaviors that are roughhousing, joking, or being playful (Sargent et al., 2020). Relatedly, ensuring that questions are interpreted consistently across participants is important. Not surprisingly, 6th-grade youth tended to have more comprehension difficulties. For these youth, including more examples and definitions when using violence or sex-related terminology (e.g., intercourse) was found to be helpful. Understanding what topics youth have already been exposed to may be helpful in identifying if there are survey items that cover concepts they have not yet learned.

While we have identified several strategies for adapting questions to alleviate comprehension challenges, further research is needed to assess the validity and reliability of these revised measures. In general, the development and testing of measures, especially among younger-aged youth, is an important goal for the field (Exner-Cortens et al., 2016; Smith et al., 2015). Recently, a new measure for adolescent dating violence victimization and perpetration (the MARSHA) was developed through comprehensive formative research and psychometric testing with a nationally representative sample (Rothman et al., 2022). Similar processes for developing contemporary measures for sexual and other forms of violence among youth are warranted, especially as forms of violence shift over time (e.g., with technology).

We interviewed only a small number of English-speaking youth in one large urban area drawn from the networks of the research team. The use of these measures in more diverse populations requires additional cognitive testing with youth of diverse racial, ethnic, socioeconomic, gender, and sexual orientation identities and from suburban and rural locations. There are other measures related to interpersonal violence that may benefit from cognitive testing with middle school students, such as measures of gender-equitable attitudes or recognition of abusive behavior (Hill et al., 2022; E. Miller et al., 2020). Overall, we have shown that youth as young as 6th grade are able to answer questions about experiences of interpersonal violence. While there were some comprehension and interpretation issues, there are ways to modify and clarify questions. Our results underscore the importance of cognitive interviewing for adapting survey items and can guide future development and psychometric testing of interpersonal violence measures for middle school youth.

Author Biographies

Avanti Adhia, ScD, is an assistant professor in the University of Washington Department of Child, Family, and Population Health Nursing. The goal of her research is to prevent intimate partner and sexual violence, primarily among adolescents and young adults, by understanding the causes and consequences in addition to the role of policies and interventions in preventing and reducing violence.

Nicole Casanova, MPH, is an adolescent health performance coordinator at the Washington State Department of Health. She works to improve youth access to and experience with sexual and reproductive healthcare. Her background is in clinical quality improvement, community-based participatory methods, and program evaluation.

Megan Rogers, MS, is an evaluation manager at the Northwest Center for Public Health Practice in the University of Washington School of Public Health. She has expertise in the evaluation of programs, trainings, and training needs assessments. She has extensive experience in evaluating community-based and school-based programs.

Betty Bekemeier, PhD, MPH, RN, FAAN, is the director of the Northwest Center for Public Health Practice at the University of Washington School of Public Health and a School of Nursing professor. She is a public health systems and practice-based researcher, focusing on structures and practices of state and local health departments in relation to health outcomes and reducing disparities.