A formative evaluation of a web-based intervention for women with a sexual assault history and heavy alcohol use

Abstract

Objective:

Sexual assault (SA) among college women is widespread and is associated with negative consequences including heavy drinking. However, women with SA histories are rarely the target of alcohol interventions and existing alcohol interventions do not address the distal factors that contribute to heavy drinking in this group, such as emotion regulation and distress tolerance. The goal of this study was to evaluate a newly developed web-based alcohol intervention targeting college women with SA histories.

Method:

Heavy drinking college women with SA histories (N = 21) reviewed a series of brief web-based alcohol reduction and regulatory (i.e., emotion regulation, distress tolerance) skill modules and provided feedback on each module.

Results:

Directed content analysis of open-ended survey responses resulted in three themes: intervention content (i.e., what was said in the intervention), intervention delivery (i.e., the look and feel of the intervention), and areas for improvement (i.e., how to enhance the intervention). Quantitative ratings indicated that participants found skill modules moderately relevant and engaging, and qualitative themes and subthemes highlighted important areas for improvement.

Conclusions:

Incorporating user feedback early in the intervention development process provides critical information for content and delivery modifications that may enhance the target population’s engagement and satisfaction with the final product.

Sexual assault (SA) is a pervasive problem on college campuses. Approximately one in four women report experiencing an attempted or completed rape by the end of college (Humphrey & White, 2000; Krebs, Lindquist, Warner, Fisher, & Martin, 2009). In addition, upwards of one-third of young women transitioning into college have already experienced SA (Blayney, Scalco, Radomski, Colder, & Read, 2019; Humphrey & White, 2000; Krebs et al., 2009) and college revictimization tends to be the norm rather than the exception (Blayney et al., 2019; Walsh et al., 2014). A large literature has linked SA to a range of negative outcomes, including heavy drinking and distress (Dworkin, Menon, Bystrynski, & Allen, 2017). Though alcohol use is common in college students (Schulenberg et al., 2019), college women with SA histories report heavier and more frequent drinking relative to their non-victimized peers (Testa & Livingston, 2018). Consistent with the self-medication hypothesis (Khantzian, 2003), women with SA histories may drink heavily in order to temporarily alleviate ongoing distress. However, these short-term self-regulatory attempts may negatively reinforce and maintain maladaptive coping (Miranda, Meyerson, Long, Marx, & Simpson, 2002) and even increase risk for revictimization (McCauley, Calhoun, & Gidycz, 2010). To disrupt this cycle, intervention efforts aimed at reducing heavy drinking among college women with SA histories are greatly needed.

The Importance of Emotion Dysregulation

Emotion dysregulation has emerged as an important factor underlying the link between SA and negative outcomes (Messman-Moore, Walsh, & DiLillo, 2010; Messman-Moore, Ward, & Zerubavel, 2013; Messman-Moore, Ward, Zerubavel, Chandley, & Barton, 2015). According to Gratz and Roemer (2004), emotion regulation involves an awareness, understanding, and acceptance of emotional experiences as well as distress tolerance and other emotion-related change strategies to facilitate adaptive behavior. Skills in emotion regulation aim to reduce the intensity and duration of emotions rather than avoiding or eliminating the experience (Gratz & Roemer, 2004). For college women with SA histories, tolerating and managing post-assault distress can be difficult, especially when there are deficits in regulatory abilities. Such deficits have the potential to draw at-risk women toward more maladaptive strategies in order to alleviate distress, such as heavy drinking. Despite our growing understanding of the importance of emotion dysregulation in maintaining negative outcomes for those with SA histories (e.g., Messman-Moore et al., 2010; Messman-Moore et al., 2013), this critical target has yet to be integrated into alcohol interventions for this population.

Lack of Alcohol Interventions Targeting College Women with SA Histories

Currently, no known interventions specifically target heavy drinking among college women with SA histories. Moreover, only a handful of alcohol interventions have considered how SA histories might influence intervention outcomes (e.g., Brahms et al., 2011; Clinton-Sherrod et al., 2011; Gilmore et al., 2015). Among those, the focus has been on reducing heavy drinking via psychoeducation, personalized normative feedback, and/or protective strategies, but without accounting for the distal factors that influence or maintain drinking patterns, like emotion dysregulation or difficulty tolerating distress. In addition, most of the referenced interventions (e.g., Brahms et al., 2011; Clinton-Sherrod et al., 2011) have been administered in person, which can be costly, time consuming, and difficult to disseminate at a larger level. Alternative approaches – such as web-based administration – have the potential to improve intervention reach and dissemination by offering college women with SA histories a “virtual” safe space to learn the necessary skills to reduce heavy drinking and related outcomes. However, as part of intervention development, it is critical to hear directly from the target population in order to ensure that such an intervention would address their unique needs and ensure adequate interest and participation with the online content.

Incorporating College Women with SA Histories into Intervention Development

Interventions are typically developed with a target population in mind, yet whether these individuals are included in the development process varies by field of study. For example, although this is common in community-based participatory research, it is not standard practice in many other fields (Altman, Huang, & Breland, 2018; Lyon & Koerner, 2016). This is a critical flaw as the decisions made during development (e.g., content, delivery) have the potential to dramatically influence intervention impact (Dopp, Parisi, Munson, & Lyon, 2019). User-centered design, an innovative approach from the technology sector, incorporates the target population into the development process to maximize user engagement and satisfaction with the final product (Lyon & Koerner, 2016). This is especially important for underserved populations – ones who are not typically the focus of interventions – to ensure that their unique preferences and needs are met (Dopp et al., 2019). Qualitative methods can complement this process by offering key insight into users’ preferences and needs in their own words. Thus, drawing on user-centered design principles, we sought to address these gaps in the present study.

The Present Study

SA continues to be widespread on college campuses and is linked to a number of negative outcomes, including heavy drinking. Few alcohol interventions have considered how SA histories might influence treatment outcomes and none have targeted the distal factors that contribute to heavy drinking in this high-risk group, such as emotion regulation and distress tolerance. To build more effective interventions, the target population must be incorporated into intervention development to address users’ preferences and needs prior to conducting clinical trials. With user-centered design principles, we collected feedback from heavy drinking college women with SA histories as part of the development of a brief web-based intervention composed of alcohol reduction strategies and regulatory skills (i.e., distress tolerance, emotion regulation).

Methods

Participants and Procedures

Twenty-one heavy drinking college women with SA histories participated in an intervention development pilot study. Names and contact information of female students were acquired from the University Registrar’s Office and randomly selected individuals (N = 1,400) were sent an email invitation containing the study description and a link to a 10-minute online screening survey. Interested women (22% of those invited, n = 305) provided informed consent and completed the screening. Eligibility criteria for the study included women enrolled at the university who (1) were ages 18 or older; (2) reported a lifetime SA history (attempted or completed oral, anal, or vaginal penetration); (3) engaged in at least two heavy episodic drinking episodes (4+ drinks in two hours; NIAAA, 2003) in the past month; and (4) consumed an average of seven or more standard drinks per week in the past month. Eligible participants (8% of those interested; 2% of those invited; n = 25) were invited to complete a two-hour online survey involving background questionnaires and feedback on intervention modules (see Intervention Content and Structure). A total of 21 women (84% of eligible; 2% of those invited) completed the survey within the required timeframe (2 weeks) and received a $50 Amazon gift card. All participants were provided a list of resources on alcohol and sexual violence. Procedures were approved by the University’s Institutional Review Board.

On average, participants were 20.71 years old (SD = 1.40). With regard to race, approximately two-thirds (66.7%) identified as White, 23.8% as Asian/South Asian, 9.5% as Multiracial/Other. Regarding ethnicity, 10.0% identified as Hispanic. Thirty-eight percent of participants were sophomores, 14.3% juniors, 33.3% seniors, and 14.4% fifth year seniors. In the past month, participants consumed an average of 19.52 drinks (SD = 11.36) per week and engaged in an average of 8.67 heavy episodic drinking episodes (SD = 7.32).

Intervention Content and Structure

The current intervention, titled Women’s Emotions and Behaviors (WEB), was designed by the first, fourth, and last author. Intervention content was based on skills derived from cognitive behavioral therapy (Ellis, 2008) and included an emphasis on the transactional influence of thoughts, emotions, and behavior, as well as cognitive behavioral skills for reducing alcohol misuse derived from BASICS and relapse prevention (Cronce & Larimer, 2011; Larimer & Cronce, 2007; Monti et al., 2002). Additionally, intervention content was based on skills derived from dialectical behavior therapy (Linehan, 2014) targeting emotion regulation and distress tolerance. Feedback on preliminary intervention content was obtained from experts in the field of alcohol and emotion regulation interventions. The intervention was composed of 14 skill modules, including seven alcohol reduction strategies and seven regulatory skills (see Table 1). The entire intervention was designed and disseminated online; participants were able to click through skill content at a time and location that was convenient for them. Each module included an animated 2-minute introductory skill video which provided an overview of the skill. Afterward, participants viewed skill-specific content that included a combination of text, images, and interactive exercises (e.g., matching activities, open-ended questions). In a few skill modules, hyperlinks directed participants to described resources (e.g., gratitude journal, rideshare apps). Modules were designed to take 5–10 minutes each. As part of intervention development, we used a planned missingness design in which participants received nine of the 14 skill modules at random. All participants viewed all skill modules to which they were assigned. On average, each skill module was viewed by 13 participants. After viewing each skill module, participants responded to open-ended and rating scale items where they provided module-specific feedback.

Table 1.

Descriptions of all alcohol reduction and regulatory skill modules

Alcohol Reduction Skill Modules	Description	Regulatory Skill Modules	Description
Alcohol Health and Norms	Psychoeducation about alcohol’s effects on behavior; normative feedback	Labeling Emotions	Psychoeducation about identifying emotions and vulnerability factors
Protective Behavioral Strategies	Behavioral skills to reduce negative consequences related to alcohol use	Opposite Action	Changing unwanted or ineffective emotions by acting opposite of how one feels
Managing Negative Thinking	Cognitive restructuring to challenge appraisals relevant to alcohol; ABC model	Cope Ahead	Coping effectively with distress by utilizing problem solving, relaxation, and mental rehearsal
Drink Refusal	Skills for refusing alcohol in high-risk situations; assertiveness training	STOP	Stop, Take a step back, Observe, and Proceed mindfully to navigate a crisis situation
Seemingly Irrelevant Decisions	Understanding how a variety of choices may unexpectedly lead to heavy alcohol use	Pros and Cons	Evaluating short-term and long-term benefits and costs of a difficult decision
Analysis of Triggers	Identifying and modifying cues (people, situations, emotions) for heavy alcohol use	Radical Acceptance	Techniques for accepting current reality in order to reduce distress and suffering
Sources of Support	Engaging social supports; community resources for alcohol use and distress	Half-Smiling/Willing Hands	Techniques for accepting current reality using the body in order to reduce distress and suffering

Note. ABC model = activating event, belief, consequence model.

Measures

Screening for SA history.

Lifetime SA history was assessed with the Traumatic Life Events Questionnaire (Kubany et al., 2000). Participants reported how often an individual touched sexual parts of their body or made them touch sexual parts of the individual’s body against their will or without their consent during childhood (i.e., before age 13), adolescence (i.e., between 13–18), and adulthood (i.e., after age 18). Response options ranged from 0 = Never to 6 = 5 or more times. Participants who responded 1 or more times during any age period were asked whether the incident involved attempted or completed penetration (oral, vaginal, or anal). Response options were 0 = No and 1 = Yes. Any endorsement of attempted or completed penetration was coded as having a SA history.

Screening for alcohol use.

Average number of standard drinks consumed during a typical week in the past month was assessed with the Daily Drinking Questionnaire (Collins, Park, & Marlatt, 1985). From this measure, the total number of drinks reported per week was summed. Heavy episodic drinking frequency (NIAAA, 2003) in the past month was assessed by asking: “How many times did you consume 4 or more drinks containing any kind of alcohol within a two-hour period in the past month?” Participants responded on a scale of 0 to 30 times.

Qualitative intervention feedback.

Following each skill module, participants were presented with five open-ended questions to assess skill applicability (“What would make this exercise/skill more applicable to you?”), usefulness (“What would make this exercise/skill more useful for you?”), likes (“What did you like the most about this exercise/skill?”), dislikes (“What did you like the least about this exercise/skill?”), and further suggestions for improvement (“Is there anything else we could do to improve this exercise/skill?”).

Quantitative intervention feedback.

Immediately following each skill module, participants provided ratings to assess familiarity (“How familiar were you with the materials presented in this exercise/skill prior to today?”), relevance for self (“How relevant is this exercise/skill for you and your life?”), relevance for others (“How relevant do you think this exercise/skill would be for other college women?”), and engagement (“How engaging was the presentation of the material for this exercise/skill?”). Response options ranged from 0 = Not at all to 5 = Extremely. Participant responses were averaged for each skill module to attain mean scores, with higher scores indicative of greater familiarity, relevance, and engagement.

Data Analytic Approach

Both qualitative and quantitative data were downloaded into SPSS v.26 for analysis. Qualitative data were analyzed with directed content analysis (Hsieh & Shannon, 2005). The first two authors independently reviewed responses to all five open-ended questions to become familiar with the general content provided by participants. Next, transcripts were independently read a second time to generate a list of preliminary codes. These codes were discussed by the first two authors and an iterative process ensued to operationalize, refine, and finalize codes as a study team. Salience, frequency, and extensiveness of data informed the development of codes (Creswell & Poth, 2018). Few coding disagreements emerged (kappa = .98); when these arose, disagreements were resolved through discussion. Quantitative data were summarized with descriptive statistics in SPSS to evaluate mean scores of familiarity, relevance to self, relevance to others, and engagement for each of the 14 skill modules.

Results

Qualitative Results

Three themes emerged from participant responses to the intervention: (1) content-related (i.e., what was said in the intervention), (2) delivery-related (i.e., the look and feel of the intervention), and (3) other areas for improvement (i.e., how to enhance the intervention). Themes and subthemes (see Table 2) are discussed below. We describe feedback provided that fell within each theme. Sample participant quotations are followed in brackets by the study-specific participant identifier and the name of the skill module for which the feedback was given.

Table 2.

Qualitative feedback on skill modules

	Number of codes	% of total codes	% within theme	% participants endorsing theme
Theme: Content	425	65		100
Meeting women where they are	253	39		100
Specific skills utility	167		66
Ease and satisfaction	75		30
Options and alternatives	11		4
Not meeting women where they are	172	26		100
Information framing	77		44
Skill difficulty	51		30
Relevance	44		26
Theme: Delivery	76	12		76
Want more	31		41
Want less	30		39
Likes	15		20
Theme: Improvements	152	23		90
Specific information and/or skills	83		55
More examples	49		32
More practice	20		13

Note. Total number of codes is 653.

Content

The first theme focused on intervention content. These responses fell into two areas: meeting women where they are and not meeting women where they are. Each was composed of three subthemes.

Meeting women where they are.

Within this theme, participants described how the intervention content connected with them in a way that was helpful and consistent with their values. Subthemes (see Table 2) involved skill utility, content being easy to understand, and providing a variety of options and alternatives.

Specific skills utility.

Participants described the perceived utility of the skills presented and the knowledge gained from these specific techniques. Participant responses fell in this subtheme fairly equally across alcohol reduction and regulatory skill modules. For example, one participant commented:

“I really liked the cool facts about my drinking habits. I think, putting it in terms of how many calories I’m consuming, how much money I’m spending, and what the money could actually buy, really was an eye opener and hit home harder than someone just telling me that ‘you drink too much’” [ID #19, Alcohol Health and Norms Module].

Moreover, many individuals focused on the applicability and usefulness of the intervention content in helping with distress. One participant stated, “I liked that this exercise is very applicable to everyday derailing events as well as past trauma” [ID #17, Opposite Action Module], while another woman stated, “I liked how useful it is” [ID #13, Pros and Cons Module]. Users also noted the novelty of the skill modules, with one participant stating, “I liked the understanding that one of the most important things is to learn to de-stress and relax and cope ahead of a situation. I’ve never thought of doing that before” [ID #18, Cope Ahead Module].

Ease and satisfaction.

Participants also reported that the intervention content was easy to understand and satisfying. Participant responses fell in this subtheme most often when completing the Protective Behavioral Strategies and Seemingly Irrelevant Decisions skill modules. More specifically, participants described appreciating that the information was brief, straightforward, and informative as illustrated by these comments: “What I liked most about the exercise is that it is fairly simple” [ID #11, Pros and Cons Module], and “I liked that it was short and to the point” [ID #18, Protective Behavioral Strategies Module]. Others reported satisfaction with the material presented as is. One participant stated, “I can’t suggest anything [for improvement] because it is great” [ID #19, Alcohol Health and Norms Module], and another stated, “I liked everything” [ID #20, Seemingly Irrelevant Decisions Module].

Options and alternatives.

A subset of participants also discussed how the intervention content provided them with a variety of options for skills to use, especially in relation to reducing alcohol risk. Comments of this kind were most common in the Analysis of Triggers and Protective Behavioral Strategies modules. One participant stated, “I loved how many options there were of things you can do to soothe yourself if you are feeling tempted to drink heavily” [ID #10, Analysis of Triggers Module]. Others appreciated the examples and suggestions within each module, as noted by a user who stated, “I liked the many different types of skills that can help you drink responsibly” [ID #9, Protective Behavioral Strategies Module]. Another participant echoed this sentiment, “What I liked most was learning about the ride share options” [ID #13, Protective Behavioral Strategies Module]. Others remarked on the alternatives to drinking or experiencing intense emotions. One user stated, “What I liked is that it offers good alternatives [to heavy drinking] that make you happy” [ID #5, Analysis of Triggers Module].

Not meeting women where they are.

Intervention content did not always connect with participants. Subthemes (see Table 2) involved issues with information framing, skill difficulty, and relevance.

Information framing.

One reason intervention content did not resonate with some participants was because information was framed in a way that they did not like. Participant responses fell in this subtheme most often when completing the Protective Behavioral Strategies and Sources of Support modules. In these cases, participants emphasized that “wording matters” and highlighted times in which intervention content was perceived as judgmental or hypercritical. Additionally, some participants disliked how the intervention described alcohol use as “a problem” and were not satisfied with the wording used to teach about drinking-related consequences. For example, one participant stated:

“It’s a little obnoxious that women are always taught how to stay safe, but men never seem to be taught to just respect women and leave them alone if told to do so. Also, it leads to self-blame if a woman forgets one of these skills. If she sets her drink down for a moment, and a man drugs it, she may look back later and think, ‘Well, I was being dumb and left my drink alone for a moment. It is partially my fault I was drugged’” [ID #21, Protective Behavioral Strategies Module].

Another participant noted:

“Well it felt like it was trying to shame me for drinking ‘so much’, it would be a lot more useful if was framed like ‘If you are going to drink, know these tactics for not going overboard. Here’s why a hangover sucks. Here’s how alcoholism differs from casual drinking. Here’s your brain after years of alcohol abuse’” [ID #9, Alcohol Health and Norms Module].

Others discussed that the intervention content was ‘not new’ or ‘too basic’, as illustrated by this participant who commented, “This exercise seemed a little more elementary, like I was being talked to as a 10-year-old” [ID #14, Half-Smiling/Willing Hands Module]. Others stated: “Pros and cons lists have been around for FOREVER, maybe a new strategy here” [ID #16, Pros and Cons Module], and “It’s nothing new, this advice has been told to me before” [ID #15, Sources of Support Module].

Skill difficulty.

For some participants, specific skills seemed too difficult to implement and this was especially true for the regulatory skill modules. Participant responses fell in this subtheme most often when completing the Half-Smiling/Willing Hands and Radical Acceptance modules. Of the individuals that fell into this subtheme, many believed these skills were ‘easier said than done,’ especially when experiencing intense emotions. For instance, one user said:

“I felt like there weren’t really any strategies for accepting reality, it just told me what radical acceptance was and then told me to do that. I don’t feel like it was successful or like I know how to do it” [ID #16, Radical Acceptance Module].

Others commented on the awkwardness of practicing some skills and questioned their effectiveness. One participant remarked, “It feels a little forced and I’m skeptical that it would work” [ID #15, Half-Smiling/Willing Hands Module]. Some voiced concerns that skills may be difficult to enact, and as a result, could increase distress: “What I liked least about this is that it can create a feedback loop for stress” [ID #17, Cope Ahead Module].

Relevance.

Some participants did not see certain aspects of the intervention content as relevant or applicable. Participant responses fell in this subtheme most often when completing the Drink Refusal and Sources of Support modules. Issues with relevance appeared for experiences that participants did not find problematic for them, such as pressure to drink or opportunities to establish social support. For example, one participant stated, “I don’t get peer pressured into drinking so I found this irrelevant” [ID #3, Drink Refusal Module]. Another explained, “I am just not very sociable and I don’t go out of my way to make friends so this is not very relevant to me” [ID #16, Sources of Support Module]. Other statements focused on how intervention content was too narrow, unrealistic, or unnecessary given their current life stressors or experiences. For example, one participant described how the intervention was not inclusive of her drinking experience: “This exercise seemed only geared toward people who drink when they feel negative emotions” [ID #4, Analysis of Triggers Module].

Delivery

The second theme that emerged from participant responses focused on the delivery of intervention content, including the design, organizational flow, graphics, and interactivity of skill modules. Subthemes (see Table 2) involved what participants wanted more of, wanted less of, and what they generally liked.

Wanted more.

Participants reported wanting more of certain aspects of intervention delivery. Responses fell in this subtheme most often when completing the Radical Acceptance and Labeling Emotions modules. The majority described needing more images and interaction, as emphasized in these comments: “Add some more interaction pieces into the exercises please because that helps me learn more easily and grasp the content longer” [ID #19, Cope Ahead Module], and “I would like more images to keep me interested” [ID #7, Seemingly Irrelevant Decisions Module]. Others suggested different ways to make it more interactive: “Add more interactive things like games or something that’ll help me remember all the information” [ID #19, Radical Acceptance Module]. Participants also commented on the organization and flow of the intervention with suggestions to add video captions and page numbers.

Wanted less.

Other responses focused on a desire for less in terms of intervention delivery. Participant responses fell in this subtheme most often when completing the Cope Ahead and Managing Negative Thinking modules. Most asked for skill modules to be shorter in duration and to include less repetition within and across skills. One participant commented, “What I liked least about this intervention is the repetition. I’ve noticed there are some repeated questions throughout asking me in what situations I could use this skill” [ID #14, STOP Module]. Additionally, one individual commented on intervention length and structure stating, “I want this to be less time consuming…I want less clicking through – really ruined the relaxing vibe” [ID #2, Managing Negative Thinking Module]. In contrast to other subthemes, some preferred fewer images and videos because they felt it was distracting.

Likes.

Participants reported enjoying the interactive nature of the intervention, including hyperlinks to skills-related resources. Responses fell in this subtheme most often when completing the Managing Negative Thinking and Analysis of Triggers modules. One participant emphasized her appreciation by stating, “Thanks for linking the app download to the gratitude journal” [ID #3, Analysis of Triggers Module]. Others enjoyed the use of activities to increase engagement, as demonstrated by one user, “I liked the activity portion and matching game” [ID #9, Managing Negative Thinking Module]. A few others made positive remarks about the graphics and design, with one noting, “I liked how the red coloring added to the STOP technique and the way it was presented” [ID #10, STOP Module].

Areas for Improvement

The final theme that emerged was intervention improvements. Subthemes (see Table 2) involved adding other information or skills, more examples, and more practice.

Specific information and/or skills.

Participants emphasized that the intervention could be improved with additional information and techniques. Responses fell in this subtheme most often when completing the Protective Behavioral Strategies and Labeling Emotions modules. Some individuals requested more facts about alcohol and health, such as: “Maybe give some more input on what qualifies as ‘heavy drinking’” [ID #14, Sources of Support Module], or “Talk more about how [alcohol] affects brain cells and functioning (also long-term effects)” [ID #20, Alcohol Health and Norms Module]. Others asked for more information that could be tailored to their needs. For example, one participant requested the inclusion of a blood alcohol chart: “To improve this you could add a body mass chart that shows the number of drinks you can safely drink in a given amount of time” [ID #16, Protective Behavioral Strategies Module]. Other suggestions included skills for specific emotional experiences or other intervention options: “I’d like a greater emphasis on anxiety, an emotion I have trouble regulating” [ID #10, Labeling Emotions Module], and “Stress the fact that it’s okay to bother people with your emotional burdens and also mention more outside sources like groups and counseling” [ID #13, Sources of Support Module]. Mindfulness and other relaxation techniques were also requested additions. For example, one participant stated, “This can be improved if it included deep breathing, which is my technique to achieve the same ends here, but works best when I am feeling irritated” [ID #9, Radical Acceptance Module].

More examples.

In addition to specific information and skills, participants wanted more examples including stories, scenarios, and techniques. Responses fell in this subtheme most often when completing the Seemingly Irrelevant Decisions and Opposite Action modules. For example, participants often provided responses such as, “I wish there had been more examples of the skills” [ID #6, STOP Module] or “What would improve this intervention is more example stories” [ID #2, Seemingly Irrelevant Decisions Module]. While the intervention already included multiple examples, it was important that these be more targeted. For example, participants requested, “Focus on more situations catered specifically toward women” [ID #7, Managing Negative Thinking Module] or “Do a workplace or classroom setting example” [ID #1, Opposite Action Module]. Examples were especially needed for the Opposite Action module. One individual remarked, “This could be better if there were more examples for opposite action for each emotion and if more emotions were included” [ID #5, Opposite Action Module]. In addition, participants wanted to know how to apply the skills. One user stated, “This would be more applicable if there were more examples of situations where you could use this skill” [ID #10, Pros and Cons Module], and another participant said, “Show examples of how people already have their own ways of doing this” [ID #2, Radical Acceptance Module].

More practice.

Lastly, participants requested additional practice of the skills. Responses fell in this subtheme most often when completing the Managing Negative Thinking and Labeling Emotions modules. One individual commented, “What would make this better is more practice. I liked this one too” [ID #18, Managing Negative Thinking Module], while another stated, “Add more time with practice” [ID #8, Labeling Emotions Module]. Other participants asked for more practice refusing drinks: “What would improve this is practicing not caring what others think of me and how to say no more confidently because it’s hard to say no sometimes because I’m afraid that I might annoy the person” [ID #19, Drink Refusal Module].

Quantitative Results

Descriptive statistics can be found in Table 3. For each module, participants assessed the skill on four qualities: (1) prior familiarity; (2) relevance to the self; (3) relevance to other college women; and (4) engagement. Participants were more familiar with some skills (e.g., Protective Behavioral Strategies, Pros and Cons), but less familiar with others (e.g., Alcohol Health and Norms, Half-Smiling/Willing Hands). Although participants found the skills to be relevant for themselves, they tended to expect that the skills would be more relevant for others. In terms of self-relevance, Protective Behavioral Strategies was the highest rated alcohol reduction skill and STOP was the highest rated regulatory skill. Participants generally reported moderate levels of engagement with the skills as presented.

Table 3.

Quantitative feedback on skill modules

	N	Familiarity	Relevance to Self	Relevance to Others	Engagement
Alcohol Reduction Skill Modules
Alcohol Health and Norms	8	2.75 (1.83)	4.13 (1.13)	4.25 (1.39)	3.88 (1.36)
Protective Behavioral Strategies	13	4.38 (0.65)	4.15 (0.80)	4.54 (0.88)	3.00 (1.47)
Managing Negative Thinking	15	3.33 (1.45)	3.73 (1.34)	4.33 (0.98)	3.33 (1.45)
Drink Refusal	15	4.27 (0.96)	3.33 (1.98)	4.27 (0.96)	2.87 (1.73)
Seemingly Irrelevant Decisions	14	3.00 (1.52)	3.43 (1.34)	4.36 (0.84)	2.93 (1.14)
Analysis of Triggers	13	3.46 (1.39)	3.54 (1.39)	4.08 (0.76)	2.77 (1.42)
Sources of Support	13	4.15 (0.81)	4.00 (1.35)	4.31 (0.86)	2.77 (1.36)
Regulatory Skill Modules
Labeling Emotions	15	3.40 (1.64)	3.40 (1.30)	4.07 (1.34)	3.07 (1.48)
Opposite Action	14	2.29 (1.82)	3.71 (0.99)	4.43 (0.85)	2.79 (1.53)
Cope Ahead	13	2.92 (1.55)	3.46 (1.61)	4.08 (1.26)	2.69 (1.38)
STOP	12	3.42 (1.51)	4.00 (0.85)	3.92 (0.90)	3.08 (0.99)
Pros and Cons	13	4.77 (0.60)	3.77 (1.48)	4.15 (1.14)	2.92 (1.32)
Radical Acceptance	15	2.93 (1.44)	3.87 (1.30)	4.13 (1.25)	2.80 (1.57)
Half-Smiling/Willing Hands	15	2.20 (2.15)	3.20 (1.74)	3.53 (1.69)	2.80 (1.56)

Note. Means and standard deviations are reported. N = number of participants who received each skill module and responded to feedback questions. STOP = Stop, Take a step back, Observe, Proceed mindfully.

Discussion

The current study represents a critical first step in developing a web-based alcohol intervention to address the unique needs of college women with SA histories. Informed by principles of user-centered design, we utilized qualitative and quantitative analyses to evaluate feedback on seven alcohol reduction and seven regulatory skill modules. Through this approach, we determined themes and trends in participant reactions to the online modules. Qualitative analyses revealed major themes and subthemes related to intervention content, delivery, and suggested improvements. Quantitative ratings revealed variable degrees of familiarity and relevance, yet a consistent degree of engagement across modules. These findings shed light on how college women with SA histories experience and interact with specific web-based intervention content promoting alcohol reduction and regulatory skill-building.

Qualitative analyses revealed that participants largely had positive responses to intervention content. Specifically, 60% of content-related responses involved appreciation for meeting women where they are, and the majority (66%) of these responses focused on perceived utility or helpfulness of specific skills. Valuable feedback was also garnered about the framing, relevance, and perceived difficulty of skills. Of note, concerns about the intervention framing were the most common reason the content did not resonate with participants. The frequency of this concern, as well as specific participant responses, are consistent with past suggestions that alcohol reduction skills may inadvertently be perceived as judgmental and activate inaccurate self-blame for substance-related SA experiences (e.g., Jaffe et al., 2018). However, framing feedback was also provided for regulatory skills, suggesting that there is room to improve all modules. Such feedback is vital to determining how to promote change in college women.

Quantitative ratings of each skill module added to our understanding of participant reactions. On average, participants reported moderate engagement across skill modules. Such “engagement” ratings may reflect a combination of participant interest, as well as attentiveness (vs. distractedness) within the setting where they completed the survey. Generally, it can be difficult to maintain participant engagement and attention in web-based surveys and interventions (e.g., Eysenbach, 2005), but current findings are consistent with those reported in prior web-based alcohol interventions (Gilmore et al., 2015; Jaffe et al., 2018; Lewis & Neighbors, 2015). Importantly, participants’ open-ended responses in the current study highlight ways to make the intervention delivery more engaging, for example by supplementing video, image, and text content with interactive activities such as games. Additional suggested improvements include making the skills more accessible through examples and practice, which may also reduce perceived skill difficulty. Current findings also provide some indication that participants may feel more engaged in skills which are less familiar, but still personally relevant, such as the Alcohol Health and Norms skill. Of note, participants reported moderate to high levels of familiarity with the alcohol reduction skill modules, but low to moderate familiarity with regulatory skill modules (with the exception of Pros and Cons). These regulatory skill modules represent a novel addition to web-based alcohol interventions and may help reduce distress-motivated drinking and increase interest and engagement in the overall intervention.

Quantitative relevance ratings varied across specific skill modules. On average, women rated the skills as moderately to highly relevant to themselves, yet tended to rate the relevance for other college women as even higher. These findings can be understood in the context of the “third person effect” (Davison, 1983), a concept from the field of communications and mass media that has received strong empirical support (see Perloff, 2009). This effect refers to the notion that individuals tend to believe a persuasive message will be more relevant to others, particularly if openness to the message reflects an undesirable attribute (Sun, Pan, & Shen, 2008). In the current study, participants may view alcohol reduction and regulatory skills as most relevant for those with problem drinking or dysregulation, respectively, and attributional biases designed to preserve an optimistic view of the self may lead them to see this as a larger problem (and therefore more relevant) for others. Taken in this context, it is noteworthy that three skill modules—Alcohol Health and Norms, Sources of Support, and STOP —evinced minimal discrepancy between self and other relevance ratings, suggesting openness to these skills may not be perceived as undesirable. Further research is needed to determine whether and how such perceived relevance might impact engagement in the modules and implementation of these skills.

Potential Intervention Refinements

The richness of participant responses obtained in this pilot study using simple to administer open-ended survey responses underscores the relative ease and importance of gathering user feedback prior to conducting costly clinical trials. Such feedback is especially important when targeting high-risk populations that are not typically the focus of interventions and when the newly developed intervention will be delivered exclusively via mobile technology making it difficult – if not impossible – to directly respond to participant concerns as they arise.

Results provide several potential avenues for intervention refinement, particularly with regard to content and delivery. For content, carefully framing skills to reflect a nonjudgmental stance is critical. Even when experts develop content with this in mind, soliciting feedback from the target population may reveal aspects of the material that come across as unintentionally judgmental. Additionally, increasing the personal relevance of each skill for participants is recommended. For example, although a participant may not have immediately connected with Sources of Support because they do not consider themselves sociable, increasing the relevance of this skill for all participants may help individuals recognize the potential benefit of positive social supports. Addressing requests for more example stories and specific information within skills may better convey the applicability of such skills to those who initially question their utility. However, not all skills will be relevant for all individuals, and whereas clinician-delivered treatments can be tailored to address specific skill needs and relevant examples, such decision rules must be more formalized within web-based interventions.

Participants offered a number of practical suggestions including captioning videos and adding page numbers to modules. However, feedback in these areas also highlighted a challenge with balancing competing suggestions from participants. For example, some participants wanted more practice, examples, and images, whereas others wanted fewer redundancies. However, across all suggestions for improvement, there was a strong desire for more interactional elements (e.g., games and activities). Increasing these components may make redundancies in content and delivery less frustrating while also providing more opportunities to learn and practice novel skills, which is particularly important for skills that were viewed as more difficult to implement. Another way to increase opportunities to practice is to include additional coaching outside of the presentation of skill modules, such as text-message prompts or reminders following skill delivery, consistent with empirical support for making personal commitments (Hollenbeck et al., 1989). In these ways, participant feedback provided clear avenues for refining intervention content and delivery prior to broader implementation and testing.

Limitations

There are several limitations of the current research. First, participants completed the background survey and reviewed all skill modules online at once. Though we used a planned missingness design in which participants were shown a subset of modules at random to reduce burden, the modules were presented in a standard order and there was likely deterioration in attention and feedback in later skill modules. Additionally, qualitative feedback was obtained online using open-ended questions rather than in-person interviews, which would have allowed for follow-ups to elicit more information. Moreover, this was not a treatment seeking sample, which could have impacted quantitative ratings, particularly personal relevance. Due to small sample size, differences between quantitative ratings were not statistically evaluated. Finally, our sample was limited to women who experienced attempted or completed rape to focus on those more likely to experience high levels of distress associated with SA. Nevertheless, this limits generalizability to other groups, including men and individuals who have experienced unwanted sexual contact or sexual harassment in the absence of attempted or completed rape.

Conclusion

This formative evaluation of a novel web-based alcohol intervention for heavy drinking college women with SA histories offers guidance for enhancements to this specific intervention as well as web-based interventions more generally. Results highlight the importance of involving the target population into intervention development activities and demonstrate that valuable information can be obtained from this relatively low resource approach. Importantly, despite the best efforts by experts to develop content that meets participants where they are, it is essential to check those assumptions as language or content can inadvertently be perceived as judgmental, particularly when encouraging SA survivors to reduce their alcohol use. However, much of the feedback was favorable and many participants expressed appreciation for receiving new skills to help them manage their difficulties. This was especially true of regulatory skills, which are a novel addition to alcohol interventions and underscore the need for continuing to hone our ability to deliver these often complex skills in a simple, user friendly, and engaging way.

Clinical Impact Statement:

Findings highlight the importance of involving target populations into intervention development prior to conducting costly clinical trials. Formative evaluations provide valuable feedback for intervention refinement and modification. This process may increase the likelihood of creating web-based interventions in a user-friendly and engaging way that meets the target population where they are.