A Hybrid Evaluation-Optimization Trial to Evaluate an Intervention Targeting the Intersection of Alcohol and Sex in College Students and Simultaneously Test an Additional Component Aimed at Preventing Sexual Violence

Abstract

Background

Using the multiphase optimization strategy (MOST), we previously developed and optimized an online behavioral intervention, itMatters, aimed at reducing the risk of sexually transmitted infections (STI) among first-year college students by targeting the intersection of alcohol use and sexual behaviors.

Purpose

We had two goals: (a) to evaluate the optimized itMatters intervention and (b) to determine whether the candidate sexual violence prevention (SVP) component (included at the request of participating universities) had a detectable effect and therefore should be added to create a new version of itMatters. We also describe the hybrid evaluation-optimization trial we conducted to accomplish these two goals in a single experiment.

Methods

First year college students (N = 3,098) at four universities in the USA were individually randomized in a hybrid evaluation-optimization 2 × 2 factorial trial. Data were analyzed using regression models, with pre-test outcome variables included as covariates in the models. Analyses were conducted separately with (a) immediate post-test scores and (b) 60-day follow-up scores as outcome variables.

Results

Experimental results indicated a significant effect of itMatters on targeted proximal outcomes (norms) and on one distal behavioral outcome (binge drinking). There were no significant effects on other behavioral outcomes, including the intersection of alcohol and sexual behaviors. In addition, there were mixed results (positive short-term effect; no effect at 60-day follow-up) of the SVP component on targeted proximal outcomes (students’ self-efficacy to reduce/prevent sexual violence and perceived effectiveness of protective behavioral strategies).

Conclusions

The hybrid evaluation-optimization trial enabled us to evaluate the individual and combined effectiveness of the optimized itMatters intervention and the SVP component in a single experiment, conserving resources and providing greatly improved efficiency.

Trial Registration

NCT04095065.

College freshmen who completed an optimized online intervention, itMatters, showed greater reductions in alcohol and sex-related norms and binge drinking than those who did not.

Introduction

Engagement in sexual behaviors is an appropriate developmental activity for young adults, yet as is the case for any age group, it can result in potentially negative consequences (e.g., sexually transmitted infections [STI]). In the United States (U.S.), young adults have high rates of STIs [1] with college students’ risk for STI acquisition frequently associated with alcohol use (e.g., alcohol consumption increases intentions to engage in sex without a condom) [2, 3]. In addition, norms about positive attributes of alcohol use when it accompanies sex have been linked to higher risk sexual behavior [2, 4–11]. Notably, 52% of college students believe alcohol can facilitate sexual opportunity, suggesting they may use alcohol prior to engaging in sex, and thus may be at elevated risk for foregoing condom use [12]. Most existing interventions target alcohol use [13, 14] or condom use [15–17] independently. However, given the common co-occurrence of alcohol use and sexual behaviors, interventions are needed that specifically target the intersection of these behaviors while also addressing the individual behaviors. To fill this gap, we developed and optimized itMatters, an intervention for first year college students designed to support sexual health and prevent STIs by addressing alcohol use and sexual behaviors with a specific focus on this critical intersection [18].

Application of MOST in itMatters’s Development, Optimization, and Evaluation

Our development of itMatters was guided by the multiphase optimization strategy (MOST) [18–21], an engineering-inspired framework for creating optimized (i.e., more effective, efficient, economical, and/or scalable) interventions. MOST has three phases: preparation to optimize an intervention, optimization of the intervention, and evaluation of the optimized intervention [19]. In our prior work, reported elsewhere [18, 20], we completed the first two phases. In the preparation phase, we established a theoretically and empirically driven conceptual model that provided the basis for the itMatters intervention, identified components that were candidates for inclusion in the intervention, and pilot tested these candidate components for acceptability and feasibility [20]. In the optimization phase, we conducted a series of two independent randomized experiments, or optimization trials, to assess the effect of each individual candidate intervention component on proximal outcomes. The proximal outcomes were mediators directly linked to the behavioral outcomes as specified in our conceptual model [18]. A priori, we decided to include components that achieved a main effect size of d ≥ |.15| in the optimized intervention, as long as they did not have a large antagonistic interaction with any other components. The two-optimization trials were conducted in an iterative fashion [18]. At the conclusion of the optimization trials, two components met our inclusion criteria: a component that focused on correcting erroneous descriptive norms (i.e., prevalence norms) and a component that focused on correcting erroneous injunctive norms (i.e., approval norms) [18]. The intervention also included an informational component that served as the foundation for all other intervention content. Because it was necessary for all participants to receive the informational component, it was not evaluated individually.

According to the MOST framework, the optimization phase may be followed by evaluation of the optimized intervention, typically in a two-arm randomized control trial (RCT). As we were planning the RCT, we were mindful of two fundamental principles of MOST [19]. One principle is continual optimization, the idea that MOST presents the opportunity to make an intervention increasingly effective, efficient, economical, and/or scalable by repeated cycles of preparation, optimization, and evaluation. A second principle of MOST is the resource management principle, which states that the investigator must make the best use of research resources by carefully selecting a highly efficient and appropriate experimental design. Consistent with the first principle, we wanted to improve itMatters by adding a sexual violence prevention (SVP) component in response to requests from participating schools to add content to meet the Clery Act’s educational requirements [22]. Before we added the SVP component to itMatters, it was necessary to establish its effectiveness empirically in an optimization trial. Given the urgency of the requests, we did not want to wait until after we had conducted the RCT to conduct this optimization trial, and moreover we did not have the resources to conduct another optimization trial. Therefore, following the second principle, we determined that instead of conducting a two-arm RCT to evaluate itMatters and a separate optimization trial to assess the performance of the SVP component, it would be more efficient to conduct a hybrid evaluation-optimization trial as a single 2 × 2 factorial experiment, where one factor corresponds to the evaluation objective and the other corresponds to the optimization objective.

Accordingly, in this paper, we report on how we addressed two specific goals: (a) to evaluate the optimized version of the itMatters intervention on proximal and behavioral outcomes and (b) to determine whether the SVP component had a detectable effect on proximal mediators and therefore should be added to the itMatters intervention package moving forward. We also describe the hybrid evaluation-optimization trial used to accomplish these two goals simultaneously and discuss its potential in other applications.

Methods

This section outlines the evaluation (itMatters intervention) and optimization (SVP candidate component) methods. We followed CONSORT guidelines and the trial has been registered (no. NCT04095065). The Institutional Review Board at the University of North Carolina Greensboro approved all study protocols. Informed consent was obtained from all individual participants included in the study.

Participants

Participating schools were four co-educational, 4-year, public universities in the U.S. We purposely recruited universities that varied in characteristics such as size, geographic location, and racial/ethnic composition. To be eligible to participate in the study, universities (a) were not currently implementing an online alcohol or safe sex intervention for first-year students; (b) were willing to assist in recruiting first-year students by supplying their names and email addresses; (c) permitted data collection in the academic year in which the experiment was conducted; (d) allowed random assignment of students to experimental conditions; and (e) agreed to encourage all incoming first-year students to complete the intervention components that they were randomized to receive. Students were eligible to participate if they were over 18 and an incoming (i.e., first-year student or transfer) student.

Intervention Components

The itMatters intervention is described in detail elsewhere [18]. Briefly, itMatters includes three components: general information (e.g., standard alcoholic drink definitions and instructions on proper condom use), descriptive norms (e.g., perceptions of the prevalence of college student use of alcohol before or during sex), and injunctive norms (e.g., perceptions of how acceptable peers find the use of alcohol before or during sex). The trial also tested a comprehensive SVP component for potential inclusion in future itMatters iterations. The SVP component was originally developed for student athletes [23] and was revised to meet the needs of all first year college students. It covered the three key Clery Act [22] areas: (a) essential sexual violence educational content (e.g., important definitions, prevalence, and consent); (b) information related to campuses’ responsibilities for reporting acts of sexual violence and other related crimes; and (c) bystander intervention training that also included safe and effective steps to be an active “upstander.”

Study Design

The hybrid evaluation-optimization study used a 2 × 2 factorial experimental design; we considered several alternative designs prior to selecting this one. Specifically, we considered conducting two separate experiments (one comparing itMatters vs. active comparison and another comparing SVP vs. active comparison). We also considered conducting a three-arm RCT, with experimental conditions itMatters only, itMatters + SVP, and active comparison. Either of these experimental approaches would have required substantially more participants than the 2 × 2 factorial experiment to be adequately powered [24]. Thus, we selected the 2 × 2 factorial because it allowed us to answer our research questions, estimating all desired effects, while adhering to the resource management principle [19].

Factor 1, the evaluation factor, had two levels: (i) the itMatters intervention and (ii) an active comparison focused on mental health and sleep. Including this factor allowed us to address our first aim—assessing the effectiveness of the optimized itMatters intervention. Factor 2, the optimization factor, also had two levels: (i) receipt of the SVP component and (ii) no receipt of the SVP component. Including this factor allowed us to address our second aim—assessing the performance of the newly added SVP component to determine whether we should include it as part of the itMatters intervention (i.e., meeting the a priori effect size of d ≥ |.15|). Thus, as shown in Table 1, participants were randomized to one of four experimental conditions: (a) itMatters + SVP, (b) active comparison + SVP; (c) itMatters only, and (d) active comparison only.

Table 1.

Hybrid evaluation-optimization trial experimental conditions

Levels of evaluation factor
		itMatters	Active comparison
Levels of optimization factor	SVP provided	(a) itMatters + SVP	(b) Active comparison + SVP
	SVP not provided	(c) itMatters only	(d) Active comparison only
Estimation of effects
Main effect of itMatters: mean of [(a) + (c)] – mean of [(b) + (d)]
Main effect of SVP: mean of [(a) + (b)] – mean of [(c) + (d)]
itMatters SVP interaction: [mean of [(a) – (b)] – mean of [(c) – (d)]]/2

Optimization factor is aimed at assessing performance of SVP component; Evaluation factor is aimed at assessing the performance of itMatters.

SVP sexual violence prevention.

Experimental Conditions

Participants completed all intervention content online via a learning management software, which allowed for seamless delivery of the four experimental conditions (Table 1). Participants randomized to experimental conditions (a) and (c) received itMatters. Participants randomized to experimental conditions (b) and (d), which included the active comparison focused on healthy sleep (e.g., sleep hygiene and strategies to obtain sufficient sleep) and mental health (e.g., destigmatize help-seeking for mental health problems and provide strategies to improve mental well-being). Participants randomized to experimental conditions (a) and (b) received the SVP component. On average, it took participants 25 min to complete the itMatters material, 25 min to complete the active comparison material, and 45 min to complete the SVP material. Although participants were not required to complete their assigned components in one sitting, the average completion time was 25 min for students in conditions (c) and (d) and 70 min for students in conditions (a) and (b).

Procedure

Universities provided the research team with a list of names and emails for all incoming first-year students 1 week prior to the initial implementation window. Using individual-level randomization, we assigned an equal number of first-year students to the four experimental conditions. The 3-week initial implementation window started with an initial email invitation from our team, followed by up to three email reminders for students with outstanding tasks. Students who consented immediately gained access to the baseline survey. Upon completing the survey, students could immediately access their assigned intervention components (e.g., itMatters, SVP, and/or active comparison components). After completing assigned condition, students received immediate access to the post-intervention survey. This initial implementation window occurred pre-matriculation beginning in mid-July (2 universities) and at the beginning of the fall semester in late August (2 universities). Finally, 60 days after the initial implementation window closed, students who completed the baseline and immediate post-intervention surveys received an email invitation to complete the final follow-up survey, followed by up to three email reminders to students who had not completed the final survey.

A total of 3,098 unique eligible students out of the 13,322 invited (23.3% of available student population) consented to participate and completed the baseline survey. Of the total sample, 2,298 completed the immediate follow-up and 1,836 completed the 60-day follow-up survey (Fig. 1). For participants who had duplicate entries (e.g., participants started survey but lost their internet connection so they had to restart the survey once connection was secured), the most complete entry was retained and used in analyses; if more than one entry was similarly complete, we used the earliest entry. Accounting for consent, eligibility at baseline, and removal of duplicate entries, the per-condition sample sizes ranged from 736 to 805.

Fig. 1.

Consort diagram.

The baseline and 60-day follow-up surveys took approximately 15 min to complete, whereas the immediate post-intervention survey, which did not include behavioral items, took less than 10 min to complete. Following the 60-day follow-up survey, we provided all students with access to all intervention content. We compensated students for survey completion; students received $15 for completing both the baseline and immediate post-intervention surveys and $30 for the 60-day follow-up survey for a total possible incentive of $45. We also provided universities an institutional incentive of $2,000 in compensation for their support in recruitment and retention in the study as well as access to a $1,500 marketing budget to help them to promote the study on their respective campus.

Measures

Sample characteristics

Students provided basic demographic information at the baseline survey. Response options for age ranged from 18 to 25+, as all incoming students, including transfer students, were eligible to participate in the research. Students reported their current gender identity (e.g., female, male, transgender, other), sexual orientation (e.g., heterosexual, homosexual, bisexual, other), race (e.g., white, Black, multiracial, other) and ethnicity (e.g., Hispanic/Latinx). Students also reported if they currently lived on-campus (e.g., dorm/residence hall, fraternity/sorority housing, or other on-campus housing), off-campus apartment/house, at home with family, or other housing.

Outcomes Pertaining to Evaluation of itMatters

Proximal outcomes

Two of the itMatters intervention components targeted specific proximal outcomes from our conceptual model (i.e., descriptive and injunctive norms) [20]. We identified proximal outcome measures from previous intervention research with college students [25–27]. The descriptive norms scale (α _t1 = 0.84) measured students’ perceptions of the prevalence of behaviors among college students (e.g., “In the past 30 days, approximately what percentage of college students do you think have [done the following]?”). We computed the average of four items about: alcohol use, heavy alcohol use, sex with alcohol, and hookups with alcohol, using a 10-point scale (1 = 0%–10%, 2= 11%–20%, etc.). The values for descriptive norms ranged from 1 to 10, with larger values indicating higher perceived prevalence of behaviors.

The injunctive norms scale (α  _t1 = 0.79) assessed students’ perceived peer acceptance of behaviors among college students (e.g., “In your opinion, how do most college students feel about other college students [done the following]?”). We computed the average of 4 items about: alcohol use, heavy alcohol use, sex with alcohol, and hookups with alcohol, using a 4-point Likert scale (1 = strongly disapprove to 4 = strongly approve). The values for injunctive norms ranged from 1 to 4, with larger values indicating higher perceived approval of behaviors.

Behavioral Outcomes

On the baseline and 60-day follow-up surveys, students answered behavioral questions related to alcohol and sex. Any and recent (past 30 days) alcohol use was captured (i.e., How often have you drunk alcohol?) with responses ranging from never to 10 or more days out of the past 30 days. In addition, heavy episodic alcohol use or “binge drinking” was captured by asking students to report the number of times in the past 2 weeks that they had 4 (women)/5 (men) or more drinks within a two-hour period. Students also reported if they had ever engaged in oral, vagina, or anal sex. If they reported any engagement in sexual behaviors, students then responded to questions about their most recent sexual event. They reported (a) if they had condom and/or other birth control method protected vaginal or anal sex, (b) if they had used alcohol before or during this sexual encounter, and (c) if their partner had used alcohol before or during this sexual encounter. Students then reported if they had ever had a hookup, defined for the students as: “penetrative [vaginal and/or anal] sexual behaviors with someone with whom you are not in a committed relationship [friends with benefits] or with someone you just met [one night stand].” Students who reported engaging in a hookup were asked to report on (a) condom use (if they reported vaginal or anal sex during their most recent hookup), (b) self alcohol use and (c) partner alcohol use during their most recent hookup. Notably, students who reported alcohol use as “unknown” for self and/or partner were included in the alcohol use category for both variables.

We were unable to find an existing measure that captured the intersection of alcohol use and sexual behaviors, perhaps due to the lack of interventions targeting these two behaviors jointly. Therefore, we used students’ responses to the behavioral items (described above) to create two ordinal outcome measures—one for sexual events and one for hookups. The advantages of these ordinal measures include that the measure allowed us to measure the behavioral outcome for all students (not just those who had sex before) and that it captured the increasing amount of risk that occurs when alcohol use is combined with condomless sex. In both cases, a higher value indicated that the student engaged in multiple higher risk behaviors.

For the sexual event behavioral outcome, we assigned students a score of 0 if they did not report ever having sex (no risk). Among those who reported ever having sex, we assigned students a score of 1 if they had used a condom during their most recent vaginal or anal sexual encounter and if neither they nor their partner had used alcohol (minimal risk, due to condom use and no alcohol use). We also assigned a score of 1 to students who had engaged in sexual behaviors other than vaginal or anal sex during their last sexual encounter (e.g. oral sex) and neither they nor their partner had used alcohol. We assigned students a score of 2 if they reported that either they or their partner had used alcohol (e.g., the student had oral sex or condom protected vaginal or anal sex, but the use of alcohol increased the likelihood of condom misuse or compromised decision-making). We assigned students a score of 3 if they reported condomless vaginal or anal sex without alcohol use by themselves or their partner. We assigned students a score of 4 if they reported condomless vaginal or anal sex and either they or their partner used alcohol (highest risk) as alcohol use has the potential to compromise decision making (e.g., having sex with a partner that you would not have under other circumstance and/or increase condom misuse) thus increases risk of STI acquisition [28]. We created the hookup behavioral outcome measure using the same algorithm.

Outcomes Pertaining to Assessment of the Performance of the SVP Component

To assess the performance of the SVP component, we used a series of items measuring students’ confidence in reducing/preventing sexual violence (e.g., “I feel confident in my ability to…”). Five items assessed confidence to: intervene when someone is using sexually inappropriate language, recognize signs of disinterest in someone else, recognize enthusiastic consent, seek consent from a partner, and know where to go get help if sexual violence is experienced. In addition, three items assessed students’ perceived effectiveness of strategies to reduce/prevent sexual assault (e.g., “Rate the level of potential effectiveness of the following strategy: (1) examining/assessing the environment, (2) looking for support, and (3) recognizing limits”). All eight items used a 4-point Likert scale (1 = not confident/effective to 4 = very confident/effective) and were analyzed individually.

Analytic Plan

As is the standard in intervention optimization research and factorial analysis of variance [24, 29], we defined main effects and interactions using effect coding (−1, 1). The calculation of main effects and the interaction effect are described in Table 1. To handle missing data, we used multiple imputation (100 imputations). This strategy had the advantage of making it straightforward to keep the missing data model identical across all the analyses reported here. We used PROC GLM and PROC MIANALYZE within the SAS 9.4 statistical software to conduct regression analyses with the imputed data.

We conducted three sets of analyses. First, we conducted analyses using proximal outcomes (i.e., descriptive and injunctive norms) at immediate posttest and 60-day follow-up as the outcome variables. Next, we conducted analyses using individual behavioral outcomes (i.e., condom use and binge drinking) at the 60-day follow-up as the outcome variables. These outcomes are only applicable to participants who have used alcohol (binge drinking) and those who have engaged in sex (condom use), so our analyses for binge-drinking excluded participants who reported no alcohol use at both assessments and our analyses for condom use excluded participated who reported no sex at both assessments. Since binge drinking is a count variable, we used a negative binomial regression model to assess the effects on binge drinking and since condom use is a binary variable, we used a logistic regression model for this outcome. Finally, we conducted analyses using the intersection behaviors at the 60-day follow-up as the outcome variables. Given that the intersection variables are ordinal, we conducted both ANOVA and ordinal logistic regression analyses. The results were similar, so we report the ANOVA results as they are easier to interpret. In all of our models, we used itMatters, SVP, and the interaction between itMatters and SVP as independent variables, with the corresponding pre-test measures of the outcome variables of interest (i.e., proximal or behavioral outcomes) as a covariate. Including the effects of SVP (and its interaction with itMatters) allowed us to test whether there were any iatrogenic main effects of the SVP component on the outcomes as well as whether there were any synergistic or antagonistic interactions between SVP and itMatters.

To explore whether there were differential effects of itMatters on the intersection behaviors for participants who had ever used alcohol (vs. those who had not) or for those who reported any sexual activity (vs. those who had not), we conducted several sensitivity analyses. Specifically, for each of the intersection behaviors, we conducted an ANOVA using itMatters, a binary indicator for whether the participant reported any alcohol use at baseline and their interaction as independent variables, and the baseline intersection behavior as a covariate. We repeated these analyses using a binary indicator for whether participant reported any sexual activity at baseline (and its interaction with itMatters).

For SVP, we conducted analyses using proximal outcomes (i.e., self-efficacy and protective behavior strategies) at immediate posttest and 60-day follow-up as the outcome variables. Similar to our proximal outcome models for itMatters, these SVP models included itMatters, SVP, and the interaction between itMatters and SVP as independent variables, with their respective pre-test proximal outcome measure as a covariate.

Results

Sample Characteristics and Behaviors

Of the students who completed the baseline survey, the majority (88%) were 18 years old. Demographic characteristics are reported in Table 2. Most of the sample identified as female (64%), heterosexual (83%), and White (58%). Nearly two-thirds (64%) reported living on-campus in a residence hall, whereas about one quarter (26%) reported living at home with parents or family at the time of survey completion.

Table 2.

Demographic characteristics at baseline (N = 3,098)

	N	%
Sex
Male	1,060	34.2
Female	1,986	64.1
Transgender	15	0.5
Other	22	0.7
Sexual orientation
Heterosexual	2,581	83.3
Homosexual	95	3.1
Bisexual	299	9.7
Other	53	1.7
Race
White	1,802	58.2
Black	590	19.0
Multiracial	230	7.4
Other	329	10.6
Hispanic	564	18.2
Transfer student	563	18.2
Housing
On-campus residence hall	1,993	64.3
Off-campus apartment/house	238	7.7
At home with parents/family	810	26.1
Other	30	1.0

Participants did not provide data for all variables, so sample size varies.

Table 3 summarizes participants’ reported behaviors at baseline and 60-day follow. At baseline, just over half of the students (51.7%) reported that they ever used alcohol. In addition, 71% reporting no alcohol use in past 30 days. Similarly, just under half of the students (46.8%) reported that they had ever engaged in vaginal and/or anal sexual behaviors. Among those who reported having engaged in sex at baseline (n = 1,451), nearly 45% reported known condom use, 53.6% reported use of birth control other than a condom, and 9.4% reported alcohol use by self or partner during most recent sex. Of those who reported alcohol use during their most recent sexual event (n = 137), most (n = 83) included known condom use. At baseline, over a quarter (26.8%) of the sample reported having engaged in a hookup (n = 829), and of those 28.8% reported known condom use and 15.7% reported alcohol use by self or partner during the most recent hookup. There was no statistically significant difference between the proportions of these behaviors reported at the 60-day follow-up compared to baseline.

Table 3.

Behaviors at baseline and 60 day follow-up

	Baseline (N = 3098)		60 day follow-up (N = 1836)
	N	%	N	%
Alcohol use, ever
Yes	1,601	51.7	964	52.5
No	1,325	42.8	810	44.1
Sexual behaviors, ever
Yes	1,451	46.8	872	47.5
No	1,456	47.0	893	48.6
Known condom use during most recent sexa	644	44.4	405	46.4
Used birth control (besides condom) during most recent sexa	778	53.6	496	56.9
Any alcohol use during most recent sexa	137	9.4	99	11.4
Most recent sex with alcohol use and no known condom usea	54	3.7	32	3.7
Hookup, ever
Yes	829	26.8	472	25.7
No	2,085	67.3	1,292	70.4
Known condom use during most recent hookupb	239	28.8	169	35.8
Any alcohol use during most recent hookupb	130	15.7	94	19.9
Most recent hookup with alcohol use and no known condom useb	31	3.7	22	4.7

Participants did not provide data for all variables, so sample size varies.

^aPercentages for these variables calculated out of participants who had had sex in their lifetime (N = 1,451 for baseline, N = 872 for follow-up).

^bPercentages for these variables calculated out of participants who had had hookup in lifetime (N = 829 for baseline, N = 472 for follow-up).

Evaluation Results for itMatters

Table 4 presents results pertaining to the itMatters evaluation; recall that we coded the proximal and behavioral outcome measures such that a negative coefficient reflects the desired outcome. There was a significant main effect of itMatters on the proximal outcomes of descriptive (d = 0.53) and injunctive norms (d = 0.84) at the immediate post-intervention survey. These effects were maintained at the 60-day follow-up (d = 0.46 and d = 0.45, respectively). There was a significant effect of itMatters on binge drinking; itMatters significantly reduced binge drinking at the 60-day follow-up (B = −0.10, 95% CI = −0.20, −0.01). Note that there is no accepted way to compute Cohen’s d from a negative binomial analysis, which is why we provide the 95% CI instead.

Table 4.

Model coefficients (p-value) on outcomes from the evaluation of itMatters assessed at immediate and 60 day follow-up

	Proximal outcomes				Individual behavioral outcomesa		Intersection behavioral outcomes
	Descriptive norms		Injunctive norms		Binge drinkingb	Condom use	Intersection of alcohol and sex	Intersection of alcohol and hookups
	Immediate follow-up	60 day follow-up	Immediate follow-up	60 day follow-up	60 day follow-up	60 day follow-up	60 day follow-up	60 day follow-up
itMatters Main	−0.334	−0.349	−0.226	−0.115	−0.102	0.053	−0.003	−0.004
Effect	(<.001*)	(<.001*)	(<.001*)	(<.001*)	(0.04*)	(0.57)	(0.87)	(0.80)
SVP main effect	0.131	0.041	−0.021	−0.011	−0.019	0.123	−0.024	−0.011
	(<.001*)	(0.21)	(0.06)	(0.33)	(0.70)	(0.18)	(0.20)	(0.42)
itMatters × SVP	−0.018	0.027	0.032	0.016	0.041	0.019	−0.006	−0.004
Interaction	(0.47)	(0.39)	(0.004*)	(0.15)	(0.40)	(0.83)	(0.72)	(0.76)

^aBinge drinking and condom use analyses were conducted on multiple-imputed data from participants who reported any alcohol use (N = 1,787) or sexual activity (N = 1,614), respectively, at either baseline or 60-day follow-up.

^bThese results are from a negative binomial analysis for count data, based on 97 imputations, as the model did not converge for 3 of the imputations. Notably, it could be questioned whether it was acceptable to impute this variable with a normal distribution and then round to the nearest integer to produce count data. We ran the model with non-imputed data and found the same results.

*Indicates p values <.05 and corresponding baseline survey responses for each outcomes were included as covariates in the analytic models.

In terms of potential effects of the SVP component on the outcomes, there was a significant main effect of SVP on descriptive norms at the immediate follow-up (d = 0.20) in the undesired direction (i.e., an iatrogenic effect); this effect was not retained at the 60-day follow-up. There were no significant itMatters × SVP interaction effects on descriptive norms at either the immediate or 60-day follow-up, but there was a significant antagonistic interaction effect on injunctive norms at the immediate follow-up (d = 0.12). (An antagonistic 2-way interaction occurs when the presence or higher level of one candidate component is associated with reduced effectiveness in another [19]). There were no significant effects on behavioral outcomes from SVP or the itMatters × SVP interaction at either time.

Finally, our sensitivity analyses showed no significant interaction between itMatters and ever using alcohol or engaging in sexual behaviors. This suggests that itMatters did not differentially affect intersection behaviors for participants who had engaged in any alcohol use or sexual behaviors at baseline.

Optimization Results for SVP Component

Table 5 presents the results pertaining to the performance of the SVP component; recall that we coded these measures such that a positive coefficient reflects the desired outcome. At the immediate follow-up, the SVP component had a significant main effect on the efficacy variables related to intervening when someone was using inappropriate language (d = 0.29) and knowing where to get help (d = 0.22). The SVP component also had a significant main effect on students’ use of two protective behavioral strategies: assess the environment (d = 0.29) and look for support (d = 0.23). None of the effects were significant at the 60-day follow-up (data not presented). There were no significant main effects of itMatters and no significant itMatters × SVP interactions on the SVP proximal outcomes.

Table 5.

Optimization of the SVP component: factorial ANOVA results of the effect of candidate components on proximal mediators assessed at immediate follow-up (p values)

Proximal outcomes
	Self-efficacy					Protective behavior strategies
	Ability to intervene when inappropriate language used	Ability to recognize signs of disinterest/discomfort	Ability to recognize enthusiastic consent	Ability to seek consent	Know where to get help	Assess environment	Look for support	Recognize limits
SVP Main	0.066	0.019	0.029	0.004	0.068	0.090	0.067	0.042
Effect	(<.001*)	(0.08)	(0.004*)	(0.72)	(<.001*)	(<.001*)	(<.001*)	(<.001*)
itMatters Main	−0.006	−0.003	−0.012	−0.012	−0.012	0.019	0.011	0.008
Effect	(0.63)	(0.76)	(0.26)	(0.27)	(0.37)	(0.14)	(0.35)	(0.50)
itMatters × SVP	0.018	−0.006	0.007	−0.010	0.009	−0.009	−0.009	−0.012
Interaction	(0.18)	(0.57)	(0.50)	(0.34)	(0.48)	(0.48)	(0.45)	(0.30)

*Indicates p value <.05; Baseline responses of the proximal mediators were included as covariates in the analytic models.

Discussion

Evaluation of the itMatters Intervention

We found that the itMatters intervention significantly reduced descriptive and injunctive norms related to the intersection of alcohol and sexual behaviors as well as significantly reduced binge drinking. However, we did not find significant effects on behavioral outcomes related to either condom use or the intersection of alcohol and sex. At first glance, these results suggest that although the optimized intervention successfully reduced one risky behavior (i.e., binge drinking), it was not effective at reducing behaviors related to the intersection of alcohol and sex (the primary target of our intervention). Importantly, however, there are several potential alternative explanations.

One alternative explanation is that very few students reported engaging in alcohol use and sexual behaviors simultaneously (only n = 137 and n = 130 of the 3,098 students in our baseline sample reported alcohol use during sexual and hookup events, respectively). These very low observed base rates are particularly challenging as our intervention is designed to prevent sexual behaviors in the context of alcohol, and any evaluation of a prevention intervention, particularly one related to substance use, relies on observation of the comparison group engaging in the “bad” behaviors that are being prevented. We expect that one reason our base rates were so low is we had fewer “higher risk” students in our sample compared to the general college student population. Consistent with this possibility, we found lower base rates of alcohol use and sexual behaviors in our study compared to other studies of first year college students (e.g., 51.7% of students in our sample reported ever drinking compared to 58.4% from a 2015 study [30]; 46.8% of students in our sample reported ever having vaginal sex compared to 66.5% in the past year from a 2014 study [31]). This possibility is consistent with other research showing that “higher risk” students are more difficult to recruit and retain as research participants [32]. Notably, we attempted multiple and diverse recruitment and retention strategies (e.g., different incentive plans and student ambassadors, to be described in a forthcoming manuscript), but our experience suggests that additional innovative strategies are needed to improve the representativeness of college student samples to help ensure accurate assessment of the effectiveness of interventions on behavioral outcomes. These low base rates may also reflect the timing of our data collection—in order to prevent risk behaviors before they began, we conducted our study very early in the fall semester (in fact, two of the four schools in our sample surveyed students prior to matriculation, when students may have had fewer opportunities to engage in alcohol use and sexual activity then they would once they started school).

A second alternative explanation is that our conceptual model, which specifies that changing descriptive and injunctive norms will change behavior, is incorrect. Given the substantial literature describing a complex relationship between norms and behaviors [33–37], we do not believe this is the case. However, it is possible that norms have a more delayed effect on behavior; most conceptual models, including ours [20], do not specify the expected time frame for the uptake of behaviors or associated outcomes (e.g., STI) to occur. The final follow-up survey was administered approximately 60 days after the intervention which suggests that if there is indeed a delayed effect, it may take more than 60 days to emerge. Thus, the evaluation of norms-based interventions to promote sexual health among college students may need to include a much longer-term follow up (e.g., in the sophomore year) and/or require booster sessions to see an effect on distal behavioral outcomes [38].

Lastly, students who completed itMatters viewed alcohol use with sex as less prevalent and less approved of at both follow-up surveys compared students who did not complete itMatters. It is possible, however, that they did not internalize these norms, which could explain how changes in norms did not translate into changes in behavior. Thus, changes in norms could operate similarly to changes in knowledge: students might have learned factual information and reported it back correctly, but not internalized this information in a way that changed their behavior. Future work should explore this possibility, as it has important implications for norms-based interventions.

Assessment of the Effectiveness of the SVP Component

Using the same a priori effect size criterion as in our previous optimization trials [18], we found that the SVP component had an immediate effect on some of the proximal outcomes, but these results disappeared by the 60-day follow-up. Interestingly, SVP significantly increased descriptive norms (i.e., moved outcome in the undesired direction) and the combination of itMatters and SVP content significantly increased injunctive norms (i.e., antagonistic interaction, which moved outcome in the undesired direction). These results may be partially explained by the differences between the sex-positive messaging of the itMatters content and a focus on the potential “dangers” of sex through a SVP lens. Accordingly, consistent with the MOST framework, we will revise and retest the SVP component before including it in the itMatters intervention. In future optimization trials it may be useful to treat the three Clery Act [22] foci—educational (e.g., prevalence), informational (e.g., campuses’ responsibilities), and behavioral (e.g., how to be an active upstander)—as individual components, as we did during the optimization phase for itMatters, to test the individual and interactive contributions of each component on the outcomes. Overall, the results underscore how the hybrid evaluation-optimization trial let us identify not only whether the SVP component worked (i.e., main effect) but also how it worked in the presence of the existing intervention content (i.e., interaction effect).

The Hybrid Evaluation-Optimization Trial

The present article describes an experiment that, to our knowledge, is the first to combine evaluation of an optimized intervention with assessment of the performance of a component under consideration for addition to the optimized intervention. Factorial optimization trials conducted within the MOST framework often examine many components simultaneously [39–45]; our own optimization trials each examined five components [18]. Thus, it is possible to examine the performance of several components in a hybrid evaluation-optimization trial; for example, a 2⁵ experiment could include one factor representing evaluation of the optimized intervention and four factors examining new components under consideration for addition to the intervention package. This approach has the potential to speed up the process of continual optimization considerably by enabling the evaluation trial and the optimization trial to be conducted simultaneously. Any components that were found to be ineffective in the optimization trial could be redesigned, and then tested in the next experiment in which the optimized intervention is evaluated. Alternatively, the results of the optimization trial may lead to revisions to the conceptual model that in turn suggest components representing new intervention strategies; these also can be examined. Moreover, the hybrid evaluation-optimization trial generally uses fewer experimental participants compared to separately conducting an RCT to evaluate the intervention package and an optimization trial to assess the performance of the candidate components.

It should be noted that it is not always appropriate to integrate the evaluation and optimization trials. For example, in the research reported here we were considering adding the SVP component to the optimized itMatters intervention, as opposed to replacing a component already in itMatters with the SVP component. If the components under consideration are intended as replacements for components currently in the optimized intervention, then the hybrid approach is not appropriate. However, when appropriate the hybrid evaluation-optimization trial can be very efficient and economical.

Limitations

Our findings should be interpreted in the context of several limitations. First, the percentage of the eligible student population who completed the baseline survey was relatively low (23.3%), as were the overall base rates of behaviors (alcohol and sex independently and in conjunction). Although universal online behavioral interventions have the capability for broad reach [46–48], without a completion mandate imposed by the university they may not reach the students at highest need (e.g., heavy drinkers and those who use condoms inconsistently) or those who do not read their emails. Second, our surveys only asked about participants’ “most recent” sexual event and did not specify the timeframe on which this event occurred. It is possible that some of the events reported in the follow-up survey actually occurred pre-intervention, diluting the meaning of our outcome variable. In other words, some students who reported alcohol use during their most recent sexual encounter at both times may not have had sex between the pre-test and follow-up surveys (i.e., they had changed their behavior dramatically), but the results would look like they had not changed their behavior at all. Third, as mentioned above, norms may have a delayed effect on behavior and therefore a longer-term follow-up survey (e.g., 6 months) may have captured such effects. Finally, although universities were geographically diverse, the overall study sample was predominantly White.

Conclusions

This study demonstrated that itMatters effectively changed proximal and behavioral (binge drinking) outcomes. We did not observe effects on sexual behaviors in the context of alcohol use, possibly due to the noted study limitations. Future research must consider strategies for improving participation in intervention research to gather data on a more representative sample, especially from students with elevated likelihood of engaging in behaviors of interest including alcohol use and sexual behaviors of college students. Furthermore, there may be a need to extend follow-up data collection into the sophomore year to assess behavior change over longer periods of time. Preliminary evidence on proximal outcomes suggests that a revised SVP component may have the potential to demonstrate sufficient effectiveness to merit inclusion as an active component in the itMatters intervention. This study is unique in that it used a hybrid evaluation-optimization trial, which enabled us to both evaluate the optimized intervention package, itMatters, and assess the effects of a SVP component that we were considering for inclusion in itMatters. The hybrid evaluation-optimization design, which greatly improved time and resource efficiency, could be used to even greater advantage by including several candidate components for testing. MOST is a particularly important tool for developing, optimizing, and evaluating health promotion interventions [19, 20, 40, 49–51].

Funding

This work was supported by the National Institute on Alcohol Abuse and Alcoholism [R01 AA022931, PI: L.M.C.], National Institute of Drug Abuse [P50 DA039838, PI: L.M.C.], and the National Center for Advancing Translational Sciences [UL1 TR000127 and TR002014; PI: L. Sinoway]. The content of this work is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Compliance With Ethical Standards

Authors’ Statement of Conflict of Interest and Adherence to Ethical Standards The authors declare that they have no conflict of interest.

Authors’ Contributions Amanda E. Tanner conceptualized the approach, and drafted and revised the manuscript. Kate M. Guastaferro assisted with conceptualization and drafting. Kelly L. Rulison supervised data analyses, conducted data interpretation, and assisted with drafting. David L. Wyrick assisted with literature review and drafting, and provided editorial commentary. Jeffrey J. Milroy collected data, assisted with literature review, and provided editorial commentary. Sandesh Bhandari conducted the analyses and assisted with drafting. Shemeka Thorpe collected data and assisted with literature review. Samuella Ware collected data and assisted with literature review. Linda M. Collins assisted with conceptualization and data interpretation, and provided editorial commentary.

Ethical Approval All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent All procedures, including the informed consent process, were conducted in accordance with the ethical standards of the responsible human subjects committee and with the Helsinki Declaration of 1975, as revised in 2000.