Examining Engagement Effects in an Adaptive Preventive Intervention for College Student Drinking

Abstract

Objective:

This study determined characteristics of engagement and whether engagement in an adaptive preventive intervention (API) was associated with reduced binge drinking and alcohol-related consequences.

Method:

Incoming students were recruited for a sequential multiple assignment randomized trial (SMART; N=891, 62.4% female, 76.8% non-Hispanic white) with an assessment-only control group. The API occurred during the first semester of college with outcomes assessed at the end of the semester. The API involved two stages. Stage 1 included universal intervention components (personalized normative feedback [PNF] and self-monitoring). Stage 2 bridged heavy drinkers to access additional resources. We estimated the effect of engagement in Stage 1 only and in the whole API (Stages 1 and 2) among the intervention group, and the effect of the API vs. control had all students assigned an API engaged, on alcohol-related outcomes.

Results:

Pre-college binge drinking, intention to pledge a fraternity/sorority and higher conformity motives were most associated with lower odds of Stage 1 engagement. Action (readiness to change) and PNF engagement were associated with Stage 2 engagement. API engagement was associated with significant reductions in alcohol-related consequences among heavy drinkers. Compared to control, we estimated the API would reduce the relative increase in alcohol-related consequences from baseline to follow-up by 25%, had all API students engaged.

Conclusions:

Even partial engagement in each component of the “light-touch” API rendered benefits. Analyses suggested that had all students in the intervention group engaged, the API would significantly reduce the change in alcohol-related consequences over the first semester in college.

Introduction

College student drinking is a public health concern (Hingson et al., 2009; Hingson & White, 2014; Patrick et al., 2022) with 50.1% of 18–25-year-olds using alcohol in the past month and 29.2% binge drinking in the past month (i.e., 4+/5+ drinks in a row for females/males) in 2021 (Substance Abuse and Mental Health Services Administration [SAMHSA], 2022). Males, non-Hispanic white individuals, and Greek members are at greater risk of engaging in binge drinking (see White & Hingson, 2013 for a review). Furthermore, first year students are at particularly high risk for alcohol-related consequences (Borsari et al., 2007; Scott-Sheldon et al., 2014). Alcohol use consequences include sexual assault, injuries, lower grades, and death (White & Hingson, 2013). Given risks to health and safety of self, others, and institutions, prevention and intervention efforts are necessary.

Prevention and intervention methods (e.g., personalized normative feedback, self-monitoring, Web-BASICS) exist and are efficacious (Carey et al., 2016; Larimer et al., 2022; Murphy et al., 2022; National Institute for Alcohol Abuse and Alcoholism [NIAAA], 2019; Scott-Sheldon et al., 2014; Tanner-Smith et al., 2022) but are underused (Czyz et al., 2013; Helle et al., 2022; Wu et al., 2007). Web-based technology may be an avenue to increase use of such methods given that most youth access the majority of their health information from online sources (Escoffery et al., 2005; Horgan & Sweeney, 2010; Kazemi et al., 2017; Schwartz et al., 2014). Prevention and intervention strategies can be further maximized by using a sequential approach of providing a universal intervention to everyone in a given population, identifying non-responders to the universal intervention, and then using a more intensive or motivational method to target non-responders (Collins et al., 2004). This approach ideally maximizes benefits in a cost-effective way and increases the reach and scalability of alcohol use interventions (McKay, 2009; Nahum-Shani et al., 2017). Although use of adaptive preventative interventions (APIs) may not be warranted for all students, those in high-risk populations (i.e., those intending to pledge Greek sororities or fraternities; Patrick et al., 2021) may benefit from such an approach.

APIs, like other interventions, are frequently examined through randomized controlled trials to determine efficacy. Such trials are typically analyzed according to the intention-to-treat principle, which analyzes all participants according to their randomized group assignment even when they fail to engage with their assigned intervention (Sheiner & Rubin, 1995; Gupta, 2011; Piantadosi, 2017). Thus, the analyses estimate the effect of being assigned to a particular intervention condition, not necessarily the effect of the intervention itself (Hernán & Robins, 2017). However, in reality, the effect of the intervention likely depends on the pattern of participant engagement. When participant engagement is incomplete or varies, only considering intention-to-treat results can potentially be misleading because effective interventions can appear ineffective due to low engagement. This may be especially true in web-based interventions where low engagement rates are typical (Kelders et al., 2012), potentially leading to lessened intervention effects (Donkin et al., 2011; Manwaring et al., 2008). Indeed, there have been calls to examine the impact of engagement on outcomes in web-based interventions (e.g., Lee et al., 2010). Intention-to-treat analysis is useful because it may reflect the treatment noncompliance and deviation that occur in real-world settings and avoid overoptimistic estimates (Gupta, 2011); however, assessing whether those who engage in the intervention experience positive outcomes can enhance our understanding of intervention efficacy.

Patrick et al. (2020; 2021) designed a two-stage sequential multiple assignment randomized trial (SMART). Comparing students who were randomized to API vs. control (i.e., intention-to-treat analysis), there were no significant main effect differences in binge drinking frequency or alcohol consequences (Patrick et al., 2021). However, there were indications that some students benefited more than others. Students planning to join fraternities and sororities who were randomized to the API (vs. control) reduced their binge drinking (Patrick et al., 2021). Additionally, early intervention (i.e., before college started) reduced binge drinking frequency for alcohol-using students who reported conformity and social reasons for alcohol use, and the late intervention (i.e., after college started) reduced binge drinking frequency for alcohol-using students who reported drinking for enhancement and coping reasons (Lyden et al., 2022). Although the intention-to-treat analysis suggested the intervention was not efficacious on average, it was unknown whether this was because the intervention was not effective or because not enough people engaged with it.

If level of engagement corresponds to decreases in alcohol use or consequences, then interventions can be improved to enhance the likelihood of engagement following principles of implementation science (Becker et al., 2021; Bauer & Kirchner, 2020). Previous studies have found that increased engagement predicts positive health-related behavior changes, including reductions in drinking (Castro et al., 2017; Donkin et al., 2011; Carrà et al., 2017; Linke et al., 2007; Norman et al., 2019) and higher likelihood of follow-up responses (Eysenbach, 2005; Coughlin et al., 2021). However, no work has examined the impact of engagement on outcomes of an intervention for college student alcohol use in a methodological and empirical way. Most work has investigated engagement as the outcome of interest (e.g. subject or intervention characteristics associated with engagement). For example, older individuals, women, and those with higher levels of education tend to demonstrate greater engagement (Beatty & Binnion et al., 2016; Garnett et al., 2018; Murray et al., 2013; Nordholt et al., 2020). There is some evidence that binge drinking predicts nonengagement (Castro et al., 2017). If engagement varies based on these or other characteristics, then interventionists can identify those individuals who are engaging—and potentially benefiting—from the intervention, as well as those who need more support to increase engagement to bolster intervention effects.

Building on the previous analysis of the API’s main outcomes (Patrick et al., 2021), the current study examined engagement in the API by addressing three related aims. Aim 1 was to identify characteristics associated with engagement in the intervention. We examined engagement in Stage 1 components (personalized normative feedback (PNF), self-monitoring (SM), or both) based on demographics, mental health, and alcohol-related variables. We also examined engagement with the assigned Stage 2 bridging strategy based on the same factors, as well as Stage 1 timing and Stage 1 engagement. Aim 2 was to examine the effect of engagement (Stage 1 only and both stages) on binge drinking and alcohol-related consequences among intervention participants. Aim 3 was to examine the effect of engaging in the API (both stages) compared to the control group.

Method

Participants

The M-bridge SMART study recruited incoming first-year college students (N=891) from a large Midwestern university to participate in a universal and resource-efficient adaptive preventive intervention (API). Using the Registrar’s list, 1,500 incoming first-year students were invited to participate in the study. Participants were randomly assigned to an assessment-only control condition in which they did not receive any API (1/3 of enrolled students [N = 300]) or the intervention condition in which they did receive the API (2/3 of students [N = 591]).

All students (N=891) completed surveys in August 2019 (baseline) and at the end of first semester (December 2019). Participants received a pre-incentive ($5) with the mailed recruitment letter, $25 for completing the baseline survey, $2 for completing biweekly self-monitoring surveys, and $30 for completing the follow-up survey. Additionally, two drawings were conducted at the end of the first semester that provided the opportunity to win one of four $500 drawings (see Patrick et al., 2020 or Patrick et al., 2021 for further information on the procedures for this study). This study was approved by an institutional review board.

Adaptive Preventive Intervention (API) Protocol

There were two stages of the API (a full description of the protocol can be found in Patrick et al., 2020). In Stage 1, the 591 students assigned to the API group were randomized 1:1 to begin a universal intervention early (before classes started in August 2019) or late (after classes started in September 2019). The universal intervention involved two components: web-based personalized normative feedback (PNF) and biweekly self-monitoring (SM) surveys. Once a student reported heavy drinking (defined as those who reported 2+ occasions of binge drinking [4+/5+ drinks in for females/males] or 1+ occasion of high-intensity drinking [8+/10+ drinks for females/males]) in their biweekly SM survey (completed in either September or October), they began Stage 2 of the API. At Stage 2, students were re-randomized 1:1 to a bridging strategy that consisted of either an automated resource email or invitation to interact with an online health coach. Both the resource email and the health coach invited students to complete the online version of the evidence-based Brief Alcohol Screening and Intervention for College Students (Web-BASICS; LaBrie et al., 2013), which provides personalized feedback about alcohol use, norms, and consequences of drinking. Stage 2 ended in November 2019. Thus, there were four possible APIs: early/coach, early/email, late/coach, and late/email because of the two randomizations in Stage 1 and Stage 2. All 591 students randomized to an API could engage with the universal intervention (PNF and SM), while only students who were flagged as heavy drinkers could engage with the bridging strategy (online health coach or resource email).

Measures

Alcohol-Related Measures

Binge drinking was assessed at baseline (pre-college) and follow-up by asking participants the number of occasions they consumed 4+/5+ drinks in two hours for females/males in the past 30-days. Response options were “none,” “once,” “twice,” “3–5 times,” “6–9 times,” and “10 or more times.” Pre-college binge drinking was dichotomized to “Never” and “Some” because 72% of API students reported no binge drinking in the past 30-days so we collapsed the categories Follow-up binge drinking was based on the number of occasions in the past 30 days, and Mid-semester binge drinking (reported during self-monitoring) was assessed by asking participants how many times they had 4+/5+ drinks in a row in the last two weeks; both were coded using the midpoint of each response option, as: 0=“none,” 1=“once,” 2=“twice,” 4=“3 to 5 times,” 7=“6 to 9 times,” and 10=“10 or more times”) and analyzed as a count variable as in the original main outcomes paper (Patrick et al., 2021).

The Brief Young Adult Alcohol Consequences Questionnaire (B-YAACQ; Read et al., 2006) was used to assess alcohol consequences (e.g., doing embarrassing things, hangover) in the past 30-days at baseline and follow-up. The 24 items were summed to create an alcohol consequences total score and was analyzed as a count outcome.

Definitions of Engagement

Engagement in an API component was determined by whether a student interacted with the component or not. Indicators of engagement included engaging in Stage 1 universal components (PNF, SM, or both), engaging in the assigned Stage 2 bridging strategy for indicated students, or engaging in the whole API (both stages, if eligible). During Stage 1, students randomized into the API group (N=591) were eligible to engage in the PNF and SM components of the API. PNF engagement was assessed via electronic paradata. Levels of engagement in the Stage 1 PNF component were viewing zero pages, viewing 1–4 pages, or viewing all five pages of feedback that were sent to the student’s email. PNF engagement was coded as any or none based on whether the student viewed at least one of the five pages of feedback sent via email. SM engagement was determined by whether students filled out no surveys, filled out at least one of the 4 possible surveys, or completed all surveys required of them (up to 4; students stopped taking the surveys once they were flagged as a heavy drinker). SM engagement was coded as completing at least one of the possible four bi-weekly surveys. Stage 1 engagement was based on engaging in both PNF and SM, as defined above.

Eligibility for Stage 2 was based on heavy drinking (determined by SM responses); heavy drinking students (N=158) were re-randomized to a bridging strategy (i.e., a resource email or online health coach). Stage 2 engagement was assessed via electronic paradata. Engagement with health coaching was not accepting the online health coaching invitation, opening the online portal but responding to zero messages, or sending between 1 to 21 messages to the health coach in the portal. For the resource email bridging strategy, engagement was not clicking the link, opening and partially completing the screening (ranging from 1–85% completion), or completing 100% of the online screening. Stage 2 engagement was coded as either (a) accepting the invitation from the health coach and sending at least one online chat message to the health coach, or (b) clicking the link to Web-BASICS in the resource email.

Engagement in the whole API was determined based on a student’s eligibility and completion. For non-heavy drinkers, whole API completion was defined as completing Stage 1 (i.e., the only stage for which they were eligible). For heavy drinkers, whole API completion was defined as completing Stages 1 and 2, as defined above.

Other Measures at Baseline

Demographics.

Participants were asked to provide demographic information such as biological sex, race/ethnicity, pre-college intention to pledge Greek life, parents’ highest level of education, and family history of drinking problems.

Mental Health.

The Patient Health Questionnaire-9 was used to assess depression (PHQ-9; Kroenke et al., 2001). Response options were 0=“Not at all” to 3=“Nearly every day.” Responses from the first eight items were averaged to create mean scores. Suicidal ideation was measured using the ninth item in the PHQ-9 (i.e., “Thoughts that you would be better off dead or of hurting yourself”). Anxiety was assessed using the General Anxiety Disorder-7 (GAD-7; Spitzer et al., 2006). Response options were 0=“Not at all” to 3=“Nearly every day.” The seven items were averaged to create mean scores. Perceived stress was measured using the Perceived Stress Scale (PSS; Cohen et al., 1983) and assessed past month stress with response options 0=“Never” to 4=“Very often.” Responses were averaged for a mean score.

Alcohol-Related Attitudes/Beliefs.

To measure perceived norms of binge drinking, participants were asked “what percent of University of Minnesota first-year students do you think had five or more drinks in a sitting?” where those who identified as “female” were asked what percent of females they thought had five or more drinks in a sitting. Drinking motives were assessed using the Drinking Motives Questionnaire-revised (Cooper, 1994), which consists of 20 items and four subscales: Coping (e.g., To forget your worries), Social (e.g., To celebrate occasions with friends), Enhancement (e.g., Because it is fun), and Conformity (e.g., Because your friends pressure you to drink). Participants were asked to report how frequently they drink alcohol for each reason with response options “Almost Never/Never” (1) to “Almost Always/Always” (5). Responses were averaged to create a mean score for each subscale. Readiness to change was assessed with the 12-item Readiness to Change Questionnaire (Rollnick et al., 1992), which has three subscales: Pre-contemplation (e.g., I don’t think I drink too much), Contemplation (e.g., My drinking is a problem sometimes), and Action (e.g., I am trying to drink less than I used too). Participants were asked to identify how they felt about their drinking and whether they agree/disagree with the statements provided. Response options were “strongly disagree” (−2) to “strongly agree” (+2). Sum scores were created for each subscale and ranged from −8 to +8.

Stage 1 intervention timing was determined by whether the participant was randomized to begin the intervention “early” (i.e., before classes started in August 2019) or “late” (after classes started in September 2019) and was coded as binary.

Statistical Analyses

Aim 1: Predictors of API Engagement

Logistic regression models were fit to estimate the associations between baseline measures and the odds of engaging in PNF, SM, both Stage 1 components, and Stage 2 bridging strategy. We included demographics, baseline mental health, baseline alcohol-related attitudes/beliefs, pre-college binge drinking, and Stage 1 intervention timing as possible predictors of engagement in Stage 1 components. For predicting Stage 2 engagement among heavy drinkers, the assigned Stage 2 bridging strategy (email resource or online health coach), previous engagement in PNF and mid-semester binge drinking were added as additional predictors. We estimated the unadjusted odds ratios of engaging in Stage 1 and Stage 2 for all predictors. We used a data adaptive approach to select the best predictors of engagement, and those empirically associated were included in the models moving forward. To select the best predictors of engagement in PNF, SM, both Stage 1 components, and Stage 2, we used stepwise and forward regressions with AIC criteria and then estimated the adjusted odds ratios of engagement for the selected predictors.

Aim 2: Estimating the Effect of Engagement on Alcohol-Related Outcomes Among Those Assigned to the Intervention Condition

Two log-link Poisson regression models were estimated with robust standard errors. For model 1, we estimated the effect of engaging in Stage 1 on alcohol-related outcomes at follow-up (binge drinking and alcohol consequences) for students in the API group. The possible confounders we adjusted for were Stage 1 intervention timing and pre-college binge drinking (some vs. never), Greek life intention, and baseline perceived norm of campus binge drinking (selected predictors of Stage 1 engagement from the Aim 1 stepwise model). For model 2, we estimated the effect of engaging in the whole API (both the Stage 1 intervention and assigned Stage 2 bridging strategy) on alcohol-related outcomes at follow-up for students flagged as heavy drinkers. We adjusted for Stage 1 intervention timing, Stage 2 bridging strategy (online health coach vs. resource email) and pre-college binge drinking, baseline perceived norm of campus binge drinking, readiness to take action score, and readiness to contemplate change score (predictors that were significant in either the Stage 1 engagement or Stage 2 engagement stepwise models from Aim 1).

Aim 3: Estimating the Per-Protocol Effect of API vs. Assessment-Only Control on Alcohol-Related Outcomes in the Full Sample

The per-protocol effect is the effect that would have been observed had all students engaged with the whole API they were assigned to. However, this cannot be consistently estimated from a simple comparison of students who engaged in the API to the control group. The control and the API engagement groups are unlikely to be balanced in baseline characteristics because they are not generated from randomization (i.e., there was selection into intervention engagement for those who engaged). Inverse probability weighting (IPW) can correct for this bias (Hernán & Hernández-Díaz, 2012) and has been used extensively to correct for non-engagement in randomized trials (e.g., SMARTs; Cain & Cole, 2009; Hernán et al., 2006; Kasari et al., 2014; Naar et al., 2019; Robins & Finkelstein, 2000; Sabri et al., 2021) as well as observational studies (Nahum-Shani et al., 2012).

To analyze the effect of API (vs. control) on the alcohol-related outcomes, we used the same longitudinal generalized estimating equations (GEE) (Zeger & Liang, 1986) log-link marginal mean model with robust standard errors from the intention-to-treat analysis (refer to Patrick et al., 2021 for details). However, each individual who engaged in the API was inversely weighted by their predicted probability to engage in the API. Specifically, students who engaged in the API but were predicted to have a low probability of engagement were weighted more because these students were unlikely to engage but did, and they represent others with similar characteristics. Students who engaged in the API but were predicted to have a high probability of engagement were weighted less and students who were randomized to an API but did not engage have a weight of 0. Weights are time-varying weights and the probabilities of engagement in Stage 1 and Stage 2 were predicted separately.

Results

Characteristics of Engagement in the API (Aim 1)

Table 1 presents baseline characteristics of students assigned to an API who engaged and did not engage in Stage 1 and Stage 2, respectively. Overall, 63.6% (376/591) of students randomized to an API engaged at all with the Stage 1 components (both PNF and SM), while 43.9% (260/591) completed the entire universal intervention (i.e., viewed all pages of PNF and completed all SM surveys required of them). Among heavy drinkers in Stage 2, 63.9% (101/158) of heavy drinkers engaged at all with the Stage 2 bridging strategy to which they were re-randomized, while 29.1% (46/158) fully completed their bridging strategy. Approximately 30% of students in the API group reported pre-college binge drinking. Baseline characteristics of the full sample (N=891) are presented in Supplemental Table 1.

Table 1:

Baseline characteristics of students randomized to an API by Stage 1 and Stage 2 engagement

	No stage 1 engagement (N=215)	Stage 1 engagement (N=376)	No stage 2 engagement (N =57)	Stage 2 engagement (N = 101)
	M(SD)/N(%)	M(SD)/N(%)	M(SD)/N(%)	M(SD)/N(%)
Sex
Female	131 (60.9%)	241 (64.1%)	31 (54.4%)	69 (68.3%)
Male	84 (39.1%)	135 (35.9%)	26 (45.6%)	32 (31.7%)
Race/ethnicity
White	169 (78.6%)	282 (75.0%)	49 (86.0%)	88 (87.1%)
Asian	20 (9.3%)	40 (10.6%)	2 (3.5%)	7 (6.9%)
Black	3 (1.4%)	15 (4.0%)	0 (0%)	0 (0%)
Hispanic/Latinx	11 (5.1%)	18 (4.8%)	3 (5.3%)	2 (2.0%)
Other/Multiracial	12 (5.6%)	21 (5.6%)	3 (5.3%)	4 (4.0%)
Pre-college intention to pledge Greek life
No	119 (55.3%)	251 (66.8%)	25 (43.9%)	54 (53.5%)
Yes/Undecided	96 (44.7%)	125 (33.2%)	32 (56.1%)	47 (46.5%)
Parents’ highest level of education
Bachelor’s or higher	169 (78.6%)	304 (80.9%)	48 (84.2%)	77 (76.2%)
Less than bachelor’s	46 (21.4%)	72 (19.1%)	9 (15.8%)	24 (23.8%)
Biological family member with clear drinking problem
No	138 (64.2%)	236 (62.8%)	34 (59.6%)	53 (52.5%)
Yes	77 (35.8%)	139 (37.0%)	23 (40.4%)	48 (47.5%)
Average patient health questionnaire score (0–3)	0.52 (0.52)	0.58 (0.58)	0.47 (0.39)	0.67 (0.66)
Suicidal ideation score (0–3)	0.08 (0.37)	0.07 (0.31)	0.07 (0.42)	0.10 (0.36)
Average perceived stress score (0–4)	1.30 (0.62)	1.31 (0.69)	1.28 (0.56)	1.40 (0.65)
Average general anxiety disorder score (0–3)	0.59 (0.63)	0.63 (0.63)	0.51 (0.48)	0.66 (0.64)
Pre-college binge drinking
Never	132 (61.4%)	291 (77.4%)	19 (33.3%)	43 (42.6%)
Yes	83 (38.6%)	85 (22.6%)	38 (66.7%)	58 (57.4%)
Baseline alcohol consequences (0–24)	1.82 (2.76)	1.18 (2.13)	3.51 (3.21)	2.71 (2.66)
Perceived norm: % of first-year students who binge drink (0–100)	25.80 (19.30)	20.90 (16.40)	29.30 (18.00)	27.20 (19.00)
Average social motives score (0–5)	1.92 (1.65)	1.56 (1.62)	3.04 (1.30)	2.74 (1.37)
Average enhancement motives score (0–5)	1.70 (1.54)	1.38 (1.47)	2.79 (1.25)	2.51 (1.17)
Average coping motives score (0–5)	0.97 (0.97)	0.81 (0.99)	1.53 (0.96)	1.41 (1.01)
Average conformity motives score (0–5)	0.91 (0.91)	0.71 (0.82)	1.25 (0.70)	1.17 (0.81)
Readiness for pre-contemplation score (−8 to 8)	−1.82 (2.98)	−1.79 (3.27)	−1.30 (2.43)	−1.64 (2.53)
Readiness for contemplation score (−8 to 8)	−4.76 (3.01)	−4.89 (2.89)	−3.93 (2.95)	−4.23 (2.50)
Readiness to take action score (−8 to 8)	−2.70 (3.45)	−2.92 (3.27)	−3.32 (2.73)	−2.33 (2.88)
Timing of intervention
Early	112 (52.1%)	183 (48.7%)	-	-
Late	103 (47.9%)	193 (51.3%)	-	-
Bridging intervention
Coach	-	-	27 (47.4%)	53 (52.5%)
Email	-	-	30 (52.6%)	48 (47.5%)
PNF engagement
None	-	-	32 (56.1%)	79 (78.2%)
Partial or full	-	-	25 (43.9%)	22 (21.8%)
Mid-semester binge drinking frequency (0–10)	1.10 (1.52)	0.93 (1.57)	2.96 (1.43)	2.91 (1.52)

PNF=Personalized Normative Feedback

Table 2 presents the unadjusted odds ratios of engaging with the different components of the API for various baseline covariates. Characteristics significantly associated with lower odds of PNF engagement included higher perceived norms of binge drinking and pre-college binge drinking—notably, students who reported pre-college binge drinking had lower odds of engaging in PNF by nearly 40% compared to students who did not report pre-college binge drinking. Greek-life intention and higher perceived norms of binge drinking were significantly associated with lower odds of SM engagement. Greater endorsement of all four drinking motives was also associated with reduced odds of SM engagement—notably, greater conformity motives were associated with 34% reduced odds of SM engagement. Students who reported pre-college binge drinking also had lower odds of engaging in SM by approximately 60% compared to students who did not report pre-college binge drinking. Greater total baseline alcohol consequences were also significantly associated with lower odds of SM engagement. Greek-life intention and higher perceived norms of binge drinking were associated with a significant decrease in engagement of both Stage 1 components (hereafter Stage 1 engagement). Greater enhancement, coping, and conformity motives were significantly associated with reduced odds of Stage 1 engagement. Pre-college binge drinking and greater total baseline alcohol consequences were also associated with lower odds of Stage 1 engagement. Greater depression and action increased the odds of engaging in the Stage 2 bridging strategy. Previous engagement with PNF was a significant predictor of engaging in Stage 2—the odds of engaging in Stage 2 nearly tripled when a student opened at least one page of personalized normative feedback in Stage 1.

Table 2:

Unadjusted odds ratios of engagement for different baseline covariates

Engagement
Predictor	aPNF	aSelf-monitoring	aStage 1	bStage 2
	Odds ratio (95% CI)	Odds ratio (95% CI)	Odds ratio (95% CI)	Odds ratio (95% CI)
Demographics
Female (vs. Male)	1.20 (0.84, 1.72)	1.18 (0.72, 1.95)	1.145 (0.81, 1.617)	1.81 (0.93, 3.53)
Non-white race/ethnicity (vs. White)	1.19 (0.78, 1.80)	1.38 (0.75, 2.56)	1.225 (0.82, 1.829)	0.91 (0.35, 2.33)
Greek life intention (Yes/Undecided vs. No)	0.77 (0.54, 1.10)	0.56 (0.34, 0.90) *	0.62 (0.44, 0.87) **	0.68 (0.35, 1.31)
Parents’ highest education (College degree or higher vs. Less)	0.75 (0.49, 1.14)	1.33 (0.69, 2.56)	0.87 (0.575, 1.318)	1.66 (0.71, 3.88)
Family member with drinking problem (Yes vs. No)	1.22 (0.85, 1.75)	1.01 (0.61, 1.67)	1.06 (0.75, 1.50)	1.34 (0.69, 2.58)
Mental Health
Average patient health questionnaire score (0–3)	1.19 (0.86, 1.64)	1.06 (0.68, 1.66)	1.22 (0.90, 1.67)	1.94 (1.01, 3.73) *
Suicidal ideation score (0–3)	0.98 (0.58, 1.66)	0.68 (0.38, 1.23)	0.94 (0.57, 1.54)	1.24 (0.49, 3.13)
Average perceived stress score (0–4)	1.03 (0.780, 1.34)	0.99 (0.69, 1.44)	1.02 (0.79, 1.31)	1.38 (0.81, 2.36)
Average general anxiety disorder score (0–3)	1.02 (0.78, 1.35)	1.13 (0.76, 1.69)	1.11 (0.85, 1.45)	1.57 (0.86, 2.88)
Drinking-related predictors
Pre-college binge drinking (Some vs. Never)	0.61 (0.42, 0.88) **	0.41 (0.25, 0.68) ***	0.47 (0.32, 0.67) ***	-
cMid-semester binge frequency (0–10)	-	-	-	0.98 (0.79, 1.21)
Baseline perceived norm % binge (10% increments, 0–100)	0.88 (0.80, 0.96) **	0.84 (0.74, 0.95) **	0.86 (0.78, 0.94) ***	0.94 (0.79, 1.12)
Baseline alcohol consequences (0–24)	0.94 (0.88, 1.01)	0.90 (0.83, 0.98) *	0.90 (0.84, 0.96) **	0.91 (0.81, 1.02)
Average social motives score (0–5)	0.94 (0.85, 1.05)	0.80 (0.69, 0.93) **	0.87 (0.79, 0.97)	0.84 (0.66, 1.08)
Average coping motives score (0–5)	0.96 (0.80, 1.14)	0.72 (0.58, 0.90) **	0.84 (0.71, 0.99) *	0.89 (0.64, 1.23)
Average enhancement motives score (0–5)	0.94 (0.84, 1.06)	0.77 (0.66, 0.90) ***	0.87 (0.77, 0.97) *	0.82 (0.62, 1.09)
Average conformity motives score (0–5)	0.90 (0.74, 1.10)	0.66 (0.51, 0.85) ***	0.77 (0.63, 0.93) **	0.87 (0.57, 1.33)
Readiness to change
Pre-contemplation score (-8 to 8)	1.04 (0.98, 1.09)	0.96 (0.89, 1.04)	1.00 (0.95, 1.06)	0.95 (0.83, 1.08)
Contemplation score (-8 to 8)	1.03 (0.97, 1.09)	0.96 (0.89, 1.05)	0.99 (0.93, 1.05)	0.96 (0.85, 1.08)
Action score (-8 to 8)	1.01 (0.95, 1.07)	0.99 (0.92, 1.07)	0.98 (0.93, 1.03)	1.14 (1.01, 1.29) *
Assigned intervention and previous engagement
Late intervention (vs. Early)	1.16 (0.82, 1.64)	0.89 (0.55, 1.45)	1.15 (0.82, 1.60)	0.71 (0.37, 1.37)
Email intervention (vs. Coach)	-	-	-	0.82 (0.43, 1.56)
PNF engagement (Some vs. none)	-	-	-	2.81 (1.39, 5.68) **

^aEngagement among API students (N=591).

^bEngagement in the Stage 2 bridging strategy is only among heavy drinkers (N=158).

^cMid-semester binge frequency is obtained from self-monitoring surveys in Stage 1 and is only considered a predictor of Stage 2 engagement among heavy drinkers. Stage 1 engagement is engaging in both PNF and self-monitoring components. PNF=Personalized Normative Feedback. CI=Confidence Interval

^***p<0.001

^**p<0.01

^*p<0.05

Table 3 presents the adjusted odds ratios of engaging with the API interventions for predictors selected by stepwise regression. Adjusting for family history of drinking problems, higher perceived norms of binge drinking and pre-college binge drinking were associated with lower odds of PNF engagement while contemplation (readiness to change) was associated with higher PNF engagement. Only conformity motives were associated with self-monitoring engagement such that higher conformity motives were associated with lower self-monitoring engagement. After adjusting for perceived norms of binge drinking, pre-college binge drinking and Greek-life intention were significantly associated with decreased odds of engaging in Stage 1. Furthermore, the significance and effect of PNF engagement, contemplation, and action (readiness to change) on the odds of engaging in Stage 2 was similar, even after adjusting for factors such as depression and having a family member with an alcohol problem.

Table 3:

Adjusted odds ratios of engagement

Selected Predictors of Engagement	Adjusted OR (95% CI)
a PNF Engagement
Pre-college binge frequency (Some vs. Never)	0.56 (0.36, 0.86)**
Readiness to change contemplation score (−8 to 8)	1.08 (1.01, 1.16)*
Baseline perceived norm % drink alcohol (10% increments, 0–100)	0.89 (0.79, 0.99)*
Family member with a drinking problem (Yes vs. No)	1.37 (0.93, 2.03)
a Self-Monitoring Engagement
Pre-college binge frequency (Some vs. Never)	0.59 (0.34, 1.03)
Average conformity motives score (0–5)	0.75 (0.56, 0.99)*
Greek life intention (Yes/Undecided vs. No)	0.68 (0.41, 1.15)
a Stage 1 Engagement
Pre-college binge frequency (Some vs. Never)	0.54 (0.36, 0.81)**
Greek life intention (Yes/Undecided vs. No)	0.68 (0.47, 0.99)*
Baseline perceived norm % binge (10% increment, 0–100)	0.91 (0.82, 1.01)
b Stage 2 Engagement
PNF engagement (Some vs. None)	2.97 (1.39, 6.32)**
Readiness to change action score (−8 to 8)	1.22 (1.04, 1.43)*
Readiness to change contemplation score (−8 to 8)	0.84 (0.72, 0.98)*
Average patient health questionnaire score (0–3)	1.71 (0.80, 3.65)
Family member with a drinking problem (Yes vs. No)	1.71 (0.82, 3.58)
Parents’ highest education (Less than bachelors vs. More)	1.93 (0.76, 4.87)

Note:

^aEngagement among API students (N=591).

^bEngagement in the assigned Stage 2 bridging strategy is only among heavy drinkers (N=158). Stage 1 engagement is engagement in both PNF and self-monitoring components.

^***p<0.001

^**p<0.01

^*p<0.05

PNF=Personalized Normative Feedback; OR=Odds Ratio; CI=Confidence Interval; Freq=frequency.

Effect of Engagement in Stage 1 and Stage 2 on Follow-Up Outcomes Among the API Group (Aim 2)

Of the 591 students randomized to an API, those who engaged in both Stage 1 components averaged .93 binge drinking occasions and 1.98 alcohol consequences in the month before the end of the fall semester (follow-up). API students who did not engage in both Stage 1 components had slightly higher binge drinking frequency (1.3 occasions/month) and almost 50% higher alcohol consequences (2.96 consequences /month) at follow-up, on average. After adjusting for Stage 1 intervention timing and possible confounders, the estimated effect of engaging vs. not engaging in both Stage 1 components among the 591 API students was not statistically significant for both outcomes; engaging in both components reduces the average binge drinking frequency by roughly 2% and the average alcohol consequences by roughly 5% at follow-up, after controlling for confounders (Table 4).

Table 4:

The adjusted effect of Stage 1 and Stage 2 engagement on alcohol-related outcomes at follow-up

	Ratio of mean outcome (95% CI)	P-value
Effect of engagement on binge drinking frequency (# of occasions/month) at follow-up
aStage 1 (Yes vs. No)	0.98 (0.79, 1.21)	0.836
bWhole API (Yes vs. No)	0.87 (0.70, 1.08)	0.214
Effect of engagement on alcohol consequences (# of consequences/month) at follow-up
aStage 1 (Yes vs. No)	0.95 (0.84, 1.08)	0.446
bWhole API (Yes vs. No)	0.78 (0.67, 0.90)	0.001

Note:

^aThe effect of engaging in Stage 1 on the alcohol-related outcomes is assessed among all API students (N=591) and was adjusted for Stage 1 intervention timing and predictors of Stage 1 engagement selected from stepwise regression.

^bThe effect of engaging in the whole API (both Stage 1 components and assigned Stage 2 bridging strategy) was assessed on the alcohol-related outcomes among heavy drinkers (N=158) and was adjusted by Stage 1 intervention timing, Stage 2 bridging strategy and predictors of Stage 1 engagement and Stage 2 engagement selected from stepwise regression. API=Adaptive Preventive Intervention; CI=Confidence Interval.

However, among the 158 heavy drinkers, the adjusted effect of engaging in the whole API (both Stage 1 components and the assigned Stage 2 bridging strategy) on alcohol consequences was significant. After controlling for Stage 1 intervention timing, bridging strategy and other confounders, the estimated average number of alcohol consequences at follow-up was 22% lower (10% to 33% lower) for heavy drinkers who engaged in both stages of the API, compared to heavy drinkers who did not engage in both stages (Table 4). Regarding binge drinking frequency in heavy drinkers, the adjusted effect of engaging in the whole API reduces the estimated average number of binge drinking occasions at follow-up by 13%, but this finding was not significant (Table 4). In the observed data, 101 heavy drinkers who engaged in both stages averaged 4.5 alcohol consequences and 2.2 binge drinking occasions at follow-up, and the 57 heavy drinkers who did not engage in both stages averaged almost 6 alcohol consequences and 2.3 binge drinking occasions at follow-up.

Per-Protocol Effect of API vs. Assessment-Only Control (Aim 3)

Table 5 and Figure 1 show the per-protocol effect of API vs. control on the alcohol-related outcomes. As a comparison, the intention-to-treat effect comparing those randomized to the API vs. control is also shown (Patrick et al., 2021). Among students in the control group, both binge drinking frequency and alcohol consequences roughly doubled from baseline to follow-up; binge drinking frequency increased from a model-estimated mean of 0.56 (95% CI: 0.43, 0.73) occasions in the past 30-days before college to 1.12 (95% CI: 0.93, 1.35) occasions in the past 30-days at the end of fall semester. Alcohol consequences increased from a model-estimated mean of 1.19 consequences (95% CI: 1.00, 1.44) in the past 30-days at baseline to 2.27 consequences (95% CI: 1.92, 1.35) in the past 30-days at follow-up.

Table 5:

Effect of API for intention-to-treat (randomized to API) and per-protocol (engaged in API) groups versus control (N=891)

Ratio of follow-up to baseline
	API Ratio (95% CI)	Control Ratio (95% CI)	Ratio of ratios (95% CI)	P-value
Binge drinking frequency (past 30-days)
Intention-to-treat a	1.68 (1.42, 1.99)	2.00 (1.58, 2.52)	0.84 (0.63, 1.12)	0.243
Per-protocol	1.52 (1.19, 1.94)	2.00 (1.58, 2.52)	0.76 (0.54, 1.06)	0.107
Alcohol consequences
Intention-to-treat a	1.64 (1.47, 1.83)	1.90 (1.58, 2.27)	0.87 (0.70, 1.07)	0.187
Per-protocol	1.42 (1.18, 1.71)	1.90 (1.58, 2.27)	0.75 (0.58, 0.97)	0.029

Note:

^aIntention-to-treat analyses are from the previously published outcomes paper of the M-Bridge study (Patrick et al., 2021) shown as comparison.

Ratios are shown for log-link models. P-values test the null hypothesis that the ratio of ratios is 1. API=Adaptive Preventive Intervention; CI=Confidence Interval

Figure 1: Intention-to-treat and Per-protocol effect of API vs. control: Model-estimated marginal means over time (N=891).

^a Intention-to-treat analyses from the previously published outcomes paper of the M-Bridge study (Patrick et al., 2021) are shown for comparison purposes. API=adaptive preventive intervention.

The per-protocol effect shows that had everyone engaged in the API, there is a statistically significant reduction in the relative increase in alcohol consequences from baseline to follow-up. Specifically, we estimated that the relative increase in alcohol consequences from baseline to follow-up would be 25% lower (3% to 42% lower) for students in the API group compared to students in the control group, if all API students had engaged in the components that they were eligible to complete.

For binge drinking, if all students engaged with the API interventions, then the relative increase in average number of binge drinking occasions from baseline to follow-up would be 24% lower than the control group, but this finding was not significant.

Discussion

The purpose of the present study was to determine the characteristics of first-year college students who engage in an API and examine whether engaging in both Stage 1 components and in the whole API was associated with reduced binge drinking and alcohol-related consequences, compared to API students who did not engage and to heavy drinkers who did not engage. Finally, we estimated the per-protocol effect of API vs. control (the effect API vs. control effect that would have been observed had all API students engaged). In terms of engagement, almost 2/3 of students engaged in universal interventions in Stage 1 of the API, and almost 2/3 of the eligible heavy drinkers engaged in the indicated Stage 2 of the API bridging them to other resources. It is challenging to put these results into perspective as definitions of engagement are inconsistent in the literature (Perski et al., 2017; Yardley et al., 2016). However, given that a quarter of mobile health apps are only used once after download (Vaghefi & Tulu, 2019), and on average half of participants adhere to web-based interventions (Kelders et al., 2012), our rates favor comparably to other work. Mental health, alcohol-related attitudes/beliefs, Stage 1 intervention timing, and prior Stage 1 engagement were differentially associated with intervention engagement. Encouragingly, average alcohol consequences at follow-up were significantly lower among heavy drinkers who engaged in both stages of the API compared to heavy drinkers who did not, which aligns with the harm reduction approach of college student drinking that highlights the importance of taking small steps toward reducing alcohol-related consequences (Marlatt et al., 2013). Indeed, findings suggested that heavy drinking young adults who engage in an API, even minimally, may experience harm reduction within the first three months after intervention.

The per-protocol effect provides evidence that the API would yield a statistically significant mitigation in the relative change in alcohol consequences from baseline to follow-up compared to control, had all students assigned to an API engaged with it. In particular, the estimated increase in alcohol consequences from baseline to follow-up would have been 25% lower in the API than those in the control group. As a contrast, there was no significant difference in the relative change in alcohol consequences from baseline to follow-up between students randomized to the API vs. control in the intention-to-treat analysis (Patrick et al., 2021). The per-protocol effect of API vs. control on binge drinking frequency suggests that had all students in the intervention group engaged with API, the relative increase in average binge drinking frequency from baseline to follow-up would be 24% lower than the control group. Although this difference was not significant, the effect size was larger than the intention-to-treat effect (16%; Patrick et al., 2021). Furthermore, these results suggested that this brief, cost-effective intervention reduced the harms associated with alcohol use among first-year college students within their first semester if they engaged in the intervention content. Overall, determining the per-protocol effect along with the intention-to-treat effect provides a more robust understanding of intervention efficacy.

Findings from this study suggested conformity motives may have an impact on whether one engages in intervention components such as self-monitoring. Drinking motives have a robust relationship with drinking outcomes, with coping motives strongly relating to alcohol-related consequences and social and enhancement motives to alcohol consumption (see Cooper et al., 2016 for a review). However, conformity motives have mixed associations with alcohol-related outcomes (Cooper et al., 2016). Although coping motives, enhancement motives, and social motives were associated with lower odds of engaging in the self-monitoring component of the API prior to the stepwise regression, only conformity motives were associated with lower odds of engaging in the self-monitoring component of the API in the adjusted model. Because first-year college students are attempting to forge new relationships and experiences, messages about the social norms of drinking are delivered via social cues that reflect either passive peer pressure (i.e., subtle; alcohol use is normal in college) or active peer pressure (i.e., overt; being offered alcohol). Whether the person conforms to the given social context depends on their level of self-monitoring: those with a higher level of self-monitoring change behavior to conform based on social cues reflecting passive or active peer pressure, and those with a lower level of self-monitoring change behavior in active peer pressure situations (Snyder & Gangestad, 1986). Therefore, first-year college students may have been less likely to engage in a self-monitoring intervention because they could have been experiencing active peer pressure to engage in drinking and/or passive peer pressure via implicit social norms regarding alcohol use. Thus, they conform to be like peers in the moment. In essence, asking someone to monitor their drinking when they are being pressured to drink to fit in, when conformity is the stronger force, may not be as effective of an intervention approach as potentially a PNF intervention, which directly impacts implicit normative messages.

Considerable evidence has indicated that social norms are associated with alcohol use and related consequences (e.g., Arterberry et al., 2014; Dumas et al., 2018). Prior studies have found that PNF interventions are associated with reductions in alcohol-related consequences (Cronce & Larimer, 2011; Lewis & Neighbors, 2006; Neighbors et al., 2016). Notably, our results have provided evidence of another potential impact of PNF interventions by showing that those who engaged in the PNF intervention at Stage 1 had greater odds of engaging in the Stage 2 bridging strategy. Correcting misconceptions about alcohol-related social norms via PNF has the potential to encourage those who engage in heavy drinking to continue accessing resources designed to reduce alcohol-related harms. This is in line with research suggesting that challenging perceived norms around drinking is one of the most consistent mechanisms of alcohol behavior change in college student populations (Reid & Carey, 2015).

Readiness to change has been understood as a person’s motivation to change a behavior (e.g., heavy alcohol use) in combination with self-efficacy in changing that behavior and includes five stages of change: precontemplation (i.e., do not want to change), contemplation (i.e., ambivalent to change), preparation (i.e., plan for change), action (i.e., implementing plan), and maintenance (i.e., behavior change becomes normative) (DiClemente et al., 2004). In the current study, being in the action stage of readiness to change was associated with greater odds of engaging in the Stage 2 bridging strategy, while contemplation was associated with lower odds of engagement in the Stage 2 bridging strategy. As would be expected, people engaging in heavy drinking who were in the contemplation stage may not have been ready to make changes in their behavior and did not want to continue accessing resources to reduce their drinking. However, heavy drinking students in the action stage have made the decision to change and have implemented a plan to reduce their alcohol use, which would potentially increase the probability that they would engage with additional resources to learn more about how to continue reducing their alcohol use.

Higher levels of depression among heavy drinkers were associated with greater odds of engaging in the Stage 2 bridging strategy. Although experiencing depression can lead to drinking alcohol to cope, people with co-occurring depression and alcohol-related problems could be more likely to engage in any intervention to relieve their depressive symptoms. Indeed, prior research has suggested that those who have co-occurring mood and substance use disorders are more likely to seek treatment (Kaufmann et al., 2014).

As indicated by the findings from the present study, common risk factors for alcohol use in young adulthood such as prior binge drinking, social norms that encourage heavy drinking, and being a member of a fraternity or sorority (Barry, 2007; Patrick & Schulenberg, 2014) may be associated with lower intervention engagement. Pre-college/mid-semester binge drinking, perceived norms of greater binge drinking, and Greek life intention were associated with lower odds of engaging in Stage 1 and self-monitoring. Despite the fact that people who have these risk factors have the greatest need of intervention, they are not accessing the resources needed to reduce harm associated with their alcohol use. Further research regarding the development of accessible resources within the university system via an implementation science approach (Bauer & Kirchner, 2020; Becker et al., 2021) would potentially enhance intervention engagement among students at greatest risk.

Limitations and Future Directions

There were limitations to this study. Participants were recruited from a predominately white institution and most of the sample was non-Hispanic white; study findings may not generalize to other populations. Considering the sample size of individuals assigned to the Stage 2 intervention was small and the way in which engagement with the Stage 2 intervention was defined (i.e., at minimum accepting an invitation/clicking on a link), reductions seen in alcohol outcomes could be related to either longer interaction with intervention elements (29.1% of heavy drinkers fully completed the Stage 2 bridging strategy) or the motivation to minimally engage with the intervention (63.9% of heavy drinkers in Stage 2 engaged, at least minimally in the bridging strategy). Although there was an attempt to avoid monetary incentivization of API participation, we cannot guarantee that participants randomized to an API did not partially engage in API stages due to the monetary incentives provided for survey completion. Other work has found that college students are more likely to participate in web-based intervention studies if incentives are offered (Neighbors et al., 2018). There may be important differences in what may increase engagement in a research context versus a clinical context (i.e. monetary incentives versus motivational enhancement). Therefore, further research on how incentives may compel people to complete intervention studies, as well as how incentives relate to the efficacy of such interventions, are encouraged. Self-selection bias may have impacted who decided to participate and continue in the study. Data were also reliant on self-report and retrospective recall and may have the potential for recall and social desirability bias. Further research is needed to determine whether the effects of the API last longer or are impacted by degree of engagement (partial versus full) in the intervention, especially among more diverse populations. As mentioned, definitions of engagement vary in the literature (Perski et al., 2017; Yardley et al., 2016). Although minimal engagement did have some impact on consequences in this study, more nuanced or stringent measures of engagement could better capture its relationship to outcomes such as reductions in binge drinking or high-intensity drinking. Additionally, future studies would benefit from examining how to increase engagement among the highest-risk populations.

Conclusions

Intervention engagement has important implications for harm reduction via brief intervention efficacy. Indeed, one goal of harm reduction related to college student alcohol use is to assist students in reducing alcohol use and alcohol-related consequences at their own pace, where any movement toward this goal, such as one less binge drinking occasion per month or reduction of even one consequence per month, as found in this study, is considered progress (Marlatt et al., 2013; Marlatt & Witkiewitz, 2002). In this study, the majority of eligible students (approximately 2/3) engaged with the intervention content, and their outcomes were significantly better if they did. However, the students at greatest risk (e.g., based on pre-college binge drinking) were the least likely to engage. This points to the importance of identifying and engaging students at risk in future work. Therefore, future studies using implementation science could determine where the gaps are occurring between implementation and full engagement of APIs and interventions designed to reduce alcohol-related harms on college campuses. This “light touch” API was brief, cost-effective, and resulted in a significant reduction in alcohol-related consequences compared to control, if students at least partially interacted with all the components (e.g., simply clicking Web-Basics email link or viewing one page of PNF). Clinicians and university officials are encouraged to implement similar intervention protocols for incoming first-year students, especially those students who are at greater risk for heavy drinking.

Public Health Significance.

This study focuses on the characteristics associated with engaging in a brief adaptive preventive intervention and concludes that engagement was associated with reduced alcohol-related consequences among first year college students. Identifying characteristics associated with intervention engagement (e.g., intending to pledge into a fraternity/sorority, pre-college drinking motives and behaviors, readiness to change) can inform efforts to improve intervention implementation and thereby improve outcomes.