Do Research Assessments Make College Students More Reactive to Alcohol Events?

Abstract

This study examines possible synergistic effects of alcohol-related events and post-event assessments on changes in college student readiness to change alcohol use, frequency of alcohol use, and negative consequences. Students were participants in a longitudinal study of drinking behavior. A portion of those reporting negative alcohol events/consequences (e.g., injury, vomiting, memory loss) during the parent study were randomly selected to participate in the present study (n = 492) and randomized to a post-event assessment (n = 296) or a no-assessment control (n = 196). Participants in the post-event assessment group were interviewed soon after their event, and participants in both conditions were interviewed three months after their event. Linear regression models showed higher 3-month readiness to change alcohol use in participants who received a post-event assessment than those who did not. There were reductions in drinking days, heavy drinking days, and further consequences post-event, but no differences by assessment group. However, female participants showed greater reductions in drinking days and heavy drinking days if they were assigned to assessment compared to control. There also was greater post-event reduction in drinking days among assessment group participants with high pre-college alcohol severity compared to low pre-college alcohol severity. Conversely, participants who reported high aversiveness of their event and were in the control group showed greater reduction in heavy drinking days than those assigned to the assessment group. Findings suggest that college student heavy drinking is reactive to alcohol events, whereas reactivity to post-event assessments may depend on gender, alcohol severity, and event aversiveness. This work highlights the importance of considering possible interactions among extra-therapeutic factors in clinical outcome research.

The alcohol field has begun to examine sources of research reactivity that may influence participants’ cognitive or behavioral changes, occurring outside of main treatment effects. Research reactivity can be defined as extraneous influences on outcomes that are a result of the overall research experience or a particular element of the research design. For example, the decision to enter a research study may activate internal processes such as self-reflection or expectancy for change, and this activation may facilitate naturalistic changes occurring within the study timeframe. Such processes are implicated in the ubiquitous “placebo effect” in psychological and medical outcome research (Price, Finniss, & Benedetti, 2008). Another example is the parallels between an in-person research assessment and common elements of brief alcohol intervention, including exploration of substance use with an empathic person (Clifford & Maisto, 2000). Randomization to experimental and control conditions should hold such influences constant across groups. However, noteworthy improvements in control conditions have prompted a closer look at research reactivity as a factor that may limit between-groups effect sizes in clinical trials.

Research on reactivity to assessment procedures has shown that assessment exposure, duration, and follow-up frequency are related to subsequent changes in substance use (Clifford, Maisto, Franzke, Longabaugh, & Beattie, 2000; Clifford, Maisto, & Davis, 2007). Gender-specific research has also found assessment reactivity; female clinical trial participants demonstrated a roughly 40% reduction in frequency of alcohol use prior to receiving any treatment (Epstein et al., 2005). While the reasons for these findings are unknown, a clinical or human connection and provision of ongoing contact have been suggested as mechanisms (Clifford et al., 2000; Clifford & Maisto, 2000; Maisto, Clifford, & Davis, 2007; Sobell & Sobell, 1981; Stout, Rubin, Zwick, Zwyiak & Bellino, 1999).

Naturalistic changes in response to specific alcohol-related events are another potential source of bias in clinical trials. This form of reactivity is particularly important in studies with non-treatment seeking individuals who are enrolled in research trials conducted in opportune settings, such as hospital emergency departments, trauma units, and college counseling centers. These interventions typically are delivered in response to an alcohol-related illness, injury, or citation, and attempt to capitalize on the cognitive and emotional salience of the precipitating alcohol event (Barnett, Monti, & Wood, 2001; Longabaugh, Wirtz, Beattie, Noel, & Stout, 1995). Yet, there is growing evidence that change happens in response to these events, independent of change due to intervention (Dunn et al., 2003; Morgan, White, & Mun, 2008).

A common finding in brief treatment research is a reduction in the target behavior for all groups following assessment and intervention (Jenkins, McAlaney, & McCambridge, 2009). Assessment or event reactivity may explain why some comparative outcome studies show non-significant differences between greater and lesser forms of brief treatment among non-treatment seeking young adults, particularly in the short-term (e.g., Borsari & Carey, 2005; Gwaltney et al., 2011; White et al., 2006). This is often seen as a significant effect of time that may make group differences more difficult to detect. There may also be synergism between assessment and event reactivity effects. Specifically, an immediate reduction in alcohol consumption following a negative alcohol-related event could be enhanced or occur only when made salient via an in-person assessment interview. For these reasons, it is important to examine behavior change proximal to alcohol events while testing whether such changes differ between assessment and non-assessment groups. It is also unknown if reactivity effects differ as a function of time (immediate only vs. short-term), or by participant or event characteristics.

The present study combines clinical research perspectives with epidemiologic data to investigate the influence of a post-alcohol event assessment on changes in college student behavior. We examined motivational readiness, alcohol use and related consequences in those who experienced a recent negative alcohol event/consequence and were randomized to an in-person post-event assessment group or a no-assessment control group. Our primary objectives were: (1) to compare the assessment to no-assessment group on subsequent (3 months post event) readiness to change alcohol use; (2) to investigate change in alcohol use and negative alcohol consequences post event and over the 3-month follow-up period, and whether post-event changes differed between the assessment and no-assessment groups; and (3) to explore participant gender, alcohol severity, and event aversiveness as moderators of assessment effects over time. We expected the post-event assessment group would show greater readiness to change than the no-assessment group. We also expected post-event reductions in alcohol use and problems for both groups, with greater reductions occurring in the assessment group. Our moderator objectives were exploratory and therefore we did not predict directions for these effects.

Method

The sample was derived from a larger longitudinal study of college student alcohol use (N = 1,053; 57.5% female) conducted with three cohorts of incoming college students at three colleges/universities in the northeast. To be eligible for inclusion, students were: (1) under 21 years old, (2) enrolled full-time, (3) registered to live on-campus during their first year, and (4) not international students. All procedures were approved by the Institutional Review Boards at the participating institutions.

Procedures

Parent study

Participants in the parent study were recruited during the summer prior to their first year of college. All baseline assessments were completed prior to college matriculation using a commercially available web survey software system. Throughout their freshmen and sophomore years, enrolled participants completed brief (five minute) biweekly web-based surveys of alcohol use and consequences for the prior seven days. Participants received $2 for each completed survey, a bonus of $20 for completing seven of eight surveys per semester, and an opportunity to win $100 after completing each survey.

Present study

Each week during the academic year (32 weeks in freshman and sophomore years), 25% of students who had reported one or more negative alcohol-related events/consequences on their biweekly survey were randomly selected to participate in the present study (n = 533). Participants were only selected once, and after selection were removed from eligibility. Selected participants were randomly assigned to an in-person post-event assessment (assessment group, n = 296) or a no post-event assessment control (n = 196). Participants in the post-event assessment group completed their interview within one week of the index event. Both groups completed an assessment three months following the event. Participant compensation was $25 for the post-event assessment and $30 for the three-month assessment. See Figure 1 for additional information on participant flow.

Figure 1.

Flow of participants through selection and randomization process

Notes. ^a. Of selected participants, 41 were selected late in the spring semester. Biweekly data were not collected in the summer so the 3 months of biweekly data for these participants were incomplete. These students are not in included in the present analyses. ^b. Students were randomly assigned to the post-event assessment group at a 3:2 ratio.

In the post-event assessment, the interviewer identified the index event by referring to the participant’s responses from the index biweekly survey; the day and nature of the event were reviewed and recorded. The review and assessment battery were designed to reflect alcohol event or infraction assessments often utilized in brief intervention studies with opportunistically recruited adolescents and young adults. A Timeline Followback (Sobell & Sobell, 1995) of alcohol use in the 30 days before the event was interviewer administered and other measures were self-administered (e.g., readiness to change, event characteristics, reactions to the event, alcohol-related consequences). These sessions averaged 76 minutes in length (SD = 18 minutes). Only measures relevant to the present analyses are described below.

Measures

Baseline covariates

A number of measures for the present study were collected at initial enrollment in the parent study. The Graduated Frequency for Alcohol was used to measure participant alcohol use; this measure has shown acceptable reliability and concurrent validity with diary-based consumption measures (Greenfield & Rogers, 1999). The Alcohol Use Disorders Identification Test assessed alcohol severity with a score of 8 or more indicating harmful use; this measure had excellent internal consistency at α=.93 (AUDIT; Saunders, Aasland, Babor, De La Fuente, & Grant, 1993). Readiness to change was measured with the single item Contemplation Ladder (Biener & Abrams, 1991) originally designed to measure readiness to change smoking and adapted for use with alcohol samples (Becker, Maio, & Longabaugh, 1996). The item states: “Each rung of this ladder represents where a person might be in thinking about changing their drinking.” Answer options ranged from 0 (no thought of changing) to 10 (taking action to change).

Biweekly measures

On each biweekly survey, participants provided the number of standard drinks consumed on each day for the prior seven days. The present study examines number of days drinking and gender-adjusted (5 or more daily drinks for men; 4 or more daily drinks for women) number of heavy drinking days for each seven-day period. Participants also were asked to indicate whether they had experienced any of 13 negative alcohol-related events/consequences in the past week. Items were derived from common measures of alcohol-related consequences with alphas in each week ranging from.61 –.70 and correlating well with annual measures of problems (r =.70 –.74). For each endorsed consequence, participants were queried as to its aversiveness: “How much did this experience bother you?” on a scale from (0) “not at all” to (3) “very much.” For purposes of the study, we created a count of consequences and derived a maximum and average aversiveness rating.

Interview measures

At the 3-month in-person assessment, readiness to change was measured with the Contemplation Ladder (Biener & Abrams, 1991).

Data Analysis

Chi-square and t-test analyses examined sample differences. Linear regression was used to test differences in readiness to change alcohol use measured three months post-event between the two assessment groups. Moderation of assessment group effects on readiness was examined in a hierarchal regression model, with gender and mean-centered pre-college AUDIT score and mean-centered aversiveness in the event selection week entered in the first step and the interactions of assessment by gender, assessment by AUDIT, and assessment by aversiveness entered in the second step.

Generalized Estimating Equations (GEE; Liang & Zeger, 1986) were used to examine number of drinking days, number of heavy drinking days, and number of negative alcohol consequences from the biweekly surveys over the pre (4 weeks) and post-event (12 weeks) time interval. In the first model, GEE analyses examined the main effect of assessment interview group (assessment, control), the slope for time pre-event (biweekly assessment number), the change in intercept post-event (time-varying covariate of 0 = pre-event and 1 = post-event, see Singer and Willett (2003)), and the time X event interaction, which indicated whether there was a difference in slopes pre- vs. post-event. The second model added terms that tested differences in assessment group slopes pre-event (group X time), assessment intercepts post-event (group X event), and whether the assessment group slopes and intercepts differed post-event (group X time X event).

For moderation analysis of post-event drinking and heavy drinking days and negative alcohol consequences, GEE analyses were conducted using a nested approach similar to that described above. First, the main effect of gender, AUDIT score, aversiveness of consequences in the event selection week, and moderator X time examined whether these variables affected the outcome or the slope of change in the outcome. The next model examined whether the main effect of assessment group differed by gender, AUDIT score, or event aversiveness (moderator X group). The final model examined whether the effect of assessment group on rate of change (i.e., slope) in the outcome differed by any of the three variables (moderator X group X time).

All GEE models were run with an exchangeable correlation structure. The biweekly drinking frequency variables were run in Poisson GEE models, and biweekly negative consequences were dichotomized and analyzed with a binomial GEE model. Baseline control variables for regression and GEE models were: gender, dichotomized race (white or multi-race vs. not white or multi-race), pre-college alcohol severity (AUDIT score), dummy-coded recruitment site (public college or university vs. private university), dummy-coded recruitment cohort (first semester vs. second semester selection), and the baseline level of the dependent variable.

Results

The participants in this study (n = 492) were 18.4 years old (SD = 0.5) and 59.5% female. Two-thirds identified their racial/ethnic identity as Non-Hispanic white (61.9%), 13.7% were Asian-American, 9.0% indicated more than one race/ethnicity, 7.7% were Latino/Hispanic, and 6.6% were African-American.

Sample Group Differences

Participants eligible for the current study due to alcohol-related events

Of students who reported having an alcohol-related event/consequence on the biweekly surveys, 67% were white and 10% identified as being multi-race; these students were more likely to report alcohol events than other ethnic groups, χ² (5, 1005) = 44.22, p <.001. Eligible students also showed higher rates of pre-college past-month drinking (3.0 [SD = 5.4] vs. 0.7 [SD = 2.7]; t(1002) = 9.21, p <.001) and heavy drinking (1.2 [SD = 2.8] vs. 0.2 [SD = 1.2]; t(1002) = 7.64, p <.001) days, and AUDIT score (4.6 [SD = 4.4] vs. 0.7 [SD = 1.8]; t(1001) = 20.39, p <.001) than non-eligible students.

Eligible participants who were randomly selected for the current study

Eligible participants reporting a negative alcohol event/consequence experienced a mean of 2.2 (SD = 1.6) consequences and a corresponding average event aversiveness rating of 1.48 (SD =.81; Max = 1.78[.95]) in the event week. The most commonly endorsed items were sickness and memory loss (36–40%), followed by sadness and regretted behaviors (14–28%), and passing out, injuries, fighting, trouble with authorities, or driving under the influence were less frequent (4–15%). Randomly selected compared to non-selected students did not differ by gender, frequency of pre-college drinking, or pre-college number of heavy drinking days, but they were more likely to identify as white or multi-race, χ² (5, 684) = 30.69, p <.001 and showed higher AUDIT scores (5.2 [SD = 4.5] vs. 3.2 [SD = 3.6]; t(686) = 5.18, p <.001). There were no significant differences on key variables between randomly selected students assigned to assessment compared to those assigned to control.

Examining Assessment and Event Reactivity

Readiness to change alcohol use

Table 1 shows coefficient estimates for the main effects regression model. Having received an assessment after the alcohol-related event significantly predicted higher 3-month follow-up readiness to change alcohol use compared to not receiving a post-event assessment (Effect Size_change =.29). Two-way interaction effects for assessment group by gender, AUDIT score or average aversiveness rating¹ were nonsignificant.

Table 1.

Predictors of 3-month readiness to change

Variable	B	b (se)	t (df)	p
Gender (1 = female)	−0.02	−0.10(0.29)	−0.33(388)	.740
Race (1 = white or multi-race)	−0.05	−0.34(0.33)	−1.02(388)	.309
Pre college AUDIT score	−0.08	−0.06(0.04)	−1.62(388)	.106
Pre college readiness to change	0.33	0.34(0.05)	6.75(388)	<.001
Average event aversiveness	0.03	0.14(0.29)	0.49(388)	.624
Public collegea	0.01	0.13(0.48)	0.27(388)	.786
Public universitya	0.03	0.10(0.18)	0.57(388)	.568
Semester one selection	0.04	0.23(0.33)	0.69(388)	.490
Assessment interview	0.16	0.96(0.31)	3.11(388)	.002

Notes. N =492.

^aReference group = private university. Main effects model shown [F(9, 379) = 7.01, R² =.14, p <.001].

Alcohol use and consequences before and after a negative alcohol-related event

Figure 2 shows mean number of drinking days over the 16-week pre- and post-event timeframe. The GEE model did not show a significant slope for drinking pre-event or a significant change in the intercept for drinking post-event. However, there was a significant negative slope for post-event drinking days over time (B = −0.09(.04), p <.05). The second step of the model showed no assessment group differences pre- or post-event. Therefore, our analyses showed a post-event decline in drinking days over the 3-month follow-up period that did not differ by assessment groups. Of the covariates examined, being female showed a significant negative main effect (B = −0.12(.05), p <.05), and pre-college AUDIT score showed a significant positive main effect (B = 0.05(.01), p <.001) on number of drinking days.

Figure 2.

Number of drinking days from the biweekly assessments over the pre and post assessment period

Notes. Index event week depicted by peak at time 0 not included in GEE analysis.

Figure 3 shows number of heavy drinking days pre- and post-event. The first step of the model showed a marginally significant and positive slope for heavy drinking prior to the index event (B = 0.11(.06), p =.052); there was no change in intercept post-event, but there was a significant negative slope for heavy drinking days over time (B = −0.17(.06), p <.005). In the second step, no assessment group effects or interactions with time were significant, indicating that the post-event assessment showed no additional influence on post-event reductions in heavy drinking days. Of the covariates examined, pre-college AUDIT score was positively related to number of heavy drinking days (B = 0.07(.01), p <.001).

Figure 3.

Number of heavy drinking days per week from the biweekly assessments over the pre- and post-assessment period

Notes. Index event week depicted by peak at time 0 not included in GEE analysis.

Consistent with results for alcohol use, there was a significant negative slope for further consequences (OR:.70 [95%CI:.52,.91]; see Figure 4) over the 3-month follow-up period, indicating a decline in negative consequences; this effect did not differ by assessment groups. Also consistent with the alcohol models, pre-college AUDIT score showed a positive main effect for negative consequences (OR: 1.10 [95%CI: 1.07, 1.12]). In addition, students selected from a public university showed greater odds of experiencing a negative alcohol consequence (OR: 1.44 [95%CI: 1.12, 1.86).

Figure 4.

Number of negative drinking consequences over the pre and post assessment period

Notes. Index event week depicted by peak at time 0 not included in GEE analysis.

Student gender, pre-college AUDIT score, and event aversiveness as moderators of assessment group effects

Two-way interactions of assessment group by gender, AUDIT, or event aversiveness were nonsignificant. For number of drinking days, there was a three-way interaction effect whereby the slope of drinking in the assessment group, compared to control, was more negative for females (B = −0.03(.01), p <.05) and for students with high pre-college AUDIT scores (B = −0.003(.001), p <.05), indicating these individuals showed greater drinking reduction if they were in the assessment group. The analyses for heavy drinking days also showed a negative slope for females assigned to the assessment compared to control group (B −0.03(.02), p <.05). For average aversiveness in the event week, there was a significant three-way interaction in the opposite direction. The slope in heavy drinking for those in the assessment group, compared to control, was more positive at higher levels of aversiveness (B = 0.03(.01), p <.01). This indicates that participants who reported having a more negative response to their event showed a greater increase in heavy drinking in the assessment compared to control group. There were no significant interaction effects for negative consequences in the follow-up period.

Discussion

Entering a research study and experiencing research assessment procedures are design factors that have been implicated in short-term gains achieved by control groups (Bernstein, Bernstein, & Heeren, 2010). Such processes may be particularly important among non-treatment seekers that become engaged in research due to experience of negative alcohol-related events or consequences. College students were included in the present study because they had a recent negative alcohol-related event/consequence, and these students showed higher rates of pre-college alcohol use and alcohol severity (AUDIT score) than students who were not included. The present sample appeared descriptively similar to college students identified as requiring intervention on college campuses nationwide (see Barnett et al., 2008).

Our analyses showed a post-event reduction in drinking and heavy drinking frequency, as well as number of further consequences, across both groups that persisted over the 3-month timeframe. While statistically significant, these changes were modest, and this effect magnitude may be related to the type of alcohol events or consequences most commonly experienced (i.e., primarily sickness, memory loss, or regretted behaviors). Previous work has shown somewhat larger event effects with mandated college students, and greater reductions in frequency of drinking for events qualified as “serious” compared to those qualified as “non-serious” (Morgan et al., 2008). This study suggests event reactivity with regard to consumption and consequences, but further research into effect variation by event type and within different samples (e.g., mandated vs. voluntary students) is needed. There was a shift in readiness to change alcohol use from baseline to the 3-month post-event assessment that was greater for the assessment compared to the no-assessment group. This indicated that the assessment was related to an increase in readiness to change, but this cognitive shift did not translate to a differential behavioral shift. Our results showed no effects for the post-event assessment on alcohol consumption or consequence outcomes, but it is unknown whether our data collection procedures (i.e., biweekly assessments within the larger naturalistic study that preceded and occurred throughout the sub-study 3-month timeframe) could have muted assessment reactivity effects. Reviews on the behavioral effects of research assessments have suggested that these effects are variable and at times, minimal (Bernstein et al., 2010) as well as possibly “fragile” or otherwise poorly understood (McCambridge, 2009). A recent meta-analytic review of drinking reduction in the control groups of BMI trials found effect heterogeneity significant enough to hinder any interpretation of a pooled estimate (Jenkins et al., 2009), although individual studies have supported assessment or screening effects in college student drinkers (Kypri, Langley, Saunders, & Cashell-Smith, 2006; McCambridge & Day, 2007; Walters, Vader, Harris, & Jouriles, 2009). Our results suggest no additional effect of assessment on post-event changes in drinking, heavy drinking, and further consequences among college students who experienced alcohol consequences, but who were not involved in a clinical intervention trial.

While indicating no assessment reactivity effects in the overall sample, our findings do suggest reactivity by particular student characteristics. Specifically, greater reduction in drinking was seen for females and for more severe alcohol users assigned to a post-event assessment compared to the control. It would be reasonable to conclude that students with greater histories of alcohol use and consequences made conclusions about their alcohol use in response to the review of their event and relevant measures. This combination may have made them more amenable to self-reflection or more susceptible to cognitive dissonance, which may have lead to subsequent behavior change. This is consistent with conclusions of prior work (Clifford et al., 2000; Kypri et al., 2006), but has not been empirically examined. Our findings are also consistent with work suggesting that females may be more likely to change following a critical alcohol event or may engage in greater self-exploration in the context of an interpersonal interaction (Carey, Henson, Carey, & Maisto, 2009). An unexpected result was that individuals who had more aversive events and were assigned to the assessment group showed greater increases in heavy drinking compared to those assigned to control. Perhaps face-to-face inquiry regarding the event, alcohol use, and alcohol-related consequences felt punitive to those already feeling strong negative effects of the event. Taken in sum, these moderation findings suggest that student behavioral and emotional reactivity to events is complex and worthy of further study.

The study has some limitations to consider. A longitudinal naturalistic study of college student drinking provided the data on processes of interested to clinical outcome research with this population. Data collection procedures provided ongoing reports of alcohol use in the 3-month study timeframe, and this offers more detailed information on behavioral fluctuations than a summary measure. However, these biweekly self reports, including reporting prior to inclusion in this sub-study, could have created reactivity effects in the assessment and no-assessment groups. Cognitive or behavioral reactivity to a 5-minute biweekly survey could be non-existent, short-term, or could increase with the repeated reporting over time; within this study, these possible effects cannot be determined. Yet, research on reactivity to daily ecological momentary assessment has suggested minimal effects for both alcohol use and motivation among college student drinkers (Hufford et al., 2002), but further research into the extent of reactivity effects is needed. Finally, we have emphasized the importance of considering event and/or assessment reactivity in opportunistically recruited samples, but it is unknown how our findings would replicate in other settings, such as acute hospital care, with young adults mandated to alcohol intervention, or with voluntary treatment-seeking individuals.

This study provides a unique look at the possible interaction to two kinds of reactivity among college student participants. The present work found support for motivational reactivity to assessments in this primarily non-contemplative to contemplative sample but no behavioral assessment reactivity effects for frequency of drinking, heavy drinking, or further consequences. Event reactivity was found across groups and differential assessment reactivity was found within particular student subgroups. The present work, combined with literature to date, suggests that reactivity effects are small but not negligible, and may be related to a number of factors worthy of continued examination. These directions may have important implications for brief intervention trials with young adult alcohol users, and may contribute to clinical knowledge as to when minimal intervention may be sufficient.