Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2021 Nov 1.
Published in final edited form as: Psychol Sch. 2020 Jul 7;57(11):1724–1740. doi: 10.1002/pits.22422

Sex differences in the acceptability and short‐term outcomes of a web‐based personalized feedback alcohol intervention for high school seniors

Diana M Doumas 1,2, Susan Esp 1,3, Rob Turrisi 4, Laura Bond 5, Sherise Porchia 2, Brian Flay 6
PMCID: PMC7595264  NIHMSID: NIHMS1620700  PMID: 33132444

Abstract

Despite the escalation of alcohol use through high school, the majority of research on school-based alcohol interventions has been conducted with junior high students or first and second year high school students. Preliminary research indicates a brief, web-based personalized feedback intervention developed for college students (eCHECKUP TO GO) may be a promising program for high school seniors. Although these studies demonstrate positive intervention effects, there is some evidence for greater program efficacy for females in this age group. The current study investigates sex differences in program acceptability of the eCHECKUP TO GO and its relationship to short-term alcohol outcomes among high school seniors (N = 135). Overall, the majority of students reported they found the program to be acceptable (i.e., user-friendly and useful). However, contrary to our hypothesis, results indicated that male students reported significantly higher perceptions of program acceptability than females. Although, we did not find sex differences in alcohol outcomes, program user-friendliness was related to reductions in alcohol use for males. Results of this study add to the literature supporting the eCHECKUP TO GO for high school seniors and highlight the importance of program user-friendliness for males. Implications for implementing the program as a school-based intervention are discussed.

Keywords: high school, alcohol, web-based, personalized feedback

Adolescent alcohol use is a significant problem in the United States, with nearly 60% of students reporting alcohol use by the end high school (Johnston et al., 2019). Alcohol use in high school is associated with a wide range of negative consequences including interpersonal problems, unwanted sexual activity (Arata et al., 2003), dating violence, suicide attempts (Miller et al., 2007), and impaired neurocognitive functioning (Hanson et al., 2011; Nguyen-Louie et al., 2015). Additionally, high school alcohol use is associated with poor school attendance (Patte et al., 2017), academic problems (Balsa et al., 2011; Rasberry et al., 2017), and failure to complete high school (Kelly et al., 2015; Townsend et al., 2007; Wang & Fredricks, 2014).

High School Seniors

National survey data indicate alcohol use escalates through high school with the highest rates of drinking reported by high school seniors (Johnston et al., 2019). Specifically, 30.2% of high school seniors surveyed reported using alcohol within the past 30 days and 42.9% of high school seniors reported being drunk at least once in their lifetime (Johnston et al., 2019). Further, hazardous drinking is more prevalent among older high school students, with risky drinking practices (i.e., drinking games) peaking at ages 17 – 19 (Borsari et al., 2013; Zamboanga et al., 2016). Hazardous drinking is also associated with a wide range of negative consequences and high risk behavior including having a hangover, passing out, engaging in regretted sexual activities, and drinking and driving (Borsari et al., 2013).

Research indicates there is a need for evidence-based practices for addressing adolescent alcohol use (Burrow-Sanchez et al., 2008). This may be particularly true for high school seniors who report the highest rates of drinking (Johnston et al., 2019) and hazardous drinking behavior (Borsari et al., 2013). Despite the need for alcohol interventions for high school seniors, a seminal review of alcohol preventive interventions concluded that there is limited research conducted with high school students aged 16 and older (Spoth et al., 2008). Similarly, more recent reviews indicate that the majority of alcohol intervention research targets junior high school students (Strøm et al., 2014) or students in their first two years of high school (Champion et al., 2013). These findings, coupled with the high rates of alcohol use and associated consequences, highlight the importance of identifying effective alcohol interventions for high school seniors.

Providing Alcohol Intervention through the School Setting

There are several advantages to providing alcohol interventions to adolescents through the school setting relative to clinic-based interventions (Wagner et al., 2004). First, the school is an identifiable setting where the program can be disseminated to reach all adolescents (Doumas et al., 2015). Next, interventions delivered through school-based programs have the advantage of reaching adolescents who may not readily seek treatment elsewhere (Glass et al., 2015; Reavely et al., 2010). Further, the school represents a unique setting for intervention efforts because of the interaction of personal, social, and academic context that affect a student’s life and substance use choices (Doumas et al., 2015).

Researchers have found that effective school-based interventions include material that is engaging to adolescents (Wagner et al., 2004). According to the Elaboration Likelihood Model (ELM) (Petty & Cacioppo, 1986), information processed through central routes is processed more actively and leads to stronger behavior change than information processed through peripheral routes. Presenting information using language that is easy to understand and includes personally relevant content increases elaboration (Petty et al., 2009). Interventions that are perceived as user-friendly and useful (i.e., acceptable) increase the likelihood of central processing and, in turn, intervention effectiveness. Web-based programs may be particularly appealing to adolescents due to their novelty and game-like appearance (Schinke et al., 2006). Additionally, adolescents may perceive personalized feedback alcohol interventions as particularly relevant due to the easy to use format and individualized nature of the feedback (Doumas, 2015).

The eCHECKUP TO GO

The eCHECKUP TO GO (San Diego State University Research Foundation, n.d.) is a brief, web-based personalized feedback program identified as a highly effective, low cost alcohol intervention by the National Institute on Alcohol Abuse and Alcoholism (NIAAA) CollegeAIM Alcohol Intervention Matrix (NIAAA, 2015). The program includes standardized alcohol screening instruments and provides personalized feedback designed to help students make better decisions about their drinking. The developers designed the program to reduce risk factors for alcohol use including normative perceptions of peer drinking, positive alcohol beliefs, and positive alcohol expectancies. Because the eCHECKUP TO GO program is brief, can be disseminated to large groups of students within one class period, and is inexpensive ($1075 per year for unlimited use), the program is well-suited for use as a school-based intervention.

Although originally developed for college students, researchers have identified the eCHECKUP TO GO as a promising school-based intervention for high school seniors (Doumas & Esp, 2019; Doumas, Esp, Flay, & Bond, 2017; Doumas, Esp, Johnson et al., 2017; Doumas et al., 2020). Initial randomized controlled trials examining the efficacy of the eCHECKUP TO GO with high school seniors indicated that students participating in the program reported a reduction in risk factors for alcohol use including normative beliefs about peer drinking, positive beliefs about alcohol, and positive alcohol expectancies (Doumas, Esp, Johnson et al., 2017), alcohol use (Doumas, Esp, Flay, & Bond, 2017), and alcohol-related consequences (Doumas & Esp, 2019) relative to students in a control group.

There is, however, some evidence to suggest that program effects for the eCHECKUP TO GO may be stronger for females (Doumas, Esp, Johnson et al., 2017; Doumas et al., 2020). Specifically, in a study examining changes in risk factors for alcohol (e.g., perceptions of peer drinking, beliefs about alcohol, positive alcohol expectancies) among high school seniors, researchers found significant intervention effects for females only (Doumas, Esp, Johnson et al., 2017). Further, in a study replicating and extending that work, researchers found positive results for males and females for normative perceptions of peer drinking frequency, normative perceptions of peer heavy episodic drinking, and positive alcohol expectancies, but decreases in normative perceptions of peer drunkenness were significant for females only (Doumas et al., 2020). Additionally, for alcohol outcomes, there were no sex differences in peak drinking quantity, but reductions in frequency of alcohol use were significant for females only. Thus, although there is growing support for the efficacy of the eCHECKUP TO GO for high school seniors, initial research demonstrates significant intervention effects for a wider range of outcomes for females than males in this age group. Based on ELM principles, it is possible that intervention effects favoring females may be related to differential levels of perceived intervention acceptability (i.e., user-friendliness and utility) between male and female high school seniors.

The Current Study

The purpose of the current study is to evaluate sex differences in program acceptability, short-term alcohol outcomes, and the relationship between acceptability and outcomes among high school seniors. We selected high school seniors because seniors have the highest rates of alcohol use (Johnston et al., 2019) and hazardous drinking (Borsari et al., 2013) among high school students, yet alcohol intervention research for this age group is limited (Spoth et al., 2008; Champion et al., 2013; Strøm et al., 2014). Because sex differences in program efficacy have emerged in intervention effects for risk factors for alcohol use and drinking behavior, it is important to understand if males and females are experiencing the program differently. Pilot research on the acceptability of the eCHECKUP TO GO conducted with ninth grade students indicated the majority of students found the program to be both user-friendly and useful (Doumas, 2015). Although findings suggested the program was more positively perceived by those who reported alcohol use, sex differences in program acceptability were not examined. This study extends this research by examining sex differences in the program acceptability, alcohol outcomes, and the relationship between the two among high school seniors completing the eCHECKUP TO GO intervention. Based on the ELM and preliminary studies suggesting that females may respond to the eCHECKUP TO GO intervention more favorably, we hypothesized that 1) program acceptability would be higher for female students than male students, 2) female students would report larger reductions in alcohol use compared to male students at a 30-day follow-up, and 3) intervention acceptability would be positively related to alcohol outcomes, with sex differences favoring females.

Method

Research Design

We used a quasi-experimental design to examine sex differences in perceived intervention acceptability. First, we assessed sex differences in perceived intervention user-friendliness and utility at baseline. Second, we examined sex differences in alcohol outcomes from baseline to a 30-day follow-up. Finally, we explored sex differences in the relationship between intervention acceptability and changes in alcohol use.

Participants

Participants in the current study were students recruited from two urban high schools in the Northwest. Parents of all seniors at each school (N = 867) were sent consent forms. A total of 39.4% (n = 342) provided consent and of these students with parental consent, 91% (n = 311) agreed to participate in the study. Participants in this study were students randomized to receive the eCHECKUP TO GO intervention and who reported drinking at least once in their lifetime (n = 135; 57.8% female and 42.2% male). Participant ages ranged from 15 to 18 (M = 17.15, SD = 0.47). Participants were primarily Caucasian (89.5%), with 3.0% Hispanic, 3.0% Asian, 1.5% American Indian/Alaskan Native, 1.5% Hawaiian/Other Pacific Islander, and 1.6% other. Overall, 91.1% (n =123) of the 135 participants completed the 30-day follow-up assessment. We found no differences in demographic characteristics or alcohol outcome variables between those who completed the follow-up assessment and those who did not.

Procedure

Data collection occurred in the fall of 2018. A member of the research team contacted the principals at the high schools to invite the school to participate in the study. All seniors registered at the school were eligible to participate. The school contacted parents of all seniors via letter by mail at their permanent addresses provided by the registrar’s office. The letter contained a parental consent form and a project-addressed, stamped envelope. The letter asked parents to return signed consent forms indicating permission for their adolescent to participate in the study. In addition, the letter provided a phone number and email address so that parents could ask questions prior to signing the consent form. The researchers sent reminder letters to the student’s home address and sent letters home with the student to give to the parent. Researchers asked participants with parental consent to assent prior to participating in the study survey.

Students with parental consent met at the school’s computer lab to participate in the study. A member of the research team described the research study and invited the students to participate. Students who agreed to participate received a unique personal identification number (PIN) to maintain confidentiality and a URL to access the survey. Students then logged onto the survey website where they read a welcome screen explaining the research and were asked for their assent to participate. Once students gave assent by clicking “Agree,” they were taken to a screen that asked them to enter their PIN and were then directed to begin the survey, which took aproximately15 minutes to complete. Students who did not have parental consent or who did not provide assent remained in their classroom with their teacher and completed an alternative assignment. Students completed the online intervention program immediately after completing the survey. The researchers invited all participants who completed the baseline survey to participate in a 30-day follow-up survey. The University Institutional Review Board and the School District Research Board approved all study procedures.

The eCHECKUP TO GO Intervention

The eCHECKUP TO GO is a 30-minute web-based intervention designed to reduce drinking by changing perceptions of peer drinking norms, alcohol beliefs, and alcohol expectancies through personalized feedback. The eCHECKUP TO GO is available through the San Diego State University Research Foundation (http://www.echeckuptogo.com/). The program is customized for the participating school, including providing normative data for the specific school, referrals for the local community, and designing the website using school colors and logos. The program includes two sections: 1) online assessment survey and 2) personalized feedback.

Online Assessment

The online assessment consists of demographic information (e.g. sex, age, weight, living situation, class standing) and information on alcohol consumption and drinking behavior assessed through standard alcohol measures. The assessment includes questions about alcohol use, positive aspects of drinking and negative consequences of alcohol use, risk-factors associated with drinking, perceptions of prevalence of peer alcohol use, a “quiz” on beliefs about the effects of alcohol on physical performance and how the body processes alcohol, activities one may like to spend more time doing, and motivation and confidence in one’s ability to change drinking behavior.

Individualized Feedback

Individualized feedback is provided in the following domains: graphical depictions of quantity and frequency of drinking, estimated risk-status for negative consequences associated with drinking, genetic risk, tolerance, approximate financial cost of drinking in the past year, and normative feedback comparing one’s own drinking to national survey, as well as comparisons of perceptions of peer drinking to local high school data. The feedback section also includes the student’s responses and correct answers to the “quiz” on physical performance and a list of goals generated from the assessment questions and ways to meet those goals. The program assesses student willingness to use several protective behavioral strategies (e.g. avoid drinking games, space drinks out over time, spend time with friends who don’t drink alcohol, identify alternative social activities instead of partying). The student is then given a list of the strategies selected that can be used as steps to change drinking behavior. The program then provides resources for services in the local community.

Measures

Intervention Acceptability

The researchers assessed acceptability of the intervention using a brief survey comprised of 7 items ranked on a 4-point scale from 1 (strongly disagree) to 4 (strongly agree). We based this survey on an exit survey used in prior research demonstrating a two-factor solution with the components user-friendliness and utility (Doumas, 2015). The survey has demonstrated internal reliability with Cronbach’s alphas ranging from .71 to .84. The user-friendliness subscale is made up of four items: “The program was engaging;” “The program was easy to use;” “The feedback was easy to read;” and I was able to understand the feedback.” The utility subscale is made of three items: “I was interested in the feedback I received;” “I learned something from the program;” and “I would recommend the program to others.” Cronbach’s alphas for the user-friendliness items was .81 and for the utility items was .87.

Alcohol Use

We assessed typical weekly drinking quantity using the Daily Drinking Questionnaire (DDQ, Collins et al., 1985). Participants were asked the question “Given that it is a typical week, please write the number of drinks you probably would have each day.” A response scale is provided for each day of the week (e.g., Monday, Tuesday, etc.). We calculated weekly drinking by summing the reports for the seven days of the week. Frequency of alcohol use was assessed using the Quantity/Frequency/Peak questionnaire, (QFP; Dimeff et al., 1999; Marlatt et al., 1998). To indicate the frequency of drinking, participants answered the question “How often do you use alcohol” with responses provided on an 8-point Likert scale with options ranging from “0” to “7” (“Do not drink alcohol” to “Every day”). Items were reverse scores so higher scores represent higher levels of drinking frequency.

Power Analysis

We conducted an a priori power analysis using the G*Power 3.1.3 program (Faul et al., 2007) for a chi square analysis, a GLM repeated-measures multivariate analysis of variance (MANOVA) with two time points, and for a hierarchical regression analysis. Results of the power analysis indicated for power of ≥ 0.80 to detect a medium effect size with an alpha level of .05, a sample size of 88 is needed for the chi square analysis, a sample size of 128 is needed for the repeated measures MANOVA, and a sample of 98 is needed for the regression analysis. Thus, our sample is adequate for the proposed analyses.

Data Analysis

We conducted all analyses using SPSS version 25.0. We examined the outcome variable for outliers at baseline and follow-up assessments and adjusted outliers to 3.3 SD above the mean before conducting analyses (Tabachnik & Fidell, 2007). To evaluate intervention acceptability, frequency data was combined into dichotomous variables. “Strongly Disagree” and “Disagree” were combined to form a Disagree category and “Strongly Agree” and “Agree” were combined to create an Agree category. We used these frequency categories in chi square analyses to examine differences in perception of user-friendliness and utility of the intervention between female and male students. To assess sex differences in changes in alcohol use from baseline to the 30-day follow-up, we conducted a GLM repeated measures multivariate analysis of variance (MANOVA). We fit study outcomes with a repeated measures MANOVA with sex (male; female) as the grouping factor and time (baseline; follow-up) as the within factor. We conducted post hoc univariate analyses of variance (ANOVAs) to examine significant effects. Finally, to examine sex differences in the relationship between intervention acceptability and alcohol outcomes, we conducted two hierarchical regression analyses. For each regression analysis, the alcohol outcome at baseline was included as a control variable on Step 1, main effects (sex, user-friendliness, and utility) were entered on Step 2, and the two-way interactions (sex x user-friendliness and sex x utility) were entered on Step 3. All predictor variables were mean centered to reduce problems of multicolinearity introduced into equations containing interaction terms (Aiken & West, 1991). We used simple slopes to examine the direction and degree of significant interactions (Aiken & West, 1991). We examined the variance inflation factor (VIF) for predictors to assess multicollinearity. The observed VIF ranged between 1.0 – 2.3, with corresponding tolerance levels ranging from 0.4 – 1.0, and is below the rule of thumb of VIF < 10 (Erford, 2015; Norman & Streiner, 2008), suggesting acceptable levels of multicollinearity among the predictor variables. We calculated effect size using the phi coefficient (Φ) for the chi square analyses, partial eta squared (η2p) for the MANOVA and ANOVA analyses, and R2 for the regression analyses, with magnitude of effects interpreted as follows: Small (Φ = .10, η2p = .01, R2 = .02), medium (Φ = .30, η2p = .06, R2 = .15), large (Φ = .50, η2p = . 14, R2 = .35), respectively (Cohen, 1969; Richardson, 2011). We controlled for Type 1 error by using the Holm-Bonferroni procedure (Holm, 1979).

Results

Intervention Acceptability

Perceptions of user-friendliness were generally positive with the majority of students agreeing with statements that the program was easy to use (91.9%), that the feedback easy to read (82.1%), and that they understood the feedback (88.6%), and that the program was engaging (70.7%). Similarly, perceptions of program utility were also positive with the majority of students agreeing that they were interested in the feedback (69.9%) and that they would recommend the intervention to others (68.3%). Only 50.4%, however, agreed that they learned something from the intervention.

Sex Differences in Intervention Acceptability

Table 1 presents percent agreement, statistical contrasts, and effect sizes for the user-friendliness and utility items by sex.

Table 1.

Percent Agreement with Intervention Acceptability Items by Gender and Statistical Contrasts

% Agreement
Item Male (n = 55) Female (n = 68) χ2(1) p value Φ
User-Friendliness
 The program was easy to use. 98.1% 87.0% 5.31* .02 −.21
 The feedback was easy to read. 87.0% 78.3% 1.80 .18 −.12
 I was able to understand the feedback. 92.6% 85.5% 1.67 .20 −.12
 The program was engaging. 83.3% 60.9 % 5.91* .02 −.22
Utility
 I was interested in the feedback I received. 77.8% 65.2% 1.97 .16 −.13
 I learned something about myself. 63.0% 42.0% 4.47* .04 −.19
 I would recommend this program to others. 79.6% 59.4% 4.49* .03 −.19
Open in a new tab

Note. Φ = phi coefficient.

*

p < .05.

User-Friendliness

Results of the chi square analyses revealed significant sex differences in intervention user-friendliness. As seen in Table 1, results indicated males reported higher rates of agreement with the statements that the program was easy to use (p < .02) and the program was engaging (p < .01). Effect sizes for these items were in the small to medium range. There were no significant sex differences in rates of agreement with the statements the feedback was easy to read (p = .21) and the feedback was easy to understand (p = .22).

Utility

Results of the chi square analyses revealed significant sex differences in intervention utility. As seen in Table 1, results indicated males reported higher rates of agreement with the statements that they learned something (p < .02) and that they would recommend the program to others (p < .02). Effect sizes for these items were in the small to medium range. There were no significant sex differences in rates of agreement with the statement the feedback was interesting (p = .13).

Sex Differences in Alcohol Outcomes

Table 2 presents means, standard deviations, and statistical contrast for alcohol outcomes. Results of the repeated measures MANOVA revealed a significant main effect for time, Wilks’ Lambda = .91, F(2, 120) = 5.84, p = .004, η2p = .09. The effect size for time was in the medium to large range. The interaction effect for time x sex, Wilks’ Lambda = .98, F(2, 120) = 1.25, p = .29, η2p = .02, was not significant, indicating no significant sex differences in alcohol outcomes. As seen in Table 2, post-hoc repeated measures ANOVAs indicated a significant main effect for time for weekly drinking quantity (p < .001) and frequency of alcohol use (p < .05). The effect size for weekly drinking quantity was medium and the effect size for frequency of alcohol use was small.

Table 2.

Descriptive Statistics and Statistical Contrasts for Alcohol Outcomes

Male (n = 55) Female (n = 68) Total (n = 123) Time Time x Sex
M (SD) M (SD) M (SD) F(1, 121) p η2p F(1,121) p η2p
Weekly Drinking Quantity 10.89** <.001 .08 2.47 .12 .02
 Baseline 4.86 (6.40) 5.59 (7.31) 5.26 (6.90)
 30-Day Follow-Up 4.15 (5.81) 3.59 (4.75) 3.84 (5.24)
Frequency of Alcohol Use 3.99* <.05 .03 0.12 .73 .00
 Baseline 4.36 (1.36) 4.62 (1.38) 4.50 (1.41)
 30-Day Follow-Up 4.22 (1.52) 4.41 (1.50) 4.33 (1.51)
Open in a new tab
*

p < .05,

**

p < .01.

Sex Differences in the Relationship Between Intervention Acceptability and Alcohol Outcomes

Table 4 presents results from the regression models for weekly drinking quantity and frequency of alcohol use.

Table 4.

Summary of Hierarchical Regression Analysis for Frequency of Alcohol Use

Step Predictor ΔR2 B SE B β 95% CI
1 .61***
Baseline Alcohol Use .83 .06 .78 [.71, .95]
2 .00
Sex .01 .09 .01 [−.16, .19]
User-Friendliness −.04 .05 −.06 [−.13, .06]
Utility .02 .05 .03 [−.08, .11]
3 .02+
Sex x User-Friendliness −.12 .05 −.19* [−.22, −.01]
Sex x Utility .07 .05 .12 [−.02, .17]
Open in a new tab
+

p < .08,

*

p < .05,

***

p < .001.

Weekly Drinking Quantity

The full regression equation was significant for weekly drinking quantity, R2 = .62, F(6, 116) = 31.52, p < .001.This is a large effect size. As seen on Step 3, results showed that the sex x user-friendliness interaction term was significant (p < .01), indicating the relationship between weekly drinking quantity and user-friendliness was moderated by sex. No other main effects or interaction effects were significant. To examine the nature of the interaction, we graphed and interpreted tests of simple slopes using Aiken and West’s (1991) procedures. Figure 1 presents the significant two-way interaction between sex and intervention user-friendliness. Examination of the slopes in Figure 1 indicates that among males, those who perceived the intervention to be user-friendly reported lower levels of weekly drinking quantity than those who reported low user-friendliness, F(1,55) = 10.17, p < .01. The relationship between weekly drinking quantity and user-friendliness was not significant for females, F(1,68) = 2.06, p = .16.

Figure 1. The Relationship between User-Friendliness and Weekly Drinking Quantity for Males.

Figure 1

Open in a new tab

Note. This figure demonstrates model-based means (± 1 model-based standard error) for quantity of weekly drinking by user-friendliness for male students. The reduction in quantity of weekly drinking is significant for males who report high user-friendliness.

Frequency of Alcohol Use

The full regression equation was significant for frequency of alcohol use, R2 = .63, F(6, 116) = 33.09, p < .001. This is a large effect size. As seen on Step 3, results showed that the sex x user-friendliness interaction term was significant (p < .03), indicating the relationship between frequency of drinking and user-friendliness was moderated by sex. No other main effects or interaction effects were significant. To examine the nature of the interaction, we graphed and interpreted tests of simple slopes using Aiken and West’s (1991) procedures. Figure 2 presents the significant two-way interaction between sex and intervention user-friendliness. Examination of the slopes in Figure 2 indicates that among males, those who perceived the intervention to be user-friendly reported lower levels of frequency of alcohol use than those who reported low user-friendliness, F(1,55) = 3.96, p < .05. The relationship between frequency of alcohol use and user-friendliness was not significant for females, F(1,68) = 0.16, p = .69.

Figure 2. The Relationship between User-Friendliness and Frequency of Alcohol Use for Males.

Figure 2

Open in a new tab

Note. This figure demonstrates model-based means (± 1 model-based standard error) for frequency of alcohol use by user-friendliness for male students. The reduction in frequency of alcohol use is significant for males who report high user-friendliness.

Discussion

This is the first study to examine sex differences in high school students’ perceptions of the acceptability of the eCHECKUP TO GO and the relationship between program acceptability and alcohol outcomes. According to the ELM framework, interventions that contain content that is perceived as easy to understand and personally relevant should produce higher levels of elaboration and lead to greater behavioral change than traditional educational material (Petty et al., 2009). Results of this study indicate that although the majority of participants perceived the program to be user-friendly and useful, males reported higher levels of agreement with specific items related to both program user-friendliness and program utility. Additionally, although there were no sex differences in alcohol outcomes, findings demonstrated that for males, perceived user-friendliness of the program was significantly related to alcohol outcomes, whereas there was no relationship between program acceptability and alcohol outcomes for females.

Overall, students perceived the program to be more user-friendly than useful. These data are consistent with prior research demonstrating that among ninth grade students who have initiated alcohol use, 92.5 – 97.5% of students agreed with user-friendliness items whereas 77.5% – 92.5% agreed with utility items (Doumas, 2015). Results from this study, however, indicate that relative to the rates reported by Doumas (2015), students in this study reported lower levels of agreement with some of the user-friendliness items (70.7% – 91.9%) and considerably lower level of agreement with the utility items (50.4% – 69.9%). One explanation for this difference is that students in the current sample were high school seniors, whereas students in Doumas (2015) study were ninth grade students. It is possible that ninth grade students found the program to be more user-friendly and useful than high school seniors. Alternatively, ninth grade students may have responded in more socially desirable ways relative to the high school seniors in this sample. This explanation is supported by research indicating social desirability decreases with age among adolescents (Dadds et al., 1998).

Contrary to our hypothesis, findings from this study indicated that males perceived the eCHECKUP TO GO program to be more user-friendly and useful than females. Although there were sex differences in rates of agreement for the item “easy to use,” 87% females agreed that the program was easy to use (compared to 98.1% for males), suggesting high rates of agreement for both males and females. Sex differences for the item “engaging” were more pronounced, with only 60.9% of females agreeing the program was engaging relative to 83.3% of males. This is particularly important as the ELM suggests that engaging students in the content is essential to the formation of highly elaborated, accessible, well-integrated attitudes that are associated with behavioral change (Petty et al., 2009). For program utility, males agreed with the statements that they learned something and that they would recommend the program to other students more than females. In fact, less than two thirds of females indicated they learned something and less than half reported they would recommend the program to other students.

For alcohol outcomes, results supported positive effects for both males and females for weekly drinking quantity and frequency of alcohol use. Although some research suggests that among high school seniors, the eCHECKUP TO GO is efficacious in reducing risk factors for alcohol use for females only (Doumas, Esp, Johnson et al., 2017), some studies support intervention effects for alcohol outcomes for both males and females, with a wider-range of significant intervention effects for females (Doumas, Esp, Flay, & Bond, 2017; Doumas et al., 2020). Thus, this study adds to the growing body of literature supporting the efficacy of the eCHECKUP TO GO for both male and female high school seniors.

Finally, results of this study demonstrated that perceived program acceptability is related to alcohol outcomes for males, but not for females. Specifically, perceived user-friendliness of the eCHECKUP TO GO intervention was related to weekly drinking quantity and frequency of alcohol use for males only. Results of this study suggest that males perceived the program to be more user-friendly and useful than females and user-friendliness was related to alcohol outcomes for males only. Thus, consistent with ELM, perceived program acceptability was related to alcohol outcomes for males. However, we did not find evidence to support ELM for females. Alcohol outcomes for females were not related to either program user-friendliness or program utility, and appear to be related to other factors not measured in this study. Our hypothesis that females would report higher levels of program acceptability than males was based on results from prior studies suggesting intervention effects may favor females (Doumas, Esp, Flay, & Bond, 2017; Doumas, Esp, Johnson et al., 2017; Doumas et al., 2020). Thus, we were surprised by the finding that males reported higher levels of program acceptability and that program acceptability was related to alcohol outcomes for males only. It is, however, noteworthy that there were no differences in perceptions of the normative feedback between males and females for either user-friendliness (i.e., easy to read feedback; easy to understand feedback) or utility (i.e. interested in feedback ) items. This is an important finding as researchers have identified normative feedback as the primary mechanism of change in brief, personalized feedback interventions (Reid & Carey, 2015). Thus, high levels of acceptability concerning the personalized feedback may be related to positive outcomes for males, whereas the actual data presented in the personalized feedback may be related to positive outcomes for females. That is, males may be responding to the program overall as positive, whereas females may be responding to specific content despite finding the program to be less user-friendly and useful.

Study Strengths

The primary strength of this study was evaluating sex differences in the program acceptability of the eCHECKUP TO GO. This is the first study to investigate sex differences in program acceptability and the relationship between acceptability and alcohol outcomes among high school students. Because the eCHECKUP TO GO was originally developed for college students, establishing that the program is acceptable to high school students is important. Further, preliminary research demonstrating sex differences in outcomes favoring females suggests that males and females may be responding differently to the content of the intervention. This study provides some information regarding similarities and differences between male and female perceptions of program acceptability and the relationship of acceptability to alcohol outcomes. Additionally, this study examined program acceptability specifically for high school seniors. It is particularly important to identify relevant and effective alcohol interventions for high school seniors as seniors report the highest levels of alcohol use (Johnston et al., 2019) and hazardous drinking (Borsari et al., 2013; Zamboanga et al., 2016) although there is a dearth of research on alcohol interventions for this age group (Champion et al., 2013; Spoth et al., 2008; Strøm et al., 2014). It is important to establish appropriate and effective interventions for this age group. Findings from this study extend the literature, adding to the support for the use of the eCHECKUP TO GO with high school seniors.

Study Limitations

Although this study adds to the growing body of literature suggesting the eCHECKUP TO GO is a promising intervention for high school seniors, there are limitations. First, due to a primarily Caucasian sample form the Northwest region, generalizability is limited. Further, although participants were recruited from two high schools, the sample was relatively small. Future research with larger, more diverse samples is needed to increase the generalizability of the findings. Additionally, our examination of program acceptability was limited to a quantitative measure of user-friendliness and utility. Focus groups conducted with high school seniors could be used to gain a better understanding of program acceptability including how to make the program more engaging for female students and how to increase learning for both male and female students.

Implications

Results of this study have important implications for developing prevention and intervention programs for high school seniors. First, findings from this study and prior research (Doumas, 2015) indicate high school students perceive the eCHECKUP TO GO program to be both user-friendly and useful. Research suggests that the program may be more acceptable to students who have initiated alcohol use (Doumas, 2015) and results from the present study suggest male students may find the program more acceptable than female students. Overall, however, research supports program acceptability for the eCHECKUP TO GO for high school students.

Second, results of this study provide support for the acceptability of the eCHECKUP TO GO program for high school seniors. It is important to note, however, that high school seniors reported lower rates of program utility relative to program user-friendliness and lower rates of acceptability relative to a ninth grade sample. Although our data suggest that males and females perceived the personalized feedback similarly, approximately one third of seniors did not perceive the feedback to be interesting. Similarly, approximately 40% of females indicated they did not agree that the program was engaging. Thus, modification of how the eCHECKUP TO GO feedback is presented may be warranted for this age group. Specifically, although some feedback is provided using graphical depictions of the data, high school seniors might find the feedback more interesting if it was interactive in nature. For example, instead of providing feedback regarding protective behavioral strategies in text format, students could select an avatar and walk through a situation in which alcohol is being served at a party, making decision about which strategies to use to limit their drinking or alcohol-related consequences and experiencing outcomes. Consistent with the ELM, providing interactive elements may increase engagement and interest in the program, which in turn, might lead to increased intervention efficacy (Petty et al., 2009).

Finally, results of this study provided support for the effectiveness of the eCHECKUP to go in reducing quantity and frequency of alcohol use among high school seniors. Web-based programs are well-suited for school-based implementation as they are inexpensive, require minimal training, and can be implemented with a high degree of fidelity. Additionally, they are easy to disseminate to large groups of students and can be incorporated within course curricula. Research supports the integration of alcohol intervention programs into the PBIS framework for high school students (Bastable et al., 2015). Thus, schools can integrate the eCHECKUP TO GO program into the Positive Behavioral Interventions and Supports (PBIS) framework. A tiered approach to alcohol prevention and intervention might include implementing the eCHECKUP TO GO to all students as part of a school assembly or through advising sessions (Tier 1), intervening with groups of students who are at high-risk for alcohol use, including high school seniors, through classroom curricula (Tier 2), and providing the eCHECKUP TO GO for students who have been sanctioned for an alcohol use violation at school with the school psychologist (Tier 3). Research suggests the eCHECKUP TO GO is an effective universal Tier 1 approach for prevention with high school students (Doumas et al., 2020; Doumas et al., 2014; Hausheer et al., 2018) and an effective Tier 2 approach with high school seniors reporting high-risk drinking (Doumas & Esp, 2019; Doumas, Esp, Flay, & Bond, 2017). Results from this study add to this literature demonstrating a reduction in alcohol use among high school seniors who report alcohol use. Although we could not find evidence for Tier 3 in high school, research suggests that the eCHECKUP TO GO is also an effective Tier 3 approach for college students who received alcohol violations (Alfonso et al., 2013; Doumas, Workman, Navarro, & Smith, 2011; Doumas, Workman, Smith, & Navarro, 2011). Thus, it is likely the eCHECKUP TO GO would be an effective Tier 3 approach at the high school level.

Conclusion

This is the first study to evaluate sex differences in the acceptability of the eCHECKUP TO GO for high school seniors. Given the high rates of drinking and negative consequences associated alcohol use among high school seniors, it is important to identify appropriate and effective alcohol interventions for this age group. Although findings indicate the program was more acceptable to males then females, results from this study, coupled with outcome research suggest the eCHECKUP TO GO is a promising alcohol intervention for high school seniors. Increasing the interactive nature of the personalized feedback, however, may result in a more interesting and engaging program for high school seniors. This may be particularly important for males, as perceived user-friendliness is related to alcohol outcomes, and prior research has demonstrated a wider-range of intervention effects for females.

Table 3.

Summary of Hierarchical Regression Analysis for Weekly Drinking Quantity

Step Predictor ΔR2 B SE B β 95% CI
1 .57***
Baseline Weekly Drinking .57 .05 .75 [.48, .66]
2 .02
Sex .45 .32 .09 [−.18, 1.08]
User-Friendliness −.24 .18 −.11 [−.59, .11]
Utility .24 .18 .11 [.08, .16]
3 .04**
Sex x User-Friendliness − .47 .19 −.22* [−.84, −.11]
Sex x Utility −.04 .17 −.02 [−.38, .29]
Open in a new tab
*

p < .05,

**

p < .01,

***

p < .001.

Acknowledgments

Research reported in this publication was supported by the National Institute on Alcohol Abuse and Alcoholism of the National Institutes of Health under Award Number R21AA023880. This project was also supported in part by Institutional Development Awards (IDeA) from the National Institute of General Medical Sciences of the National Institutes of Health under Grants #P20GM103408 and P20GM109095. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. We also acknowledge support from The Biomolecular Research Center at Boise State with funding from the National Science Foundation, Grants # 0619793 and #0923535, the MJ Murdock Charitable Trust, and the Idaho State Board of Education.

The authors wish to thank Andrea Pikes, April Watts, Beverly Glouser, and Dayna Showalter for assistance with participant recruitment and data collection.

Footnotes

This study was registered with ClinicalTrials.gov (Identifier NCT03613818).

We have no known conflicts of interest to declare.

References

  1. Aiken LS, & West SG (1991). Multiple regression: Testing and interpreting interactions. Sage Publications, Inc. [Google Scholar]
  2. Alfonso J, Hall TV, & Dunn ME (2013). Feedback-based alcohol interventions for mandated students: An effectiveness study of three modalities. Clinical Psychology & Psychotherapy, 20, 411–423. 10.1002/cpp.1786 [DOI] [PubMed] [Google Scholar]
  3. Arata CM, Stafford J, & Tims MS (2003). High school drinking and its consequences. Adolescence, 38, 567–579. 10.1037/adb0000041 [DOI] [PubMed] [Google Scholar]
  4. Bastable E, Kittelman A, McIntosh K, & Hoselton R (2015). Do high schools implementing SWPBIS have lower rates of illegal drug and alcohol use? City, 29, 8 https://www.pbis.org/ [Google Scholar]
  5. Borsari B, Zamboanga BL, Correia C, Olthuis JV, Van Tyne K, Zadworny Z, Grossbard JR, & Horton NJ (2013). Characterizing high school students who play drinking games using latent class analysis. Addictive Behaviors, 38, 2532–2540. 10.1016/j.addbeh.2013.04.009 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Burrow-Sanchez JJ, Jenson WR, & Clark E (2009). School-based interventions for students with substance abuse. Psychology in the Schools, 46, 238–245. 10.1002/pits.20368 [DOI] [Google Scholar]
  7. Champion KE, Newton NC, Barrett EL, & Teesson M (2013). A systematic review of school-based alcohol and other drug prevention programs facilitated by computers or the Internet. Drug and alcohol review, 32, 115–123. 10.1111/j.1465-3362.2012.00517.x [DOI] [PubMed] [Google Scholar]
  8. Cohen J (1969). Statistical power analysis for the behavioral sciences. Routledge. [Google Scholar]
  9. Collins RL, Parks GA, & Marlatt GA (1985). Social determinants of alcohol consumption: The effects of social interaction and model status on the self-administration of alcohol. Journal of Consulting and Clinical Psychology, 53, 189–200. 10.1037/0022-006X.53.2.189 [DOI] [PubMed] [Google Scholar]
  10. Dadds MR, Perrin S, & Yule W (1998). Social desirability and self-reported anxiety in children: An analysis of the RCMAS Lie Scale. Journal of Abnormal Child Psychology, 26, 311–317. 10.1023/A:1022610702439 [DOI] [PubMed] [Google Scholar]
  11. Dimeff LA, Baer JS, Kivlahan DR Marlatt GA (1999). Brief Alcohol Screening and Intervention for College Students (BASICS): A harm reduction approach. Guilford Press. [Google Scholar]
  12. Doumas DM (2015). Web-based personalized feedback: is this an appropriate approach for reducing drinking among high school students?. Journal of Substance Abuse Treatment, 50, 76–80. 10.1016/j.jsat.2014.09.005 [DOI] [PubMed] [Google Scholar]
  13. Doumas DM, & Esp S (2019). Reducing alcohol-related consequences among high school seniors: Efficacy of a brief, web-based intervention. Journal of Counseling and Development, 97, 53–61. 10.1002/jcad.12235 [DOI] [Google Scholar]
  14. Doumas DM, Esp S, Flay B, & Bond L (2017). A randomized controlled trial testing the efficacy of a brief online alcohol intervention for high school seniors. Journal of Studies on Alcohol and Drugs, 78, 706–715. 10.15288/jsad.2017.78.706 [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Doumas DM, Esp S, Johnson J, Trull R, & Shearer K (2017). The eCHECKUP TO GO for High School: Impact on risk factors and protective behaviors for alcohol use. Addictive Behaviors, 64, 93–100. 10.1016/j.addbeh.2016.08.030 [DOI] [PubMed] [Google Scholar]
  16. Doumas DM, Esp S, Turrisi R, Bond L, & Flay B (2020). The efficacy of the eCHECKUP TO GO for high school seniors: Sex differences in risk factors, protective behavioral strategies, and alcohol use. Journal of Studies on Alcohol and Drugs, 81, 135–143. 10.15288/jsad.2020.81.135 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Doumas DM, Esp S, Turrisi R, Hausheer R, & Cuffee C (2014). A test of the efficacy of a brief, web-based personalized feedback intervention to reduce drinking among 9th grade students. Addictive Behaviors, 39, 231–238. 10.1016/j.addbeh.2013.10.011 [DOI] [PubMed] [Google Scholar]
  18. Doumas DM, Esp S, Turrisi R, & Schottelkorb A (2015). Adopting a brief, web-based feedback approach to counter high school drinking: Considerations for professional school counselors. Professional School Counseling, 18, 148–157. 10.1177/2156759X0001800105 [DOI] [Google Scholar]
  19. Doumas DM, Workman CR, Navarro A, & Smith D (2011). Evaluation of web-based and counselor delivered feedback interventions for mandated college students. Journal of Addictions and Offender Counseling, 32, 16–28. 10.1002/j.2161-1874.2011.tb00204.x [DOI] [Google Scholar]
  20. Doumas DM, Workman CR, Smith D, & Navarro A (2011). Reducing high-risk drinking in mandated college students: Evaluation of two personalized normative feedback interventions. Journal of Substance Abuse Treatment, 40, 376–385. 10.1016/j.jsat.2010.12.006 [DOI] [PubMed] [Google Scholar]
  21. Erford BT (2015). Research and evaluation in counseling. Cengage Learning. [Google Scholar]
  22. Faul F, Erdfelder E, Lang AG, & Buchner A (2007). G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191. 10.3758/bf03193146 [DOI] [PubMed] [Google Scholar]
  23. Glass JE, Grant JD, Yoon HY, & Bucholz KK (2015). Alcohol problem recognition and help seeking in adolescents and young adults at varying genetic and environmental risk. Drug and Alcohol Dependence, 153, 250–257. 10.1016/j.drugalcdep.2015.05.006 [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Hausheer R, Doumas DM, & Esp S (2018). Evaluation of a Web-Based Alcohol Program Alone and in Combination With a Parent Campaign for Ninth-Grade Students. Journal of Addictions & Offender Counseling, 39, 15–30. 10.1002/jaoc.12038 [DOI] [Google Scholar]
  25. Holm S (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 65–70. https://www.jstor.org/stable/4615733 [Google Scholar]
  26. Hanson KL, Medina KL, Padula CB, Tapert SF, & Brown SA (2011). Impact of adolescent alcohol and drug use on neuropsychological functioning in young adulthood: 10-year outcomes. Journal of Child & Adolescent Substance Abuse, 20, 135–154. 10.1080/1067828X.2011.555272 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Johnston LD, Miech RA, O’Malley PM, Bachman JG, & Schulenberg JE, & Patrick ME (2019). Monitoring the Future national survey results on drug use 1975–2018: Overview, key findings on adolescent drug use. Institute for Social Research, The University of Michigan. http://www.monitoringthefuture.org/pubs/monographs/mtf-overview2018.pdf [Google Scholar]
  28. Kelly AB, Evans-Whipp TJ, Smith R, Chan GC, Toumbourou JW, Patton GC, … & Catalano RF(2015). A longitudinal study of the association of adolescent polydrug use, alcohol use and high school non-completion. Addiction, 110, 627–635. 10.1111/add.12829 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Marlatt GA, Baer JS, Kivlahan DR, Dimeff LA, Larimer ME, Quigley LA, et al. (1998). Screening and brief intervention for high-risk college student drinkers: Results from a 2-year follow-up assessment. Journal of Consulting and Clinical Psychology, 66, 604–615. 10.1037/0022-006X.66.4.604 [DOI] [PubMed] [Google Scholar]
  30. Miller JW, Naimi TS, Brewer RD, & Jones SE (2007). Binge drinking and associated health risk behaviors among high school students. Pediatrics, 119, 76–85. 10.1542/peds.2006-1517 [DOI] [PubMed] [Google Scholar]
  31. National Institute on Alcohol Abuse and Alcoholism (2015). Planning alcohol interventions using NIAAA’s college aim alcohol intervention matrix. NIAAA; https://www.niaaa.nih.gov/ [Google Scholar]
  32. Nguyen-Louie T, Castro N, Matt G, Squegelia L, Brumback T, & Tapert S (2015). Effects of emerging alcohol and marijuana use behaviors on adolescents’ neuropsychological functioning over four years. Journal of Studies on Alcohol and Drugs, 76, 738–748. 10.15288/jsad.2015.76.738 [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Norman GR, & Streiner DL (2008). Biostatistics: The bare essentials. PMPH USA. [Google Scholar]
  34. Patte KA, Qian W, & Leatherdale ST (2017). Marijuana and alcohol use as predictors of academic achievement: a longitudinal analysis among youth in the COMPASS study. Journal of School Health, 87, 310–318. 10.1111/josh.12498 [DOI] [PubMed] [Google Scholar]
  35. Petty RE, & Cacioppo JT (1986). The Elaboration Likelihood Model of persuasion In Berkowitz L (Ed.), Advances in experimental social psychology (pp. 123–205). Academic Press. [Google Scholar]
  36. Petty RE, Barden J, & Wheeler SC (2009). The Elaboration Likelihood Model of persuasion: Developing health promotions for sustained behavioral change In DiClemente RJ, Crosby RA, & Kegler MC (Eds.), Emerging Theories in Health Promotion Practice and Research (pp. 185–214). Jossey-Bass. [Google Scholar]
  37. Rasberry CN, Tiu GF, Kann L, McManus T, Michael SL, Merlo CL, Lee SM, Bohm MK, Annor F, & Ethier KA (2017). Health-related behaviors and academic achievement among high school students—United States, 2015. MMWR. Morbidity and Mortality Weekly Report, 66, 921 10.15585/mmwr.mm6635a1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Reavley NJ, Cvetkovski S, Jorm AF, & Lubman DI (2010). Help-seeking for substance use, anxiety and affective disorders among young people: results from the 2007 Australian National Survey of Mental Health and Wellbeing. Australian and New Zealand Journal of Psychiatry, 44, 729–735. 10.3109/00048671003705458 [DOI] [PubMed] [Google Scholar]
  39. Reid AE, & Carey KB (2015). Interventions to reduce college student drinking: State of the evidence for mechanisms of behavior change. Clinical Psychology Review, 40, 213–224. 10.1016/j.cpr.2015.06.006 [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Richardson JTE (2011). Eta squared and partial eta squared as measurements of effect size in educational research. Educational Research Review, 6, 135–147. 10.1016/j.edurev.2010.12.001 [DOI] [Google Scholar]
  41. San Diego State University Research Foundation (n.d.). http://www.echeckuptogo.com/
  42. Schinke S, Schwinn T, & Cole K (2006). Preventing alcohol abuse among early adolescents through family and computer-based interventions: Four-year outcomes and mediating variables. Journal of Developmental and Physical Disabilities, 18, 149–161. 10.1007/s10882-006-9009-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Spoth R, Greenberg M, & Turrisi R (2008). Preventive interventions addressing underage drinking: State of the evidence and steps toward public health impact. Pediatrics, 121, 311–336. 10.1542/peds.2007-2243E [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Strøm HK, Adolfsen F, Fossum S, Kaiser S, & Martinussen M (2014). Effectiveness of school-based preventive interventions on adolescent alcohol use: A meta-analysis of randomized controlled trials. Substance Abuse Treatment, Prevention, and Policy, 9, 48 10.1186/1747-597X-9-48 [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Tabachnick BG, & Fidell LS (2007). Using multivariate statistics (5th ed.). Allyn and Bacon. [Google Scholar]
  46. Townsend L, Flisher AJ, & King G (2007). A systematic review of the relationship between high school dropout and substance use. Clinical Child and Family Psychology Review, 10, 295–317. 10.1007/s10567-007-0023-7 [DOI] [PubMed] [Google Scholar]
  47. Wagner EF, Tubman JG, & Gil AG (2004). Implementing school-based substance abuse interventions: Methodological dilemmas and recommended solutions. Addiction, 99 (Suppl. 2), 106–119. 10.1111/j.1360-0443.2004.00858.x [DOI] [PubMed] [Google Scholar]
  48. Wang MT, & Fredricks JA (2014). The reciprocal links between school engagement, youth problem behaviors, and school dropout during adolescence. Child Development, 85, 722–737. 10.1111/cdev.12138 [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Zamboanga BL, Tomaso CC, Cloutier RM, Blumenthal H, Kenney SR, & Borsari B (2016). Drinking game participation among high school and incoming college students. Journal of Addictions Nursing, 24(1), E1 10.1097/01.JAN.0000481966.33116.a5 [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES