Working with Patients with Alcohol Problems: A Controlled Trial of the Impact of a Rich Media Web Module on Medical Student Performance

Abstract

INTRODUCTION/AIMS

We designed an interactive web module to improve medical student competence in screening and interventions for hazardous drinking. We assessed its impact on performance with a standardized patient (SP) vs. traditional lecture.

SETTING

First year medical school curriculum.

PROGRAM DESCRIPTION

The web module included pre/posttests, Flash©, and text didactics. It centered on videos of two alcohol cases, each contrasting a novice with an experienced physician interviewer. The learner free-text critiqued each clip then reviewed expert analysis.

PROGRAM EVALUATION

First year medical students conveniently assigned to voluntarily complete a web module (N = 82) or lecture (N = 81) were rated by a SP in a later alcohol case. Participation trended higher (82% vs. 72%, p < .07) among web students, with an additional 4 lecture-assigned students crossing to the web module. The web group had higher mean scores on scales of individual components of brief intervention (assessment and decisional balance) and a brief intervention composite score (1–13 pt.; 9 vs. 7.8, p < .02) and self-reported as better prepared for the SP case.

CONCLUSIONS

A web module for alcohol use interview skills reached a greater proportion of voluntary learners and was associated with equivalent overall performance scores and higher brief intervention skills scores on a standardized patient encounter.

INTRODUCTION/AIMS

Web-based learning increasingly shapes medical education.¹ Addiction, a traditionally neglected field in undergraduate medical education, will likely be more widely covered if adapted to scalable and accessible computer-assisted formats.^2–6 Evaluations of web-based addiction and communication skills modules have infrequently employed longitudinal performance evaluations such as standardized patient (SP) encounters.^7–9 To date, no published studies have evaluated web-based approaches to teaching alcohol screening and brief intervention via observed structured clinical examinations (OSCEs) or comparable measures.

This study sought to pilot a web-based module on screening and brief intervention skills for hazardous drinking, in conjunction with efforts at NYU to expand and evaluate web-based undergraduate medical education, and designed according to current multimedia design theory emphasizing learner activation and balanced cognitive load.^10–11 With a cohort of a first year medical students, we hypothesized that the module would be more effective than traditional lecture at teaching interview skills as measured by student performance on an OSCE alcohol case. The feasibility and acceptability of the module would be evaluated via attendance rates, comparisons of pre/posttest scores of knowledge and attitudes, and direct student feedback.

DESCRIPTION

Study Design First-year medical students at New York University Medical School learn about communication skills and public and behavioral health topics within a physician, patient and society (PPS) course. Lecture, web modules, and course syllabus prepare students for faculty-facilitated small group seminars and OSCEs over 1 year. Screening and brief intervention skills for hazardous drinking have previously been taught via 3 chronological sessions: (1) lecture, (2) small-group seminar involving the students interviewing an alcohol-dependent patient with faculty facilitation, and (3) OSCE case of a healthy woman exceeding daily drinking limits who, if advised, agrees to cut down and follow up with the physician.

To pilot and evaluate a newly developed web module, 2006–2007 PPS students (N = 163) were conveniently (non-randomly) assigned by class schedule to complete a web module over a 1-week period or attend a 1-hour lecture. Group assignments were delivered via regular course communication as an initial introduction and reminder email. All current NYUSOM students are required to own laptops, communicate with faculty via email, and access course materials online. Completion (attendance) of either the module or lecture was voluntary. Within 3–5 weeks of module/lecture exposure, both groups attended seminars with an alcohol patient followed by an OSCE including the alcohol case. Baseline student demographic data were not collected. There was no a priori hypothesis regarding demographics and performance. Further, the NYUSOM IRB, which approved this protocol, prohibits the collection of student demographics (age, ethnicity) due to small numbers of certain ethnicities and age groups and to ensure deidentified reported data.

Web Module Features A Java©-based web module was developed by course (J.L., A.K.) and NYU Division of Educational Informatics (M.T.) faculty. The module is publicly accessible: http://edinfo.med.nyu.edu/alcoholscreening/ (login: demo; password: demo). During a 1-week window, students logged-in using unique IDs and passwords and were asked to electronically consent to use of pre/posttest data for research purposes. Regardless of consent status, students were able to complete the web module. A pretest consisting of 12 multiple-choice knowledge and 16 Likert-scale attitude items adapted from Saitz et al.⁴ was used to orient the learner to module content and identify knowledge deficits. Knowledge items covered safe drinking limits and assessment and advice for two alcohol vignettes. A sidebar outlined content while audio narration offered instruction and emphasized core principles. Flash© animation and hyperlinks to external sources were used to highlight important concepts while minimizing text. The central teaching mechanism was video clips of contrasting physician interviews of two alcohol patients, a male with pancreatitis and alcohol dependence who agrees to abstain from drinking and a female who exceeds safe drinking limits and agrees to cut down. A novice student interview was followed by an experienced physician, the later providing expert examples of interview techniques. NIAA guidelines (AUDIT-C and CAGE questionnaires, drinking categorization and assessment, patient education and feedback, and collaboration for action and follow-up) were demonstrated.¹² Following each 1–3 minute clip, the learner answered relevant questions within a free-text field. Learner responses were immediately paired to written expert teaching. The module concluded with a posttest repeating the same pretest items, a comparison of pre/posttest answers with key, and lastly module feedback prompts. The module was designed to require approximately 45 minutes.

Lecture The lecture shared the web module outline (alcohol epidemiology, screening approaches, and brief intervention) and was delivered by a module author and local addiction expert (J.L) using PowerPoint and video. The common module author/lecturer and outline were intended to maximize standardized content across interventions. Students indicated consent status and completed pre/posttests in writing. Video from the first but not second alcohol case was reviewed due to limited time. Throughout the hour, the lecturer stopped for Q&A concerning key teaching points.

Small Group Seminars One to three weeks after web/lecture exposure, students participated in seminars in which alcohol screening and brief intervention was reviewed, and groups interviewed an invited patient with alcohol dependence.

OSCE Alcohol Case and Performance Measures A two-case OSCE using SP raters followed 3–5 weeks after the web/lecture module. Per previously developed methods, 8 experienced female SPs were trained during a 1-day workshop to deliver a case of an adult woman with hazardous drinking who needs to cut down or abstain and then reliably evaluate student performance on behaviorally anchored rating scales.^9,13–14 Evaluation items included classifications of student performance on AUDIT-C, CAGE, and 6 brief intervention components (classification of drinking, patient education, assess stage of change, decisional balance, planning follow-up, assessing patient understanding). Item scales consisted of dichotomous (not done/done adequately) and 3 pt. (not done/partially done/done well) measures. CAGE (0–4 pt.), AUDIT-C (0–5 pt.), and composite brief intervention (1–13 pt.) combined scores were calculated post hoc. Students completed a validated self-rating (1–3 pt., poor/good/excellent) of their overall performance, the extent to which their performance reflected their clinical skills, and their level of preparation.⁹ Ratings were completed immediately following the case, after which the SP gave verbal feedback. SPs were blinded to web/lecture assignment.

Data Collection and Analysis Web module data (log-in time, pre/posttest, Q&A free-text responses) and module-specific feedback was captured electronically. Written data (lecture pre/posttests, SP, and student OSCE evaluations) were entered via Excel. Web/lecture status, seminar and faculty assignments, and SP identities were tracked. Data were analyzed using Stata SE 8.0. Analysis consisted of multiple-choice test scoring, descriptive statistics, intention-to-treat tests of differences between group means [t test, analysis of variance (ANOVA)], Cohen’s D statistics for effect sizes, and regression models estimating the effects of web vs. lecture assignment on OSCE performance while controlling for individual SPs.

EVALUATION

Of first year medical students assigned to the web (N = 82) vs. lecture (N = 81) groups, an equivalent proportion of students voluntarily completed the web module than attended lecture (82% vs. 72%, p < .07; Table 1). Four lecture-assigned students did not attend lecture but instead completed the web module, increasing total web participation to 71 students vs. 58 at lecture. The web-assigned cohort scored higher on 12 knowledge items on both the pre- (72% vs. 67% items correct, p < .03) and posttest (86% vs. 77%, p < .005), differences that were largely explained by higher scores by web students on 4 items testing knowledge of safe drinking limits (Table 1). Both groups showed similar improvement in knowledge pre/posttest. Web students rated their pretest confidence to address patients’ alcohol use higher on a 1–4 pt scale (2.3 vs. 2.1, p < .03). Both groups showed equal confidence on the same posttest item (3.1 vs. 3.1).

Table 1.

Web vs Lecture Assignment: Attendance and Pre/Posttest Knowledge

	Web Module	Lecture	ANOVA (p)
Students assigned, N	82	81
Attendance, n (%)	67 (82)	58 (72)	<.07
Pre/posttest knowledge safe drinking limits, 4 items
Pretest, % correct	61	53	<.03
Posttest, % correct	88	73	<.001
Change, % correct	+27	+20	<.08
Total 12-item knowledge test
Pretest, % correct	72	67	<.03
Posttest, % correct	86	77	<.005
Change, % correct	+14	+11	<.2

Students spent a mean 47 minutes (range 11–120 min) logged into the module. Three students were unable to complete the module due to technical difficulties relating to browser software. Three other students spent less than 20 minutes on the entire module and did not meaningfully participate in free-text exercises. The remaining 61 Q&A free-text responses of web-assigned students were coherent and appropriate. Post-module feedback was largely supportive of the module and online learning (“I like the online module almost better than having seminars because I had the opportunity to go back and review concepts that I didn’t fully learn the first time…”, “…[T]he module was a much more efficient way to receive the material….[L]ectures and classes can only be interactive to a degree and it is easy to drift off in thought of just get lost because of the lecturer’s style…the online modules require active participation.”) Other students preferred traditional lecture or expressed frustration with technical difficulties.

OSCE scores were analyzed by intention-to-treat comparisons of web vs. lecture assignment, with the 4 lecture students who completed the web module included in the original lecture cohort (Table 2). Performance on CAGE questions were no different between groups, nor were mean single-item 1–3 pt. scores for stage of change assessment, developing follow-up plans, patient education, or assessing patient understanding. Web-assigned students scored higher, but not statistically significantly so on a 0–5 pt. scale of AUDIT-C performance (3.4 vs. 3.2, p < .07), and significantly higher on 2 single-item 1–3 pt. scales for assessment of hazardous drinking (2.8 vs. 2.5, p < .02) and employing decisional balance (1.6 vs. 1.2, p < 0.003), and a composite 1–13 pt. scale of brief intervention skills (9 vs. 7.8, p < .02). Higher group mean scores reflected more student performances rated by the SPs as partially done or done well, and effect sizes were moderate (0.3–0.5). Web vs. lecture students self-rated as better prepared for the case (1–3 pt. scale; 2.7 vs. 2.5, p < .006), while both groups self-rated their case performances equally (2.3 vs. 2.3).

Table 2.

Web vs. Lecture Assignment: OSCE Performance Ratings

	Web Module (N = 82)	Lecture (N = 81)	ANOVA (p)	Effect Size*
AUDIT-C performance (0–5 pt. scale)	3.4	3.2	<0.07
CAGE performance (0–4 pt. scale)	3.6	3.6	<0.5
Brief intervention components (1–3 pt. scales: not done/partially done/well done)
Assessment of patient’s drinking as hazardous	2.8	2.5	<0.02	0.40
Stage of change assessment	2.6	2.5	<0.2
Decisional balance: reviewed drinking positives and negatives	1.6	1.2	<0.003	0.48
Developed a follow-up plan	2.4	2.2	<0.4
Educated patient regarding treatment rationale	2.5	2.4	<0.4
Checked for patient understanding	2.5	2.4	<0.2
Brief intervention skills, combined score (1–13 pt. scale)	9	7.8	<0.02	0.38

*Cohen’s D statistic

Linear regression models estimated these results did not vary by seminar group and faculty or SP identity (data not shown). Only completion of the web module predicted higher OSCE performance. Those attending lecture (N = 58) performed no differently on the AUDIT-C or brief interventions scales than students (N = 34) from either group who did not participate in the module or lecture.

DISCUSSION

This pilot of a web module for teaching guideline-based screening and brief intervention for hazardous drinking demonstrated acceptable feasibility and equivalent overall downstream effects on first year medical student clinical skills. A higher although not statistically significantly different proportion of students completed the web module than attended lecture, with an additional 5% of lecture students “defecting” to the web module contrary to course instructions. However, 6 of 71 students logging into the module were either unable (technical difficulties) or unwilling (‘clicked thru’) to participate. Written feedback from web students was largely supportive of online learning vs. classroom teaching.

Mean composite scores of web-assigned students for performance on brief intervention items were higher than or equal to the lecture group’s, which did not achieve higher scores on any items. Web-assigned students were more likely to adequately assess the patient’s drinking as hazardous and lead the patient in decisional balance. This supports the web module as equivalent to standard lecture-based instruction, with evidence for superiority on specific brief intervention components. It is not known if differences on individual items indicate meaningful contrasts in clinical skills or predict better outcomes in post-graduate practice. Effect sizes of 0.3–0.5 on the individual brief intervention items favoring the web group suggest modest differences between groups.

Improvements on several limitations are needed. Only randomization would control for baseline differences between groups, without which, it is unknown if higher knowledge and confidence scores among web students are real, an effect of web-based testing, learner style, or demographics. Presenter bias favoring the web module may have occurred by using a web author as lecturer, although this was a deliberate strategy to standardize content. We endeavored to maximize SP rating reliability, yet SPs may overestimate student performance and obscure true differences between groups.¹⁵ Finally, this web module is intended for adaptation to graduate medical education and linkage to patient outcomes, including rates of alcohol screening, referrals, and reduced drinking. Such outcomes are not relevant to first year medical student learners.

These results suggest that online clinical education can be developed using best practice multimedia principles and evaluated for impact on clinical skills. It is crucial that the growing volume of addiction-oriented online modules justify their upfront development costs and wide dissemination with evidence of effectiveness and improved public health.