Undergraduate Medical Education in Substance Abuse: A Review of the Quality of the Literature

Abstract

Purpose

To prepare to develop a medical school curriculum on substance abuse disorders (SAD), the authors conducted a review of the quality of the sparse published literature.

Method

The authors searched MEDLINE (1950–December 2008), Web of Science, PsycINFO, and PubMed to identify all studies of SAD interventions targeted toward undergraduate medical students. Of the 1,084 studies identified initially, 31 reported sufficient data to allow the authors to evaluate quality using Medical Education Research Study Quality Instrument (MERSQI) scores. The authors also determined the impact of the studies by considering three-year citation rate and journal impact factor. A detailed review of the literature provided data on contact hours and intervention content.

Results

The three-rater intra-class correlation coefficient (ICC) for total MERSQI score was 0.82 (95% confidence interval: 0.70–0.90). The mean MERSQI score was 10.42 of a possible 18 (SD 2.59; range: 6.33–14.83). MERSQI scores were higher for more recently published studies and correlated with three-year citation rate but not impact factor. The mean contact time for 26 studies was 29.25 hours (range: 0.83–200 hours).

Conclusions

The literature provides a variety of educational methods to train medical students in SAD detection and intervention skills. This literature is of variable quality and provides limited guidance for development of curricula and medical education policy. Better methods of curriculum evaluation and publication guidelines would help ensure this literature has a positive impact on educational practice and public health.

Alcohol- and drug-related disorders are major preventable and treatable personal and public health problems. In the United States alone, drug abuse is responsible each year for more than 25,000 deaths and $100 billion in total economic costs.^1–3 Physicians are often substance abusers’ first contact with the health care system, and substance abuse disorders (SAD) have been estimated to be as common as hypertension among patients in primary care practices.⁴ Therefore, if physicians and other health care providers were to be appropriately trained in the detection of and initial interventions for problem substance use, they could have an important impact on the treatment of SAD.

The Licensing Committee on Medical Education (LCME), the accreditation body for medical schools’ undergraduate programs, and the Association of American Medical Colleges have recognized this issue by including design of SA curricula among their 31 “hot topics.”⁵ Expert panels, including one sponsored by the White House, have made explicit recommendations for medical student education, emphasizing the growing evidence to support the argument that all health care providers should demonstrate competence in SAD screening, brief intervention, and referral for treatment.^6,7 Despite the building consensus, undergraduate medical education concerning substance abuse (SA) is woefully sparse. Further, the existing literature describing such SA curricula has been criticized for its lack of systematic assessment data to support specific educational approaches.⁸

The quality of medical education research in general has been widely criticized,^9–13 and researchers have attempted to objectively and systematically measure the methodological quality of published studies. In response to this challenge, Reed et al¹⁴ recently developed the Medical Education Research Study Quality Instrument (MERSQI), a reliable and validated instrument which provides a standardized framework for assessing published reports of medical education research. Higher MERSQI scores have been shown to be predictive of editorial decisions and are associated with studies with external funding; higher expert quality scoring, three-year citation rates, and journal impact factors; and the number of previous medical education publications by the first author.^14,15

As we prepared to develop a model medical school curriculum on SA, we sought to identify what works by surveying and evaluating the quality of the literature. In this study, we report MERSQI scoring of the existing literature on undergraduate medical education in SA, specifically alcohol and drug abuse, discuss general trends, and make recommendations based on this review.

Method

Identifying the literature

In January 2009, the first author (D.K.) systematically searched MEDLINE (1950–December 2008) using OVID, PsycINFO, and PubMed to identify articles concerning undergraduate medical education on SA issues. With the help of medical librarians at the New York University School of Medicine, she searched the following pairs of keywords: substance-related disorders AND medical education (undergraduate medical education), substance-related disorders AND curriculum, alcoholism AND medical education (undergraduate medical education), and alcoholism AND curriculum. We decided to exclude tobacco from the MeSH terms of “substance-related disorders,” because there was a recently published, comprehensive review of tobacco intervention training.¹⁶

These keyword searches yielded 193, 474, 161, and 256 studies, respectively, for a total of 1,084 studies. By reviewing study titles for mention of relevant terms (e.g., medical students, substance abuse, clerkship, intervention, evaluation) and then reviewing their abstracts, D.K. eliminated 1,034 studies. Further restricting the remaining 50 studies to those that involved interventions or evaluations related to medical students’ competence in SAD or alcoholism (but not tobacco) resulted in 37 studies. Two authors (D.K. and A.K.) eliminated 6 more papers because they offered insufficient data to determine MERSQI scores, leaving us with 31 studies for this review.

MERSQI scoring of the literature

The MERSQI assesses the quality of published medical education research reports on 10 items in six domains: study design, sampling method (number of institutions involved and response rate), type of data, validity of evaluation instrument, data analysis, and outcomes (see Table 1). In each domain, the maximum score is 3; the total possible MERSQI score for any study is 18, with a range of 5–18, with higher scores reflecting more rigorous study designs and valid outcome measures.

Table 1.

Summary of Medical Education Research Study Quality Instrument (MERSQI) Domain and Item Scores for 31 Studies of Undergraduate Medical Education in Substance Abuse^*

			Possible score		Actual mean (SD) scores
Domain	MERSQI item	Studies, no. (%)	Item	Maximum domain	Item	Domain
Study design	1. Study design			3	1.64 (0.52)	1.64 (0.52)
	Single group cross-sectional or single group posttest only	8 (25.8)	1
	Single group pretest and posttest	11 (35.5)	1.5
	Nonrandomized, 2 (or more) groups	10 (32.3)	2
	Randomized controlled trial	2 (6.5)	3
Sampling	2. No. of institutions studied			3	0.56 (0.21)	1.66 (0.58)
	1	26 (83.9)	0.5
	2	2 (6.5)	1
	>2	3 (9.7)	1.5
	3. Response rate, % (highest reported)				1.10 (0.49)
	Not applicable	0
	< 50 or not reported	10 (32.3)	0.5
	50–74	1 (3.2)	1
	≥ 75	20 (64.5)	1.5
Type of data	4. Type of data			3	2.29 (0.97)	2.29 (0.97)
	Assessment by study participant (e.g., self-reported data)	13 (42.0)	1
	Objective measurement (e.g., OSCE, written exam)	18 (58.1)	3
Validity of evaluation instrument†	5. Internal structure			3	0.26 (0.44)	0.97 (0.87)
	Not applicable	0
	Not reported	22 (71.0)	0
	Reported	9 (29.0)	1
	6. Content (purposeful process to instrument development)				0.65 (0.49)
	Not applicable	0
	Not reported	23 (74.2)	0
	Reported	8 (25.8)	1
	7. Relationships to other variables (criterion, predictive or discriminate validity)				0.06 (0.25)
	Not applicable	0
	Not reported	25 (80.6)	0
	Reported	6 (19.4)	1
Data analysis	8. Appropriateness of analysis			3	0.84 (0.37)	2.58 (0.72)
	Data analysis inappropriate for study design or type of data	4 (12.9)	0
	Data analysis appropriate for study design and type of data	27 (87.1)	1
	9. Complexity of analysis				1.74 (0.44)
	Descriptive analysis only (means and variances)	11 (35.5)	1
	Beyond descriptive analysis (any comparisons)	20 (64.5)	2
Outcomes	10. Outcome			3	1.27 (0.25)	1.27 (0.25)
	Satisfaction (“happiness”) ‡ Opinions and general facts ‡	3 (9.7)	1
	Attitudes/perceptions ‡	7 (22.6)	1
	Knowledge, skills (e.g. OSCEs, SPs as outcome measure)	21 (67.7)	1.5
	Behaviors (e.g., physician actual practice)	0	2
	Patient/health care outcome	0	3
Total score				18		10.42 (2.59)

^*All comments in parentheses are clarifications. OSCE indicates objective structured clinical examination; SD, standard deviation; SP, standardized patient.

^†Each of 5, 6, and 7 can apply to different instruments.

^‡Items were separated out for data abstraction but analyzed as a single item.

Before using the MERSQI in this study, two authors (D.K., A.K.) reviewed its correct application with Darcy A. Reed, MD, MPH, the author of the measure, on 5 randomly selected studies from our set of 31 in April 2009. Then, three other raters (A.T., E.G., J.L.), working independently, practiced using the MERSQI on three studies from the tobacco curriculum literature and reviewed their scoring with A.K. and D.K. to clarify the meaning of each item in the scoring schema and improve consistency among all five raters.

Each of the 31 studies in our set was rated by four authors using the MERSQI. First, D.K. and A.K. scored the studies independently. They discussed their ratings in depth and documented their decisions about applying MERSQI coding in a written MERSQI scoring manual they provided to the other raters as a training reference. Because of this highly integrated process, they decided to report their final MERSQI scores as one average score. Next, each article was scored by two of the other three raters (J.L., A.T., E.G.), using the MERSQI scoring manual. The raters’ scores (three effective raters: 1. A.K. and D.K., scores averaged; 2. E.G.; 3. either A.T or J.L.) were used to calculate intra-class correlation coefficients (ICC) to determine inter-rater reliability. Each study was scored at the highest possible level for each of the 10 MERSQI items, as per the published protocol.¹⁴ The ICC for total MERSQI score was 0.82 (95% confidence interval [CI]: 0.70–0.90).

Determining citation rates, impact factors, and correlations with MERSQI score

The first author (D.K.) used ISI Web of Knowledge to determine the three-year citation rate for each study: she looked up the article and then manually counted the number of citations for up to three years after the publication date. She used Journal Citation Reports to identify the impact factor of the journal in which the study was published.¹⁷ Spearman’s ranked correlation coefficient (ρ) was calculated to assess the relationship between ranked variables using SPSS version 18 (SPSS Inc., Chicago, Illinois).

Determining contact hours and content

The first author (D.K.) conducted a detailed review of the 31 studies, comparing curriculum descriptions and instructional techniques to make best estimates of the contact hours per student (defined as number of hours devoted to lectures, Web-based activities, seminars, and clinical activities), as well as the interventions and content included in the curriculum.

Results

MERSQI scores

Table 1 summarizes the MERSQI scores for the 31 studies by item and domain. The mean total MERSQI score was 10.42 (range: 6.33–14.83) Appendix 1 presents a detailed review of the 31 studies, ordered from highest to lowest MERSQI score.

Appendix 1.

Description of 31 Studies of Undergraduate Medical Education Interventions for Substance Abuse Disorders (SAD) Intervention Training, From Highest to Lowest Medical Education Research Study Quality Instrument (MERSQI) Score^*

Source	Participants	Interventions	Evaluations	Results	Competencies addressed†	MERSQI Score
Kahan et al, 200318	Third- and fourth-year medical students from the five medical schools in Ontario, Canada	In a randomized controlled trial, 76 students received a 3-hour workshop either on either problem drinking and alcohol dependence or on depression (control condition). The alcohol workshop consisted of a 1-hour presentation, role plays, and a demonstration of a clinical interview. It covered assessment and management of problem drinking and alcohol dependence.	Postworkshop OSCE (8 stations); all students completed a baseline questionnaire on their beliefs and self-reported clinical behavior as well as a postworkshop questionnaire on beliefs. Four months later, they were mailed a follow-up questionnaire on their knowledge, beliefs, and self-reported behavior.	The alcohol group received significantly higher assessment and management checklist scores and global rating scores and performed better on almost all aspects of clinical management of both problem drinking and alcohol dependence than did the depression group. At 4 months postintervention, the alcohol group also showed a significant increase in beliefs about self-efficacy in managing alcohol problems (from baseline) and had greater knowledge of reduced drinking strategies than did the depression group, but the two groups did not differ on other measures.	N/A	14.83
Walsh et al, 200128	Fifth-year medical students at the University of Newcastle and University of Sydney, Australia	In two controlled trials, 154 students received either a didactic alcohol education program or didactics plus skills-based training.	Pre- and postintervention videotaped interviews with SPs were assessed by raters using a written rating scale (the pretest and post-test)	The educational approach used had no effect on posttest scores. Alcohol-related interactional skills scores improved for both groups after training at both universities. (Posttest scores were higher than pretest scores).	BI	14.00
Roche et al, 199729	Fifth year medical students from a “traditionally oriented medical school” in Australia	In a randomized controlled trial involving 115 students, the experimental group received a 3-hour interactive program involving a short lecture, clinical practice, and small group feedback on clinical performance. The control group received 3 hours of a traditional didactic teaching program on the principles and practice of brief and early intervention.	10-minute videotaped encounter with an SP before and after teaching, evaluating students’ general interactional skills and how well they addressed alcohol-related issues	The educational approach had no effect on pre–post differences. Performance on alcohol-related issues and interactional skills were significantly improved equally in each group.	S, BI	13.83
Taverner et al, 200030	Senior medical students at University of Adelaide, Australia	Three instructional strategies were used to teach clinical skills in assessing and managing drug-seeking patients: didactic small group tutorial (n = 35), video-based tutorial (n = 136), or computer-aided instruction package using digitized video (n = 43).	Student feedback, performance on a case-based written examination, and a structured evaluation of interviews with SPs	No difference was seen in written examination and SP outcomes among the three groups. Student feedback was most positive for the video-based tutorial. The computer-aided instruction was most cost-effective over a 6-year period.	S, RT	13.17
Walsh et al, 199931	Final-year medical students at the University of Newcastle, Australia	In a controlled trial, two student blocks received a manual, a lecture, and a demonstration about the principles and practice of a brief alcohol intervention. Experimental students (n = 29) made a 20-minute videotape and participated in a 1.5-hour small group feedback session in addition. Control students (n = 26) did not.	Pre and postintervention questionnaires and videotaped interviews with SPs	There was no significant difference in the effectiveness of the two methods. Levels of alcohol-related knowledge, attitudes, and interactional skills were significantly improved after teaching in both groups.	BI	13.17
O’Neil et al, 198042	Third-year medical students rotating through psychiatry at the Medical University of South Carolina	One group (n = 66) received a programmed self-instruction text, called the Alcohol Learning Program (ALP). The control group (n = 60) received the less extensive alcohol teaching usually provided (also a programmed self-instruction text).	Pre- and posttest on alcoholism-related knowledge and attitude	Students receiving the ALP had a greater pre–post improvement in knowledge compared with controls. No attitude change was seen.	RT	12.83
Gopalan et al, 199224	First-, second-, and fourth-year medical students at the Johns Hopkins University School of Medicine.	The Model Alcohol Curriculum (MAC) emphasized learning state-of-the-art approaches to the diagnosis and management of alcoholism and other drug problems. Content and teaching methods were directed toward building positive attitudes toward patients and toward treatment for these disorders. The MAC included experiential aspects, including interacting with recovering alcoholics and drug addicts and attending self-help groups (e.g., AA).	Surveys recorded students’ knowledge, attitudes, beliefs in role responsibility, and confidence in clinical skills related to the diagnosis and treatment of SAD.	Significant improvements occurred in the students’ attitudes, beliefs in role responsibility, and confidence in skills during their preclinical years. These positive changes were stronger and better sustained during the clinical years in those students who participated in special programs or elective courses focusing on SA; they were not always sustained in other students.	S, BI, RT	12.50
Lee et al, 200832	First-year medical students at New York University School of Medicine	This controlled trial, compared the effectiveness of an interactive Web module (n = 82) with that of a traditional lecture (n = 81) to improve medical students’ competence in screening and interventions for hazardous drinking.	SP alcohol case, Pre- and posttest of knowledge, student OSCE evaluations	The Web group had higher mean scores on scales of individual components of brief intervention (assessment and decisional balance) and a brief intervention composite score, and they self-reported as better prepared for the SP case. Participation in instruction was higher in the Web module group.	S, BI	12.17
Matthews et al, 200225	Third-year medical students at the University of Massachusetts Medical School	This study, conducted over four years, involved student participation in a one- or two-day interclerkship about SA.	Pre- and post-interclerkship test about attitudes and knowledge of SAD (attitude questions modified from the SAAS), a final course evaluation, and AAMC graduate questionnaire data. In addition, during one year, each student’s clinical skills in SA assessment and intervention were evaluated at the end of the 6-week psychiatry clerkship using OSCEs with two SPs, one with and one without active SA issues.	Students’ attitudes toward and knowledge about SA disorders and their confidence about SA assessment and intervention all showed significant positive changes immediately after the interclerkship. The OSCE performance data demonstrated a significant sustained improvement in clinical skills in SA assessment and intervention, as measured up to 6 months following the interclerkship.	S, BI, RT	11.83
Chappel et al, 197743	Second-year medical students at the University of Nevada School of Medical Sciences	Forty-eight students were given a 28-hour course on SA, including history, epidemiology, and a conceptual overview. Topics/activities included the impact of advertising, management of overdose and withdrawal, medical complications, psychological aspects, cultural aspects, treatment methods, field trips to treatment programs, assessment of treatment programs, panel of basic and behavioral scientists, detection technology, legal aspects, role of the physician, physician susceptibility, and prevention.	Pre- and posttest knowledge and attitude measure	Significant positive changes in attitudes: Students reported feeling less upset when they encountered alcoholics, “hard” drug abusers, “soft” drug abusers, compulsive smokers, and obese overeaters, as well as having a more positive view of the physician’s role in the treatment of substance dependence. Students’ personal experience with alcohol and other drugs was shown to have an influence on attitude changes.	RT	11.33
Brown et al, 199033	Third-year medical students, rotating through family medicine rotation at Jefferson Medical College of Thomas Jefferson University	Four groups of self-selected students participated. The experimental group (n = 33) received three computer assisted instruction (CAI) modules, active discussion, role play, opportunities for applying new knowledge and clinical skills, and modeling of clinical interest by a family doctor. The immersion group (n = 12) received one week immersion experience on a SA inpatient unit. The limited formal teaching group (n=35) received 1–4 hours of lecture and the no formal teaching group (n=28) received no intervention.	End of rotation questionnaires assessing knowledge, satisfaction with SA teaching, and motivation for continued learning	The CAI course resulted in higher levels of knowledge on early diagnosis but similar attitudes and satisfaction to those of the other three groups. The CAI and immersion courses produced more favorable outcomes than lecture-based teaching and no formal teaching. Compared with no formal teaching, lecture did not produce a measurable effect.	S, RT	11.17
Bland et al, 200134	Second-year medical students during 5-week human reproduction block at the University of Ottawa Faculty of Medicine, Ontario, Canada.	All 84 students received lectures (1 hour) on SAD during pregnancy, a 2-hour problem-based small-group tutorial concerning a young pregnant woman with alcohol dependence, and a 2-hour interaction with an SP.	Questionnaire (51 items) given before and after training regarding attitudes toward and counseling for SAD during pregnancy (used shortened, modified version of SAAS in second section of survey)	Students showed statistically significant improvement in their level of comfort in dealing with women with SAD in pregnancy.	N/A	11.00
Stillman et al, 199044	Second year medical students at four medical schools in New England	Students (n = 794) were given a selection of articles about alcoholism and SA prior to SP encounters. Then, each student participated in a 3-hour mandatory session and was randomly assigned to 3 of 34 30-minute SP scenarios.	Feedback after each SP encounter, including a case-specific checklist and rating of the student’s performance on the Arizona Clinical Interview Rating (ACIR) scale	There was a statistically significant difference between checklist scores and ACIR scale scores across encounters. Student performance improved from the first to third SP encounter.	S, BI, RT	11.00
Christison et al, 200335	Third-year medical students rotating through psychiatry clerkship at the Loma Linda University School of Medicine	As part of the 6-week psychiatry clerkship, 153 students were assigned either to a 1-week addiction treatment experience at either an intensive outpatient VA addiction treatment program (70%) or to an addiction treatment program at a private university-based psychiatric hospital (30%).	Pre and postclerkship Medical Condition Regard Scale (MCRS) assessing attitudes towards patients with alcoholism, major depression, and emphysema; postclerkship psychiatric subject examination score before (historical controls) and after introduction of the one-week addiction treatment experience to the clerkship.	Students’ mean MCRS scores increased significantly for patients with alcoholism and major depression but did not change for patients with emphysema. Psychiatric subject examination scores before and after the curriculum change were not significantly different.	RT	10.83
Ramirez-Cacho et al, 200745	Third-year medical students rotating through the obstetrics–gynecology clerkship at the University of New Mexico School of Medicine	During the first 4 weeks of the clerkship, the experimental group (n = 52) was assigned to a half-day prenatal clinic designed specifically for women with SAD. The control group (n = 52) was assigned to the special prenatal clinic during the second 4 weeks.	Survey measuring comfort level and attitude at beginning and midway point of clerkship (adapted from an instrument designed and described by Bland et al.)34	At baseline, experimental and control groups had similar comfort levels and attitudes. Students in experimental group became more comfortable in talking with patients about their substance use and less judgmental in treating these patients. Compared with before the clerkship, the control group became less comfortable in talking with these patients about their habits, less aware about the prevalence of SA during pregnancy, and less aware of the efficacy of counselors.	S	10.83
Chappel and Veach, 198736	Second-year medical students at the University of Nevada School of Medical Sciences	Same as Chappel et al, 1977,43 but with 5 years of students (n = 48 for each year).	Pre and post SAAS (50 items). Five stable attitude factors—“Permissiveness” factor, “Treatment Intervention” factor, “Nonstereotypes” factor, “Treatment optimism” factor and “Nonmoralism” factor—were identified during the development of the survey.	Positive attitude change could be achieved during a SA course. When the course was scheduled in competition with demanding basic science courses, however, the students’ attitudes did not become as positive as or became more negative than when the course was part of a less demanding schedule. The study found that “Time pressure inhibits attitude change.”	RT	10.67
Weinberg and Morse, 197546	First-year medical students at the Mayo Medical School	Forty students received 4 hours of lecture and case material, took part in small-group discussions in which recovering alcoholics related their histories, and served as group leaders. Later during the year, students interviewed patients who were being treated for alcoholism and heard lectures on the pharmacology of alcohol and other psychotropic drugs.	“Understanding Alcoholism” pre- and posttest, consisting of 42 true-false items	On the pretest, students’ mean number correct was 31.0/42; the immediate posttest mean was 39.9/42, and the one-year follow-up mean score was 38.1/42.	N/A	10.33
Silins et al, 200747	First- and fourth-year medical students at an Australian university	Students took part in a structured education and clinical experience about SA. First-year students (n = 223) completed a 3-week addiction medicine learning module. Each week included 3 PBL tutorials, 6 lectures, 6 web-based learning resources, 3 interactive large-group teaching sessions, and 2 clinical skills sessions related to the “case of the week” (alcohol, benzo, or opiate). Guests with past alcohol dependence or current opioid dependence acted as speakers or interviewees. For fourth-year students (n = 222), there was a 9-week psychological and addiction medicine block. The block included 6 weeks of structured teaching sessions, with 2 PBL tutorials and 3 related seminars each week. Students also attended 3 clinical placements of 3 weeks’ duration. Students were required to interview and present the history of one patient with alcohol use disorder and one with another SAD.	Pre- and postsurvey consisting of 39 items on attitudes toward users of alcohol, tobacco and illicit drugs; confidence in managing patients with SAD; motivation toward intervening with substance users; and perception of the legitimacy of the medical practitioner’s role in dealing with patients with drug and alcohol problems. Twenty-one questions were derived from the Alcohol and Alcohol Problems Perception Questionnaire.	Men, older students and those with prior clinical experience tended to have more negative attitudes at baseline. Attitudes improved significantly after exposure to interactive learning modules. The level of dislike of problem drinkers significantly decreased after teaching. After fourth-year education, students reported a greater sense of responsibility toward providing intervention and less anticipation of discomfort working with these patients. By the end of drug and alcohol education, less than half (42%) of students reported they could imagine working with substance misusers as a career.	S	10.00
Hunter et al, 198937	Third-year medical students rotating through psychiatry clerkship at Texas A&M University College of Medicine	During a two-week clinical rotation at the Alcohol and Drug Dependence Treatment Program (ADDTP), medical student functioned as subinterns, followed patients, attended AA meetings, watched films on chemical dependency, and attended didactic sessions with attending physicians.	NBME SA exam and student feedback via questionnaire	Students who rotated on the ADDTP performed as well as the students who had not rotated at the ADDTP in previous years. Students scored significantly higher on the NBME SA exam than the national mean. Pre-/posttest results indicated an increase in knowledge by at least one SD. Students reported ADDTP as “educationally valuable.”	N/A	9.50
Klein et al, 199148	Third-year medical students rotating through psychiatry clerkship at the University of Rochester Medical Center	Recovering alcoholics served as patient instructors (PIs) to teach 62 students interviewing and counseling skills.	PI rating of students’ performance (using University of Rochester Risk Factor Interview Scale [URRFIS]) but questionable as to whether that was part of intervention. Intervention included a questionnaire and student evaluation of exercise.	Students’ overall performance was rated by PIs as a 5 (highest possible rating) or a 4 in 58% of the exercises. In terms of specific skills on the URRFFIS, students did best in the “relationship” and “general interview technique” and less well in “positive focus” and “instigating behavior change.” Students gave positive evaluations of the exercise.	S, BI, RT	9.50
Siegal et al, 198626; Siegal and Rudisill. 198349	First- and second-year medical students at Wright State University School of Medicine	Students acted as participant-observers in the Weekend Intervention Program (WIP) through which they learned about alcoholism, diagnosis and assessment, and opportunities for treatment and referral. Students observed individual and group counseling sessions, attended AA meetings, and interacted with clients/patients. (WIP had more than 10,500 clients; the number of medical students was not provided.)	Multiple-choice tests assessing knowledge, history-taking, counseling, and attitudes (65 items) of students who had participated in WIP compared with those who had not	Significantly higher overall scores were found on knowledge, skills, and attitudes among those who had participated in WIP than those who had not. No difference was found between the group tested immediately following completion of the WIP and the group tested 12 months later.	S, BI, RT	9.50
Fazzio et al, 200350	Third-year medical students rotating through psychiatry clerkship at New York University School of Medicine	The 36 students were rotating through a required 5-week psychiatry rotation on 4 general psychiatry units at a major public urban teaching hospital. Most patients were dually diagnosed with SA and mental illness. The study describes a 1-hour lecture on spirituality and AA, but that was in addition to other lectures that are part of the clerkship.	Pre- and postclerkship evaluation of attitudes toward AA (spiritually based treatments) via 17-item questionnaire	At the beginning of the clerkship, students rated a spiritually oriented approach as important in addiction treatment as a biological approach, whereas at the end of the clerkship, they rated the biological approach as more important.	N/A	8.83
Ungerleider et al, 199238	First- and second-year medical students at the UCLA School of Medicine	More than 180 students participated in 16 hours of elective, small-group, interactive teaching seminars (IATs), which covered 12-step programs and prevention/education, and included law enforcement and treatment site visits. Students taught one another other. Seminars used audiovisuals, role-plays, and SPs in a report/debate format.	Pre and post SAAS administration (50 items)	IATs changed students’ attitudes on about one-third of SAAS items and decreased students’ uncertainty about SA issues on several other items. However, some items that dealt directly with key issues did not show a change in mean score or response distribution following the IATs.	S, BI, RT	8.83
Toraker and Rydberg, 198951	Fifth-year medical students at The Karolinska Institute in Stockholm, Sweden	Students (n=140) took a 2-week course (80 hours) in alcoholism and drug dependence, including 1 week of lecture and 1 week of clinical training. Lecture topics included basic science, clinical science, prevention, early diagnosis, and legal and philosophical aspects of SA. Clinical training was in an inpatient ward with an outpatient clinic, and included seminars and community site visits.	Questionnaire, both after the week of lectures and after the clinical training, asked students to rate their attitudes toward alcoholism and drug dependence problems before and after the course (retrospective pre–post)	Students liked the course. Students’ interest in alcohol and drug dependence problems increased significantly after the course compared with their interest before it. The students were also more inclined to work in this field after the course.	S, RT	8.17
Coggan et al, 198152	Third-year medical students rotating through the family medicine clerkship at the University of Washington School of Medicine	Students (n = 19) participated in an alcoholism module, which included a 3-hour seminar, 2 site visits to a treatment center, and a visit to an AA meeting.	Students’ ratings of the alcoholism module (5-point scale); course final examination, including SP encounter	The alcoholism module received a rating of 4.3, the seminar 4.5, and the treatment center visits 4.1. Students did well in the course final examination; when a simulated alcoholic patient was included in the videotaped part of the examination, all students made the diagnosis correctly.	S, BI, RT	8.00
Oldham and Sipe 199053	Medical students enrolled in the Medical Student Professional in Residence (PIR) Summer School at the Betty Ford Center, Rancho Mirage, California	Students (n = 24) participated in the “Experiential Training Program” within a chemical dependency unit: 16 participated as mock chemically dependent inpatients and 8 as mock family member outpatients.	Pre- and postprogram administration of SAAS	There were significant score increases after the PIR training on the two discriminating factors “treatment intervention” and “treatment optimism.”	RT	7.83
Bishop et al, 199027	Second-year medical students at the Medical College of Pennsylvania	“Alcoholism Day” included small-group discussions with faculty and a person in recovery, demo interview with SP (or videotape), lecture on pharmacology and pathophysiology of alcoholism, public health aspects of alcoholism (case discussion), treatment modalities lecture and case discussion. The study reported the number of students from a “representative year” as 79.	Attendance, student evaluation of “Alcoholism Day” as a whole (excellent, good, fair), student performance on written examinations at the end of the second year	Student response to both the presentation and content of the day was positive. Attendance averaged 75% of the class, above average for lectures in the second year. Course evaluations from a representative year showed 80% of students who completed evaluations rated the day as excellent or good, 17% as fair, and 2% as poor. Fourth-year students in a required ambulatory care clerkship demonstrated knowledge of the CAGE on a written examination. Student performance on written examination at the end of the second year showed excellent retention of material presented (80% answered questions on alcoholism correctly). However, in most discriminating questions relating to screening and early diagnosis, students chose wrong answers reflecting common misbeliefs about alcoholism 40% of the time.	S, RT	7.17
Brown and Byrne, 199054	Third-year medical students rotating through the family medicine clerkship at Jefferson Medical College of Thomas Jefferson University	A group of 33 students received 3 CAI modules on early diagnosis, attitudes and screening questionnaires, as well as 2 conferences. CAI 1 included content about early diagnosis of substance abuse, CAI 2 included the SAAS, and CAI 3 had CAGE/MAST. The 3 modules and 2 conferences were followed by a clinical experience in which the students interviewed a patient at risk for SA in ambulatory or ER settings, and performed a physical exam, lab evaluation, and a chart review. They also submitted a write-up.	Evaluation forms on CAI modules and conferences immediately upon completion; on each form, students rated attributes of each experience and provided comments on positive features and suggestions for improvement. On the last day of the clerkship, students completed a questionnaire on SA teaching.	All students gave high ratings to all of the learning activities and the entire curriculum. Almost all were satisfied with the emphasis on SA in their clerkship and expressed motivation to learn more.	S, RT	7.00
Harris and Westermeyer, 197855	Second-year medical students at the University of Minnesota Medical School	Students (n = 240) participated in supervised clinical tutorials at chemical dependency treatment centers.	Evaluation forms with ratings and subjective comments	Students had both positive and negative feedback about the tutorials. Positive ratings included the educational value of the experiences at the centers. Negative feedback centered on unguided exposure at the centers and issues with the chemical dependency counselors.	RT	7.00
Welsh, 200356	Second-year medical students at University of Maryland School of Medicine	Students (n = 122) attended a 100-minute lecture on intoxication and withdrawal syndromes of 11 substances, which included a videotape incorporating clips from various commercially available films, several television news shows, and a training film displaying intoxication and withdrawal syndromes.	Eight-question, anonymous written questionnaire	More than 90% of the 89 respondents believed that the clips helped them recognize these syndromes and appreciate their potential severity. All students believed that the movie clips would help them remember the syndromes, with more than 90% reporting that it would help very much.	N/A	6.50
Confusione et al, 198219	Third-year medical students rotating through family medicine clerkship at SUNY Stony Brook	Students (n = 61) spent one day at local methadone maintenance clinic. They attended a series of seminars and interviewed two patients.	Questionnaire at end of program, evaluating students on their knowledge, attitudes, and level of participation in the drug abuse treatment program	The students on the whole showed an increase in knowledge and an improved attitude toward drug abusers’ health care needs.	S, BI, RT	6.33

^*CAGE is a common acronym for four questions that are used as a screening test for alcoholism. CAI indicates computer-assisted instruction; MAST, Michigan Alcoholism Screening Test; OSCE, objective structured clinical examination; RCT randomized control trial; SP, standardized or simulated patient; SA, substance abuse; SAAS, Substance Abuse Attitude Survey (developed by Chappel and Veach)³⁹; SAD, substance abuse disorders.

^†S= Screening, BI= Brief intervention, RT=Referral for treatment, N/A= not enough information to assess objectives.

Citation rates and impact factors

The mean three-year citation rate for the 26 studies indexed in the ISI Web of Knowledge was 1.46 (range: 0–5). The mean journal impact factor for the 17 articles indexed in the Web of Knowledge was 2.83 (range: 0.825–9.127).

Study participants, contact hours, and focus of interventions

The participants in the 31 studies detailed in Appendix 1 included first-year to fifth-year medical students in the United States (23 or 74.1% of the studies), Australia (5 or 16.1%), Canada (2 or 6.5%), and Sweden (1 or 3.2%). Eleven (35.5%) of the studies involved preclinical medical students (years 1 and 2), 17 (54.8%) involved clinical medical students (years 3, 4, and 5), 2 (6.5%) involved both preclinical and clinical medical students, and 1 (3.2%) did not report these data. Nine (29.0%) of the studies focused on alcohol abuse curricula, 4 (12.9%) studied illicit drug use curricula (e.g., marijuana, cocaine, heroin), and 18 (58.1%) included both. SA during pregnancy was the focus of 2 (6.5%) of the curricula studied. Among the 26 studies reporting contact hours, the mean was 29.25 hours (range: 0.83–200 hours). Twenty-five (80.6%) of the studies included specific information on the content of the curriculum (see “Competencies addressed” in Appendix 1).

Correlations with MERSQI score

MERSQI score was correlated with more recent year of publication (ρ = 0.35, P = .05) and three-year citation rate (ρ = 0.49, P = .01), suggesting improvement of the literature over time. It was not correlated with journal impact factor. In addition, there was no statistically significant relationship between the MERSQI score and the number of contact hours in the curriculum or whether the study contained specific information on the content of the curriculum.

Studies with the highest and lowest MERSQI scores

The study with the highest MERSQI score¹⁸ demonstrated that third- and fourth-year medical students who participated in a three-hour workshop on problem drinking and alcohol dependence demonstrated better skills at assessing and managing a patient with problem alcohol use than did students in the control group. Four months later, they maintained more positive attitudes, higher self-efficacy, and greater knowledge concerning the issue than did the control-group students.¹⁸ The study earned a high MERSQI score because it was a multiple-institution, randomized controlled trial that assessed skills-performance outcomes and had a high response rate. The study with the lowest MERSQI score¹⁹ reported, without comparison to a control group, an increase in third-year medical students’ knowledge and an improvement in their attitudes toward drug abusers following one day at a local methadone maintenance clinic, where students attended seminars and interviewed patients.

Study interventions and outcomes

While all 31 studies claimed to show improvement in educational outcomes, 8 (25.8%) only assessed students after they completed the curriculum, 11 (35.5%) showed improvement of outcomes from baseline to postintervention, and 4 (13%) found different types of instructional strategies to be equally effective. The majority of studies conducted outcome assessment immediately following students’ participation in the curriculum or within an unspecified timeframe during a clinical clerkship. A variety of assessment tools were used. Detailed information on each study is provided in Appendix 1.

Discussion

The literature shows that teaching medical students about alcohol and drug abuse can improve their knowledge, skills, and self-efficacy in the short term (i.e., four months). Although medical educators around the world are developing ways to teach this material, there is little evidence available to support one teaching method over another and there is no evidence regarding long-term retention of knowledge or skills. There has been a call for longitudinal follow-up studies to determine whether long-term practice is affected, but there is little momentum, and few resources are available for such costly endeavors.^11,20

We found that the medical education literature on teaching students about SAD, like other medical education literature, has improved over time as suggested by the positive correlation between higher MERSQI score and year of publication. This trend may be attributed to the increasing attention being paid to building a rigorous evidence base for medical education practice and policy and the development of medical education research as a discipline.^21–23 The correlation of the quality of the literature (MERSQI score) with the three-year citation rate, not the journal impact factor, may reflect the emerging nature of the work.

In this study, we found the MERSQI to be a useful tool to measure the quality of medical education research. By ensuring that raters were carefully trained, we obtained good inter-rater reliability and produced scores similar in range to those reported by the instrument developers, suggesting that the MERSQI is applicable across various types of medical education literature.¹⁴ Of note, however, is a limitation: the MERSQI does not account for whether the study reports immediate postcurriculum outcomes evaluating short-term recall versus outcomes measured at some temporal distance from the educational intervention that would indicate more enduring retention of learning. Five of the 31 studies measured longer-term outcomes.^17,24–27

Guidance for instructional design

While we had hoped to identify compelling evidence supporting certain instructional approaches over others, the inconsistent reporting in the published literature prevented us from determining the best methods for ensuring that medical students will be able to effectively identify and intervene with patients struggling with SAD. Although further work is needed, a few of our specific findings are consistent with general educational principles and theory, and therefore can provide guidance for curriculum development.

Of the six studies that compared types of instructional strategies,^28–33 four showed the different interventions to be equally effective in improving short-term outcomes.^28–31 In general, the literature showed the more effective modalities to be active and experiential, such as structured role-play with standardized patients who function as instructors. This finding is consistent with the related medical-education literature about tobacco, which suggests that the most effective curricula emphasize active practice of skills over passive instructional methods.¹⁶

In addition, despite lack of comparison groups in some studies, we found evidence that:

• Teaching interventions for medical students rotating on an obstetrics–gynecology clerkship can improve their comfort levels and attitudes toward pregnant women with SAD, a particularly important group to identify for treatment.³⁴

• Intensely immersive treatment program experiences (e.g., one week spent participating in a range of activities at a drug and alcohol treatment program) seem specifically to improve regard for patients with alcoholism and SAD, which is an important component of providing effective care to this population.³⁵

• Computer-assisted courses, which are relevant and practical, can efficiently augment third-year medical students’ knowledge and motivation for further learning about SAD. This instructional strategy is easy to scale up to large numbers of learners, which increases its educational impact.³³

Tools for evaluation of curricular effectiveness

Tools used to evaluate the SA interventions ranged from student satisfaction, which was uniformly high—and uninformative—to behavioral measures, such as those used in SP encounters, which were more likely to discriminate among students. Six studies used standardized outcome measures:^25,34–38 the Substance Abuse Attitude Survey (SAAS),³⁹ the Medical Condition Regard Scale (MCRS),⁴⁰ and/or National Board of Medical Examiners (NBME) scores. More work needs to be done to establish the reliability, validity, and feasibility of these instruments and to develop high-quality measures of clinical skills critical to the identification and treatment of SAD.

Recommendations for future research

Although MERSQI scores are a useful way to quantify the level of rigor and methodological characteristics of published studies, they do not describe the nature of study interventions. To move the field of SA medical education forward, both state-of-the-art curriculum and rigorous evaluation must be employed and disseminated so innovations that prove successful in one setting can be replicated at other institutions, with positive results. Based on this review, we offer the following recommendations for enhancing the evidence base in SA–focused medical education:

1. Researchers should conduct multi-institutional, controlled trials of educational strategies. While it is clear that interventions temporarily increase students’ knowledge, best educational practices will never be clear unless studies are conducted with appropriate comparison groups. Although such study designs are difficult and expensive, the information gained would be widely applicable.

2. Researchers should derive evaluation methods from the desired patient-level outcomes. Such outcomes include effective screening, skillful brief intervention, and appropriate and timely referral for treatment. The field would benefit from the creation and wide adoption of a set of standardized outcome measures of the critical educational domains of knowledge (e.g., NBME exams), attitudes and beliefs (e.g. SAAS, MCRS), and skills (e.g., standardized patient scenarios and checklists) that all physicians should demonstrate in this area.

3. Researchers should conduct long-term follow-up studies of at least one year. Such studies are critical to evaluate the retention of skills, but they require adequate funding for the infrastructure needed to conduct the long-term evaluation.

4. Researchers should ensure that they provide sufficient information to permit others to evaluate and reproduce their curriculum or intervention. This may be a challenge for authors who write one article about their work (few authors had more than one study published on a single project) because they must balance providing detailed, structured curriculum descriptions and describing evaluation methods and results. Further, those medical education researchers who have multiple publications about the same intervention would benefit from standards addressing how to identify their other work, similar to the standards established for other research areas.⁴¹ In addition, researchers would benefit from publication policies that impose structures for curriculum descriptions and encourage linked publications. Finally, authors could prevent loss of information by making detailed appendixes available to interested readers.

Conclusions

SAD is an important health issue and there is obvious interest in educational interventions as reflected in the volume of the publications, most of which do not provide enough evidence for effectiveness to be useful to educators building curriculum. The challenge for undergraduate medical educators is clear: Medical educators must define, design, implement, and evaluate curricula to ensure that all medical school graduates, across disciplines and specialties, have the basic skills they require to address SAD in their patients. Institutions and clinical disciplines need to collaborate on producing larger, richer, and more rigorous evaluations of these interventions to ensure that they contribute meaningfully to educational science and improve clinical outcomes.