Teaching About Substance Abuse with Objective Structured Clinical Exams

Abstract

BACKGROUND

Although residents commonly manage substance abuse disorders, optimal approaches to teaching these specialized interviewing and intervention skills are unknown.

OBJECTIVE

We developed a Substance Abuse Objective Structured Clinical Exam (OSCE) to teach addiction medicine competencies using immediate feedback. In this study we evaluated OSCE performance, examined associations between performance and self-assessed interest and competence in substance abuse, and assessed learning during the OSCE.

DESIGN

Five-station OSCE, including different substance abuse disorders and readiness to change stages, administered during postgraduate year-3 ambulatory rotations for 2 years.

PARTICIPANTS

One hundred and thirty-one internal and family medicine residents.

MEASUREMENTS

Faculty and standardized patients (SPs) assessed residents' general communication, assessment, management, and global skills using 4-point scales. Residents completed a pre-OSCE survey of experience, interest and competence in substance abuse, and a post-OSCE survey evaluating its educational value. Learning during the OSCE was also assessed by measuring performance improvement from the first to the final OSCE station.

RESULTS

Residents performed better (P<.001) in general communication (mean ± SD across stations = 3.12 ± 0.35) than assessment (2.65 ± 0.32) or management (2.58 ± 0.44), and overall ratings were lowest in the contemplative alcohol abuse station (2.50 ± 0.83). Performance was not associated with residents' self-assessed interest or competence. Perceived educational value of the OSCE was high, and feedback improved subsequent performance.

CONCLUSIONS

Although internal and family medicine residents require additional training in specialized substance abuse skills, immediate feedback provided during an OSCE helped teach needed skills for assessing and managing substance abuse disorders.

Substance abuse is a common and challenging problem in medical settings. National estimates suggest that 25% to 40% of hospital admissions and 10% to 20% of general outpatient visits are related to addiction.^1–3 The prevalence of unhealthy alcohol use is at least 7% to 20% among medical outpatients.⁴ Physician trainees in internal and family medicine often provide medical care to drug and alcohol users, but many studies have documented that physicians are reluctant to address substance abuse disorders and may have negative attitudes toward substance abusing patients.⁵ Ineffective screening and history taking may lead physicians to miss such disorders in up to two-thirds of patients.⁶ Even when substance abuse disorders are detected, physicians may not provide adequate interventions.^3,6

Substance abuse training for resident physicians is limited and often offered only electively.⁷ Although the U.S. Preventive Services Task Force has established the efficacy of screening (1996)⁸ and counseling (2004)⁹ for alcohol use disorders, consistent training is lacking in primary care residencies.¹⁰ Training has been recommended for generalist physicians in: knowledge about substance abuse symptoms, signs, sequelae, and treatment; and specific interviewing skills, including screening and brief interventions.¹¹ However, optimal methods to teach substance abuse competencies to resident physicians are currently unknown.

Residency programs are increasingly required to provide performance-based assessments to document clinical skills, and objective structured clinical exams (OSCEs) are the preferred or next best method for assessing many of these skills.¹² An OSCE is a timed, multistation examination in which learners perform tasks such as interviews, physical exams, and counseling with standardized patients (SPs) in realistic settings.^13,14 At each station learner performance is evaluated with specific checklists or global rating scales, completed by faculty proctors and/or SPs. Objective structured clinical exams enable the same clinical scenarios to be presented to many trainees¹⁵ and have become the gold standard for performance-based assessment.¹⁶ When immediate feedback on performance is provided, OSCEs can be formative in developing skills and enhancing knowledge.^17,18 Objective structured clinical exams are widely used in residency training disciplines, including internal medicine,^19,20 surgery,¹² pediatrics,²¹ rehabilitation medicine,²² psychiatry,^23,24 and family medicine.²⁵ They have also been used to teach and assess specialized skills such as delivering bad news,²⁶ addressing advanced care directives,²⁷ demonstrating cultural competence,²⁸ and ethical decision making.²⁹

Previous studies have examined the use of SPs within and outside of formal OSCEs to assess addiction medicine skills following teaching interventions. Objective structured clinical exams administered before and after educational programs have demonstrated improvements in medical students' substance abuse assessment and management skills.^30–32 Among residents, 2 studies used SPs to demonstrate the effect of intensive substance abuse training,^33,34 and 1 used SPs to describe the need for improved skills.⁵ While many OSCEs use a single substance abuse, smoking, or behavior change station, none have focused completely on substance abuse. In addition, though physicians' perceptions of their own interest and competence in managing substance abuse may influence their ability to conduct screenings, brief interventions, and referrals,^35,36 the association between these perceptions and performance measured by direct observation has not previously been studied.

Our main objective was to evaluate performance on a 5-station substance abuse OSCE designed to assess and teach competencies in addiction medicine to postgraduate year-3 (PGY-3) internal and family medicine residents. We also sought to compare residents' skills in managing alcohol and other substance abuse problems and to determine whether self-assessed interest and competence in substance abuse were associated with OSCE performance. Finally, we sought to evaluate the OSCE's educational value and to assess learning during the OSCE. This study was approved by Montefiore Medical Center's Institutional Review Board.

METHODS

Case Development

After reviewing substance abuse competencies recommended for primary care physicians,¹¹ experts in primary care and addiction medicine constructed a blueprint covering different substances of abuse (heroin, cocaine, alcohol), readiness to change stages (precontemplation to maintenance),^37,38 and representative ages, genders, and ethnicities (Table 1). Faculty then wrote case scenarios based on clinical experience, existing literature, and samples from a case resource book³⁹; and cases were piloted and refined using role play.

Table 1.

Objective Structured Clinical Exam Blueprint

Patient Profile	Stage of Change	Case Objectives
Station 1: Hector Sanche	Contemplative	Assess severity and sequelae of problem/abusive drinking
Age 35		Identify early alcohol dependence
Alcohol		Identify and respond to stage of change
Chronic daily user		Perform brief intervention
Station 2: Mary Pere	Maintenance	Take adequate drug use history
Age 42		Investigate drug treatment history
Opiates		Assess and treat pain in methadone patient
Methadone daily
Station 3: James Etinger	Precontemplative	Identify and assess adolescent/young adult crack cocaine use
Age 19		Identify panic attacks as cocaine-associated psychiatric co-morbidity
Crack cocaine		Perform brief intervention connecting symptoms with crack use
Binger
Station 4: Kate Boyle	Precontemplative	Screen and assess alcohol/other substance use
Age 62		Recognize dysthymia as psychiatric comorbidity
Alcohol		Identify and respond to stage of change
Chronic daily user
Station 5: Lloyd Evans	Precontemplative	Establish rapport with potentially hostile, active drug user
Age 23		Diagnose and explain opiate withdrawal
Heroin, cocaine		Take comprehensive drug use history
Chronic daily user		Counsel using risk reduction techniques

Faculty Development and Standardized Patient Training

Twelve faculty members in internal medicine, family medicine, and psychiatry served as evaluators. During 6 hours of faculty development, they role-played each case, underwent training to standardize ratings,⁴⁰ agreed on teaching points, and practiced giving case-specific feedback. To ensure interrater reliability, faculty compared ratings and agreed upon expected performance for specific items on each rating form. Professional actors served as SPs and were trained using detailed scripts and practice with faculty members to standardize performance and feedback.

Administration of the OSCE

The substance abuse OSCE was administered during ambulatory blocks in outpatient settings similar or identical to those in which the residents practice. Residents rotated through all 5 stations and had 2 minutes before each encounter to read background information and a list of specific tasks. During each encounter residents had 10 minutes to build rapport and assess and manage the case, while a faculty member observed and completed a rating form. At the end of each encounter, faculty provided 5 minutes of direct feedback and delivered teaching points. The OSCE concluded with a faculty-led group review of each case. For each resident the OSCE required 2.5 hours, but simultaneous introductory and review discussions allowed 10 residents to complete the OSCE in 5 hours using 2 rotations.

Performance Assessment

Rating Forms

Faculty completed a 17-item form including 15 specific items in 3 domains (6 general communication, 6 assessment, and 3 management items), and 2 global items (general organization and overall performance) (Table 2). Items evaluating general communication and the 2 global skills were uniform across stations, while items evaluating assessment and management skills were unique to each station. Each item was rated on a 4-point scale, ranging from 1 = needs much improvement to 4 = done excellently. Residents assessed their own overall station performance using the same 4-point scale. Standardized patients provided a global satisfaction rating (1 = would not come back, 2 = not sure if I would continue with doctor, 3 = would continue, 4 = one of the best doctors, would refer friends).

Table 2.

Faculty Rating Form Content

Items Used in all Stations	Items Unique to Each Station
General communication skills	Assessment of problem
Allows patient to express self	Current patterns of use
Communicates in non-judgmental fashion	Physical sequelae/dependence
Exhibits empathic/supportive attitude	Psychosocial sequelae
Probes for resistance/denial	High risk behaviors
Uses language appropriate to patient's level of understanding	Current/past cessation attempts
Strengthens treatment alliance	Stage of change
Global ratings	Management of problem
General organization	Motivational interviewing
Overall performance	Advice appropriate to stage of change
	Treatment plan appropriate to stage of change

Scale used for all items: 1 = needs much improvement, 2 = needs some improvement, 3 = done well, 4 = done excellently.

Scores

Faculty ratings resulted in both subscores for each of the 3 domains and station total scores. We used data from the specific items to calculate mean subscores for each domain in each station, and also to calculate an aggregate mean subscore for each domain across stations. In addition, we calculated mean station total scores using data from the 15 specific items for each station, and also mean station total scores for each station using 1 faculty global rating (overall performance). Finally, we calculated mean SP ratings and mean resident self-assessment ratings for each station and across all stations.

OSCE Summary Performance and Reliability

A summary OSCE score was generated for each resident by calculating a mean of the scores on the 15 specific items in all 5 stations (total of 75 items). The interstation reliability of the entire OSCE was evaluated using Cronbach's α, calculated from the 5 mean station total scores.

Prior Substance Abuse Experience

Residents' prior experience in substance abuse was assessed with 2 yes/no questions: previous substance abuse training, and attendance at 12-step meetings (e.g., Alcoholics Anonymous). Each question was analyzed separately to determine its association with the summary OSCE score. Because we were concerned that personal questions would inhibit survey responses, we did not collect information about residents' personal experience with substance abuse or its treatment.

Interest and Competence

Resident's interest and competence in substance abuse management were assessed using a 4-point scale, ranging from 1 = less skilled or interested to 4 = most skilled or interested. Summary scores for each scale were calculated, and reliability was determined using Cronbach's α. The association of each scale's summary score with the summary OSCE score was evaluated. Because we were interested in whether competence in alcohol abuse was different than in drug abuse, we asked 2 questions that were unique to alcohol abuse and then asked the same 2 questions about drug abuse. Scores from the 2 alcohol abuse competence questions were then compared with scores from the 2 drug abuse competence questions.

Learning Evaluation

We evaluated learning by analyzing the effect of feedback on performance in subsequent stations. We determined whether mean station total scores and mean subscores in the 3 domains improved from each resident's first station to their last station. In addition, we asked residents to evaluate the OSCE's overall educational value with 5 questions, each of which had 3 responses (definitely, maybe, and no).

Statistical Analysis

Differences between continuous variables were assessed using analysis of variance for 3 group comparisons and independent t tests for 2-group comparisons. Associations between continuous variables were assessed using Pearson's correlation coefficients. In analyses of the same residents' scores on 2 different questions or scales (e.g., alcohol and drug competence), paired t tests were used to compare differences. To evaluate whether station performance changed from the first to the last station, we calculated change scores and assessed their significance using 1 sample t tests.

RESULTS

Between August 2003 and November 2005, the substance abuse OSCE was administered to 131 PGY-3 residents, including 107 internal medicine and 24 family medicine residents. There were no differences between internal and family medicine residents in domain subscores, 15-item or global station total scores, or summary OSCE scores.

Resident Performance in Different Domains (Table 3)

Table 3.

Domain Subscores in all 5 Stations (N = 131)

Station	Mean(SD)
	General Communication	Assessment	Management
1. Alcohol/contemplative	2.99 (0.50)*	2.52 (0.53)	2.48 (0.67)
2. Opioid-methadone/maintenance	3.15 (0.48)*	2.72 (0.60)	2.72 (0.81)
3. Crack cocaine/precontemplative	3.24 (0.55)*	2.58 (0.63)	2.77 (0.70)
4. Alcohol/precontemplative	3.14 (0.63)*	2.75 (0.60)†	2.46 (0.76)
5. Heroin and cocaine/precontemplative	3.06 (0.47)*	2.63 (0.57)†	2.43 (0.70)
‡* Aggregate domain subscore	3.12 (0.35)*	2.65 (0.32)†	2.58 (0.44)

^*Communication scores higher than assessment or management in each station and in the aggregate (P<0.001 for all comparisons).

^†Assessment scores higher than management in station 4 (P<.001), station 5 (P<.01), and in the aggregate (P = 0.02).

^‡zAggregate mean score for each domain across all 5 stations.

In each of the 5 stations residents performed better (P<.001) in general communication than either assessment or management. Aggregate subscores were highest in general communication (mean ± SD = 3.12 ± 0.35), next highest in assessment (2.65 ± 0.32), and lowest in management (2.58 ± 0.44).

Resident Performance in Different Stations (Table 4)

Table 4.

Performance Across 5 Stations: Comparing Faculty, SP, and Resident Ratings (N = 131)

Station	Mean(SD)
1. Alcohol/Contemplative	2.71 (0.45)	2.50 (0.83)*	2.80 (0.87)†	2.54 (0.68)
2. Opioid-methadone/maintenance	2.91 (0.50)	2.84 (0.90)‡	2.92 (0.82)†	2.41 (0.79)
3. Crack cocaine/precontemplative	2.88 (0.50)	2.93 (0.74)*,‡	3.20 (0.84)†	2.45 (0.75)
4. Alcohol/pre-contemplative	2.85 (0.57)	2.71 (0.72)‡	2.63 (0.85)†	2.28 (0.69)
5. Heroin and cocaine/pre-contemplative	2.78 (0.45)	2.58 (0.85)*,‡	2.73 (0.87)†	2.21 (0.72)
Mean score across stations	2.83 (0.50)	2.71 (0.82)	2.85 (0.87)	2.38 (0.73)

^*Faculty global ratings lower than SP assessments (P<.001 for stations 1 and 3, P = .02 for station 5).

^†SP assessments higher than resident self-assessments (Po.01 for station 1, P<.001 for stations 2, 3, 4, and 5).

^‡Faculty global ratings higher than resident self-assessments (P<.001 for all comparisons).

Residents performed least well in the station portraying a contemplative alcohol abuser (mean global rating ± SD = 2.50 ± 0.83), and best in the station portraying a precontemplative cocaine user (mean global rating ± SD = 2.93 ± 0.74) (P<.001 for the difference between these 2 stations). The 2 types of faculty station total scores (15-item and global) were highly correlated (r = .87, P<.01), and both were correlated with SP scores (r = .70, P<.01; r = .68, P<.01, respectively). In 3 stations, faculty ratings were lower than SP ratings. Residents rated themselves lower than faculty in all stations except the contemplative alcohol abuser and lower than SPs in all stations.

OSCE Summary Performance and Reliability

The mean OSCE summary score was 2.83 (range 1.78 to 3.44, SD = 0.32), and the interstation reliability was moderate (Cronbach's α = 0.64).

Residents' Experience, Interest, and Competence

One hundred and thirty residents (99%) completed the pre-OSCE survey. Sixty-four (50%) had completed some training in substance abuse; and 83 (64%) had attended 12-step meetings. Residents who had prior substance abuse training performed better than those without training (mean summary score ± SD = 2.88 ± 0.27 vs 2.75 ± 0.36, P = .03), but there was no difference between those attending or not attending 12-step meetings (2.83 ± 0.31 vs 2.78 ± 0.34, P = .4).

Reliability of the interest and competence scales was high (Cronbach's α = 0.95 and 0.85, respectively). There were no correlations between summary OSCE scores and either interest or competence. Residents reported greater competence in both alcohol-specific skills (screening for and diagnosing alcohol abuse) than in the analogous drug-specific skills (mean score for 2 alcohol questions: 2.58 ± 0.68 and 2.59 ± 0.70 vs mean score for 2 drug questions: 2.30 ± 0.62 and 2.38 ± 0.73, P<.001 for the comparisons of both questions).

The OSCE Learning Experience

Among 128 residents (98%), we were able to assess changes in performance from the first to the final OSCE station (Table 5). Because some OSCE sessions had fewer than 5 residents per rotation, the number of residents beginning with each station was unequal. Including all starting points, residents' mean station total scores improved from their first station to their last station (change = 0.14 ± 0.64, P<.01), as did their scores in assessment (change = 0.15 ± 0.78, P = .03) and in management (change = 0.20 ± 0.97, P = .02), but not in general communication. When each starting point was examined separately, improvements were not observed in all rotation orders because of small numbers of residents beginning with each station.

Table 5.

Change in Resident Performance from First to Fifth Station (N = 128)

N	First Station	Fifth Station	Faculty 15-Item Station Total Mean Change Score	Mean General Communication Change Score	Mean Assessment Change Score	Mean Management Change Score
30	1	5	2.+0.24*	2.+0.20*	2.+0.30*	2.+0.15
31	2	1	2.−0.02	2.−0.06	2.−0.02	2.+0.07
28	3	2	2.+0.13	2.−0.05	2.+0.27†	2.+0.20
27	4	3	2.+0.23†	2.+0.29†	2.+0.010	2.+0.58‡
12	5	4	2.+0.16	2.+0.19	2.+0.29	2.−0.19
128	—	—	2.+0.14‡	2.+0.10†	2.+0.15*	2.+0.20*

^*Change scores different from 0, P<0.05.

^†Change scores different from 0, P<0.10

^‡Change scores different from 0, P<0.01.

Resident impressions of the OSCE are summarized in Table 6. Half thought the stations resembled real-life clinical encounters, and the majority (73%) thought they provided a good cross-section of substance abuse issues. Over three-quarters of the residents felt the OSCE helped identify strengths and weaknesses, taught them something new, and provided valuable feedback.

Table 6.

Resident Perceptions of OSCE (N = 131)

Question	Resident Responses, N (%)
	Definitely	Maybe	No
Stations resembled real-life clinical encounters	65 (50)	53 (40)	13 (10)
Stations provided a good cross-section of substance abuse issues	96 (73)	31 (24)	4 (3)
Helped identify my strengths and weaknesses	97 (74)	30 (23)	4 (3)
Taught me something ne†	101 (77)	25 (19)	5 (4)
Provided valuable feedback	105 (80)	24 (18)	2 (2)

^†OSCE, objective structured clinical exam.

DISCUSSION

Our results demonstrate that a substance abuse OSCE can teach addiction medicine competencies by providing performance-based assessments and immediate feedback. This study offers unique insights into residents' ability to interact with patients in different stages of change who are using different types of substances. It also reveals new information about trainees' interest and perceived competence in substance abuse assessment and management. As OSCEs can thoroughly test specific skill sets, our results show that assessment and management of substance abuse are equally difficult but significantly more challenging than general communication skills. We conclude that future educational interventions should target specific assessment and management skills.^6,7,11

Because of the higher prevalence of alcohol abuse⁴¹ than other substance abuse disorders in medical settings, we hypothesized that residents would perform better on the alcohol stations. However, performance was weakest on the station portraying a contemplative alcohol-abusing patient, suggesting that this station presents uniquely difficult challenges. While contemplative patients are aware of their need to change, the ambivalence that characterizes this stage can produce resistance in the doctor patient encounter.⁴² Contemplation is the stage that most calls upon the use of motivational interviewing strategies, which emphasize empathic listening and address resistance by helping patients identify their own reasons for change.⁴² These strategies may employ communication techniques such as “ask, tell, ask,” which involve asking permission before delivering therapeutic suggestions and asking for the patient's reaction afterwards, or may include coaching the contemplative patient to weigh the pros and cons of change. Motivational interviewing techniques require specialized training and practice and may not be in the general skill set of resident trainees. To our knowledge, previous studies have not examined how different stages of readiness to change affect physician performance during clinical encounters. Because motivational techniques for contemplative patients could be applied to other behaviors such as smoking cessation and weight loss, further research is needed to determine how best to train residents to work with patients in different stages of change.

The high correlation of faculty global ratings and standardized patient ratings demonstrates interrater reliability and supports the convergent validity of the rating system. In contrast, residents' self-assessments were lower than faculty or SP ratings. Previous studies have shown that self-ratings do not correlate well with objective measures of performance, and that high performers in particular tend to underestimate themselves when compared with expert raters.⁴³ Despite these limitations, self-assessment may encourage self-reflection and stimulate future learning.

We were surprised to find that residents' self-assessed interest and competence were not associated with OSCE performance. While residents' perceived themselves as more competent in alcohol use-specific than drug use-specific skills, these perceptions did not match their performance. This suggests that residents may not have accurate perceptions of their abilities in substance abuse management. A substance abuse OSCE may therefore help identify strengthens and weaknesses, as most of the residents confirmed.

Instant feedback is highly useful in forming trainee skills and has been shown to influence performance on subsequent stations with similar content.^21,44,45 Objective structured clinical exams with immediate feedback delivered at each station have been very well received by medical trainees.⁴⁴ Immediate feedback has the additional advantages of diminishing assessor fatigue and enhancing faculty teaching skills, while preserving reliability.¹⁸ In this OSCE feedback resulted in improved resident performance in subsequent stations, an important measure of skills acquisition and learning. As a formative assessment, our OSCE achieved its goal of teaching competencies in addiction medicine. Of note, residents who recounted prior substance abuse training performed better than those who had not. This finding further supports the influence of specific training on skill development.

A limitation of our OSCE is that its interstation reliability was moderate (Cronbach's α = 0.64), which may reflect the relatively small number of stations (n = 5) and the differences in raters' styles. It may also reflect the wide range of skills represented by items in the rating forms. For example, the domains of assessment and management included divergent tasks in different stations, while general communication skills were the same across stations. The limited number of stations evaluating each stage of change and substance restricts our ability to draw conclusions from performance differences. Although half of the residents had prior substance abuse training, an additional limitation is that the quantity and quality of these experiences is unknown and likely varied among respondents. We also did not control for residents' past personal experiences, which may influence both motivation and integration of structured training. Finally, in our assessment of the impact of feedback on performance improvement during the OSCE, we did not control for experience by including a control group not receiving feedback.

Conducting the substance abuse OSCE enabled us to tailor our curriculum to better meet the learning needs of our generalist trainees. Before implementing the OSCE, our curriculum included lectures during ambulatory blocks, skills training on alcohol diagnosis and management using simulated patients, and visits to 12-step meetings. New curricular innovations include inpatient lectures on addiction, cocaine use, methadone treatment, and medical detoxification; enhanced use of SPs to teach motivational interviewing skills; and seminars on prescription opiate use, buprenorphine, and physician impairment.

In conclusion, our 5-station OSCE provides a moderately reliable summative measure of our residents' skills in substance abuse and a great deal of information about resident performance in distinct skill areas. While residents perform well in general communication skills, they generally lack adequate skills for assessing and managing substance abuse disorders. In particular, residents need further training in motivational techniques for dealing with resistance in contemplative substance abusing patients. Our results also demonstrate the positive effect of feedback on learning addiction medicine competencies. Implementing the substance abuse OSCE has allowed us to employ new strategies for faculty development, to develop a novel educational program for trainees from different departments, and to develop curricular innovations in substance abuse.