Development of a Simulated Objective Structured Clinical Exam for the APPLIED Certification Exam in Anesthesiology: A Two-Year Experience Informed by Feedback from Exam Candidates

Abstract

Background:

Program directors of anesthesiology residencies agree that it is the program's responsibility to prepare residents for primary American Board of Anesthesiology (ABA) certification, although few report an Objective Standardized Clinical Exam (OSCE) program reflective of the new ABA examination. We created an authentic simulated OSCE (SOSCE) using existing resources to prepare third-year clinical anesthesia residents for the ABA APPLIED exam before graduation and identify knowledge gaps relevant to the OSCE.

Methods:

Junior anesthesiology residents and medical students acted as standardized patients for the 7 SOSCE stations. Third-year clinical anesthesia residents were evaluated on performance by faculty educators during the SOSCE and completed surveys regarding their experience. Follow-up surveys were distributed to participants after they completed the ABA APPLIED Exam.

Results:

Mean scores ranged from 82.6% correct (echocardiogram) to 97.2% correct (ultrasonography). Knowledge gaps were present in competencies explicitly stated as objectives by the ABA. Echocardiography scores improved from 76.1% in the first year to 90.0% in the second year (P = .009). Participants found the SOSCE to be valuable in preparing for the OSCE and the standardized patients' performance to be convincing. Participants felt better prepared for the ABA exam and thought that the SOSCE was authentic in content and process.

Conclusions:

An SOSCE program can be developed with preexisting resources. This program was highly rated as useful and informative, an accurate reflection of the ABA OSCE, and helpful in preparation for the examination. Development of a SOSCE program is feasible, sustainable, and valuable.

Introduction

The Objective Structured Clinical Examination (OSCE), developed by Harden in 1975, assesses clinical competence with objectivity and reproducibility based on well-defined tasks and established criteria.¹ It permits evaluation of a wide range of learner skills and offers opportunity for formative and/or summative evaluations. Anesthesiology residency program directors (PDs) were prompted to prepare graduating residents for the incorporation of an OSCE into the examination for the American Board of Anesthesiology (ABA) certification in anesthesiology in 2017. In 2018, the OSCE was incorporated into the APPLIED examination.

After residency training, most medical and surgical boards require a knowledge assessment to define competency, typically comprising written and oral examinations. The Miller pyramid of clinical assessment describes four levels: knows, knows how, shows, and does.² The written and oral components of the ABA examination address the first 2 levels respectively. The OSCE assesses cognition and behavior, or the ability of the learner to show, level 3. Simulation-based OSCEs became components of the certification process for anesthesiology in the UK in the mid-1990s,³ in Israel in 2003,^4,5 and in Canada in 2010.6 In the United States, most residents have experienced an OSCE during medical school and in the United States Medical Licensing Examination (USMLE) step 2 clinical skills examination since 2004.⁷ With the addition of the ABA OSCE, learners now complete a high-stakes summative OSCE assessment in anesthesiology.

Anesthesiology residency PDs in the United States agree that it is the responsibility of the program to prepare its residents for primary certification, and 100% report providing mock oral examinations.⁸ However, less than one-third report having any form of OSCE program, and only 2 report an OSCE with 6 to 7 stations reflecting the format of the OSCE for the ABA APPLIED examination. Of programs that have no OSCE training at all, 75% indicated the desire to start one. Lack of time, expertise, and funds are commonly cited as obstacles to developing such a program.⁸

We believe that including a full, simulated OSCE (SOSCE) in an anesthesiology residency program can help residents prepare for a high-stakes examination with formative feedback in a low-stress environment and identify knowledge gaps relevant to the OSCE. Evidence is needed to confirm that such programs are effective for the optimal use of residency resources. We report the feasibility, sustainability, and value of developing a SOSCE training program replicating the format of the ABA examination for certification and describe feedback from ABA candidates who compared their SOSCE experience with the actual certification examination experience. This study was approved by the Johns Hopkins University School of Medicine Institutional Review Board (IRB 00110777) and the requirement for informed consent was waived.

Materials and Methods

Development

Our institution is an academic tertiary care hospital in the Northeastern United States with an anesthesiology residency program averaging 25 residents per year. Since 2009, our anesthesiology residency program has offered a well-established simulation curriculum, including formative OSCE components such as standardized patient interviews. After the ABA announcement for OSCE incorporation into the APPLIED examination, a SOSCE workgroup comprising PDs, the education leadership committee, education research experts, and experienced simulation educators, was formed to develop a simulation of the proposed ABA certification assessment. The group reviewed documents published online by the ABA regarding the format of the examination.⁹ The residency PD and one of the authors (DS) participated in preparatory presentations provided by the ABA for all PDs and a mock examination presented by the Society of Education in Anesthesia in 2017. Using the published examination format, 7 of the 9 topics were selected, including 5 of the 6 from Communication and Professionalism (Practice-Based Learning & Improvement, Informed Consent, Peri-procedural Complications, Ethical Issues, and Communications with Other Professionals) and 2 of the 3 from Technical Skills (Interpretation of Echocardiograms and Application of Ultrasonography). The SOSCE excluded Treatment Options, which we felt we could incorporate into the Informed Consent scenario, and Interpretation of Monitors, which we felt would be more intuitive than Echocardiograms or Ultrasonography.

Seven faculty members were involved in the development of the SOSCE program. Each faculty member developed 1 scenario and provided an assessment tool that incorporated the learning objectives published by the ABA for the APPLIED OSCE.⁹ Stems and an assessment tool for each scenario are available in Appendix A. Scenarios were reviewed by the SOSCE workgroup. All faculty members were familiar with the content of each scenario and were comfortable with the assessment tool and with giving feedback on each resident's performance. Medical students and junior residents were recruited for the roles of standardized patients (SPs) and worked with the faculty member to develop the scenario, research the relevant background medical knowledge, and rehearse the role.

At our institution, anesthesiology residents have protected didactic days bimonthly that include 2 hours for simulation and hands-on workshop activities. This existing format was modified to accommodate a 7-station SOSCE in 2-hour sessions for graduating third-year clinical anesthesia (CA3) residents, scheduled over 4 time slots on 2 dates in 2017 and 4 time slots on 2 dates in 2018 (see Figure 1 for study flowchart). Per ABA guidelines, 7 stations were each allotted 12 minutes: up to 4 minutes to review the OSCE stem before entering the room and 8 minutes for the encounter, a total of 84 minutes of examination time. The 2-hour session included prebriefing and debriefing after completion of all stations. In this way, with 4 sessions, we were able to accommodate as many as 28 CA3 resident participants before graduation.

Figure 1.

Flowchart showing the timing of the OSCE sessions and the number of residents included in each session by study years.

Setting

In 2017, the SOSCE was conducted in an unused clinical area with patient bays and privacy curtains that reflected an authentic clinical examination area with patient monitors, stretchers, etc. Additionally, video monitors, recording equipment, and items such as an ultrasound machine were used. In 2018, the SOSCE was held in a dedicated simulation center specifically designed for standardized patient encounters. It had private examination rooms, examination tables, chairs, patient gowns, audio and video observation, and recording capability.

Administration

Each participant was paired with a faculty examiner and began the examination at a different station. The examiner accompanied the participant to each station in a round robin format, observed their performance, and evaluated them according to the assessment tool. To replicate the OSCE format, feedback was not given between stations except in the Echocardiogram station, where a transesophageal echocardiography-certified cardiac anesthesiology faculty member dedicated to this station projected echocardiogram still images and video loops and presented printed questions. After completion of the questions, that faculty member scored the echocardiogram portion of the OSCE and reviewed answers with the participant. At the end of the OSCE, the faculty member debriefed with the participant, provided specific feedback on each station, and gave an overall performance assessment. Participants then filled out a Likert-style evaluation of the SOSCE experience and had an opportunity to provide free-text comments.

Evaluation

Results from the assessment tools were tabulated for each resident. For all stations except Interpretation of Echocardiograms where a binary Correct/Incorrect scale was used, individual assessment objectives were graded on a yes/no scale with 2 or 0 points assigned, respectively. In some instances, evaluators chose to designate a third maybe option that was assigned 1 point. An overall performance evaluation inquiring whether a candidate should pass each station was also rated on a Yes/No/Maybe scale. Mean scores were calculated for each objective and for each resident's total performance.

After reviewing the resident performances, faculty evaluators, who each assessed several residents over the breadth of all stations, and SPs, who witnessed a range of performances for any given scenario, compiled a written document entitled Pearls and Pitfalls (Appendix B), where learning points for each scenario anonymously highlighted aspects of exemplary performances and common traps or behaviors to avoid. This document was distributed to participants shortly after completion of the SOSCE.

All CA3 residents who participated in the SOSCE before graduation and were eligible to sit for the ABA OSCE examination were contacted to complete a follow-up evaluation asking for feedback on how well the SOSCE prepared them for the OSCE.

The number of items on the assessment tool per station proved incongruous. We calculated scores for each station as percentages by dividing each total raw score by its maximum possible score to allow for unbiased comparisons. Based on these percentages, we calculated mean, standard deviation, and confidence intervals for each station and performed independent t tests for each station to compare results from 2017 and 2018. Resident responses to the SOSCE evaluation survey and follow-up evaluations were presented as frequencies and percentages. All statistical analyses were conducted with Statistical Package for the Social Sciences (IBM SPSS Statistics for Mac, Version 25.0. Armonk, New York: IBM Corp.), and significance level was set at a P < .05.

Results

A total of 23 (out of 24 eligible) CA3 residents in 2017 and 27 (out of 28 eligible) in 2018 participated in this activity, representing a 96% participation rate. Twenty-nine residents were female, and 21 were male; 36 were 28 to 33 years old, and 14 were > 33 years old. Resident performance data from all 50 participants were collected. Forty-five participants completed the survey evaluating the SOSCE. Follow-up evaluations comparing the SOSCE with the ABA OSCE were collected from 18 residents who participated in 2017 and 11 from 2018 who have sat for the ABA OSCE.

Overall, residents performed well on the 7 individual stations with all scores ≥ 50%, albeit with variability. Mean scores ranged from 82.3% on Interpretation of Echocardiograms to 97.2% on Technical Skills – Ultrasonography (Table 1). Independent t tests showed no significant difference in performance between the 2 years for any category except Echocardiograms. The mean score on Echocardiograms increased from 76.1% in 2017 to 90.0% in 2018 (P = .009). Data from both years were combined for each station for further analysis.

Table 1.

Descriptive Statistics for Anesthesiology Residents' Performance ^a by Scenario

Scenario	Nb	Mean (%)	SD (%)	Min (%)	Max (%)	95% CI (%)
Practice-based learning & improvement	50	93.1	12.7	50	100	89.5–96.8
Informed consent	50	93.7	10.2	61	100	90.8–96.6
Peri-procedural complications	50	87.7	9.5	67	100	85.0–90.4
Ethics	50	91.7	12.0	50	100	88.2–95.1
Communication with other professionals	50	94.4	7.1	70	100	92.4–96.4
Ultrasonography	50	97.2	6.3	70	100	95.4–99.0
Echocardiograms	43	82.6	17.9	40	100	77.1–88.1

^a Performance scores were calculated as the percent of maximum possible points based on the scoring sheet for each scenario.

^b Total sample size reflects combined post graduate year 4 anesthesiology residents from 2017 and 2018 groups.

The most frequently missed tasks are presented in Table 2. The first 4 elements are explicitly stated objectives listed in the ABA OSCE content online; the last, offering to consult a colleague in Communication with Other Professionals could be a component of the broader need for participants to understand the perspectives of other health-care professionals. In the Echocardiogram station, residents had higher scores when asked to identify views or structures on still images with normal anatomy than they did with video clips that showed pathology. Similarly, scores were lower when participants were asked for management goals based on the aforementioned pathology.

Table 2.

Most Frequently Missed Tasks by Scenario

Scenario	Task	Scorea
		0 = No	1 = Maybe	2 = Yes
Peri-procedural complications	Discusses potential causes of complication (headache)	30 (60)	1 (2)	19 (38)
	Discusses most common course and outcome of complication (postdural puncture headache)	13 (26)	5 (10)	32 (64)
Communication with other professionals	Offers to consult a colleague	19 (38)	1 (2)	30 (60)
Ethics	Assures patient conversation is kept confidential	18 (36)	3 (6)	29 (58)
Echocardiogramsb	Identifies pathology on video loop 2	12 (28)	31 (72)	…
	Identifies view on video loop 3	13 (30)	30 (70)	…
	Identifies pathology on video loop 3	13 (30)	30 (70)	…
	Identifies 1 management goal based on pathology identified in video loop 3	12 (28)	31 (72)	…
	Identifies second management goal based on pathology identified in video loop 3	8 (19)	35 (81)	…
Informed consent	Identifies all persons who will participate in the procedure	9 (18)	4 (8)	37 (74)

^a All score distributions are presented as frequency (%).

^b Echocardiogram scores are based on 0 = not completed, 1 = completed.

Participants thought that the SOSCE was a valuable experience, with 100% agreeing that the simulation was useful, informed and prepared them for the ABA OSCE examination, and should remain in the curriculum (Figure 2). All participants also agreed that performances by the SPs (medical students and junior residents) were convincing. A majority, 87%, did not feel uncomfortable being tested in front of their peers.

Figure 2.

Resident responses to simulated OSCE evaluation survey. Total sample size reflects combined Post-Graduate Year 4 anesthesiology residents from 2017 and 2018 (n = 45). For comparison purposes, the scale for negative items was reversed to show responses from strongly disagree (indicated in red) to strongly agree (indicated in green).

Follow-up evaluations were sent to residents who completed the SOSCE and the ABA OSCE examination. Twenty-nine SOSCE participants completed the post-ABA OSCE evaluation survey (Figure 3). Of those, 96% thought that the SOSCE accurately reflected the OSCE examination process and content. All of the participants agreed that the SOSCE should continue to be part of the CA3 curriculum, and 96% felt that the SOSCE gave them accurate information on what to expect on the examination. All thought that the SOSCE helped them to prepare, 61% used the Pearls and Pitfalls document when preparing for the examination, and 55% felt that the SOSCE and subsequent review of the Pearls and Pitfalls was sufficient preparation for the OSCE. The majority (89%) indicated that the SOSCE preparation permitted more focus on preparation for the oral component of the APPLIED examination; similarly, 71% disagreed that preparation for the OSCE was a distraction from preparation for the oral component. The majority (89%) did not feel that the OSCE was more difficult than the SOSCE.

Figure 3.

Resident responses to simulated OSCE follow-up survey. Note that OSCE exam refers to the actual American Board of Anesthesiology exam experience, and SOSCE refers to the simulated OSCE conducted in our program. The sample includes classes from 2017 (n = 18) and 2018 (n = 11). For comparison purposes, the scale for negative items was reversed to show responses from strongly disagree (indicated in red) to strongly agree (indicated in green).

Open comments were very positive, reflecting that the SOSCE was valuable, helpful in preparing for the examination, and an accurate reflection of the examination experience. “The whole exam was incredibly realistic and very accurate… It was extremely helpful and a large component of my preparation for the actual OSCE and was instrumental in building my confidence during the exam.” We also asked for suggestions for improvements. Several candidates reported a station on Interpretation of Monitors with analysis of changes in vital signs and correlation of a clinical event, and would have benefited from preparation for this. Several candidates also suggested more extensive questions for the Ultrasonography station including more anatomical sites with specific identification of anatomical structures.

Discussion

The SOSCE format represents the timing, pace, and content of the OSCE. A survey designed specifically to assess residency program preparation for the APPLIED examination (sent 6 weeks after release of the ABA OSCE content online) showed that while one-third of programs had an OSCE of some kind as part of the curriculum, only 2 programs had an OSCE of sufficient length and number of stations to replicate that proposed by the ABA.⁸ Based on positive feedback, we repeated the SOSCE the following year with an identical format while eliciting follow-up feedback from graduates who had taken the ABA OSCE. Although changes to the OSCE may be made over time, armed with current information our format is adaptable and flexible. The examination continues to be modified and updated; our report is representative of the format during the study period. Please consult the ABA website for the most current examination format.

Tanaka et al¹⁰ reported on the use of the ABA APPLIED Examination OSCE content as a blueprint for a 9-station mock OSCE similar to the method we employed. They reported data on 14 participants, or two-thirds of the graduating class, who rated the realism of each station. Our work builds on this report by demonstrating a broader application of the program and sustainability over multiple years. We asked participants who sat for the examination to evaluate usefulness of the experience rather than realism, as no one had a benchmark for comparison at that time. The ABA has not published national statistics on pass rates on the ABA OSCE, grading rubrics, or how many individual stations are required for an overall passing score; therefore, it was difficult for us to relate the scores our residents received on the SOSCE with performance on the ABA OSCE. However, 89% of participants did not feel that the OSCE was more difficult than the SOSCE, and all of the participants in the SOSCE passed the advanced exam. Thus, we may conclude that our SOSCE scoring results could be used as an estimate for a passing score on the ABA OSCE but not as a benchmark for a minimum passing score.

We replicated the examination format by adapting an established educational timeslot with existing designated teaching faculty on those days. Initially we conducted the OSCE in an unused postanesthesia recovery unit, subsequently in a dedicated simulation space, with no apparent difference in learner experience or performance. The SOSCE did not require additional funding. We mentored other trainees in the roles of SPs who had a valuable and informative experience developing and standardizing their roles with faculty guidance. Most participant examinees thought that the experience was authentic. Although most participants did not report feeling uncomfortable being tested in front of peers, a few did. We do not know whether this discomfort had any implications for the realism of this experience.

Scores for each station were consistent over the study period except in the Echocardiogram station, where we saw significant improvement. We attribute this to a dedicated point-of-care ultrasound curriculum introduced the year before the start of the SOSCE that focused heavily on echocardiography. Second-year SOSCE participants had the benefit of 2 full years of this training.

The use of the SOSCE has allowed us to identify areas where residency training alone may be insufficient in preparing candidates for the ABA APPLIED examination. The Pearls and Pitfalls handout given to all SOSCE participants addresses these items, and we have modified our residency curriculum to ensure that all elements of the ABA APPLIED examination are covered.

Our participants received immediate formative feedback on their performance, addressing questions and concerns in real time. Our format did not require retroactive viewing of recorded performances, but our method could be adapted to do so if we wanted to provide assessment from more than 1 faculty member, limit the number of faculty needed on site during the event, or allow candidates to view their own examination performance. For our needs, it was most efficient to provide this experience for up to 7 residents with 7 faculty over a 2-hour period, repeated 4 times over 2 days. This schedule permitted nearly all CA3s (96%), who might otherwise have been postcall or on vacation, to participate in the SOSCE. Alternate scheduling formats may better suit the needs of other programs.

In their evaluations, 100% of participants strongly agreed that the simulation should be included in the curriculum for future years. Although this is compelling evidence, it means replacing another activity in the existing protected didactic time.

As we embark on our third year, this program seems sustainable. We have made minor adjustments to the objective criteria, clarified printed instructions, and moved to a formal examination space. Ongoing data collection and feedback continue to inform our process. Based on feedback from the candidates for the ABA OSCE, the 2 most difficult stations were Interpretation of Echocardiograms, which we did simulate, and Interpretation of Monitors, which we have since incorporated into the SOSCE. The difficulty of the Ultrasonography station has been enhanced by adding anatomical sites with requirements for identification of more specific structures.

Among the limitations of this format is the potential loss of objectivity with evaluation by a known faculty member in real time. Harden et al¹ stress the need for using multiple examiners to increase objectivity, which is especially important for summative examinations. We used standardized evaluation criteria across participants and 1 faculty evaluator per participant. As this was a formative and not a summative experience, we thought this was sufficient to meet our primary objectives of representing the format, timing, pace, and content of the examination. While our faculty was comfortable giving feedback on the participant's performance on 6 of the 7 stations at the end of the examination, the Interpretation of Echocardiograms feedback was provided by faculty certified in transesophageal echocardiography interpretation. Feedback was given to individual participants immediately after completion of the station objectives by the content experts to save time required to meet with up to 7 participants at the conclusion of the examination. Some data from this station was lost in 2018, causing the discrepancy with the total number of participants. The scoring rubric for the ABA OSCE is not publicly available; therefore, we cannot compare SOSCE results with a national passing standard for the OSCE. Programs that replicate our SOSCE should be cautious when interpreting the scores of their residents compared with a national standard passing score for the OSCE or the result presented here. These comparisons may not reflect actual performance by participants due to differences in fidelity of the SOSCE, stringency of grading by faculty, and by variation in residency curriculum. Finally, we were only able to capture 58% of participants in the post-ABA examination participation survey. This is in part due to many 2018 participants not yet having taken the ABA OSCE at the time of our survey as well as the voluntary nature of responding. It is possible that participants were more likely to respond to the survey if they felt they had performed well on the OSCE, which may have biased their responses in favor of the SOSCE, although all participants passed the advanced examination.

In conclusion, development of an OSCE program can be achieved by using preexisting content blueprints and other published examples of programs such as this. Dedicated simulation space, special equipment, and high-fidelity manikins are not required for this type of simulation. Clinical areas located close together with moderate privacy, such as unused postanesthesia care unit bays, can be easily transformed into examination space, perhaps in the late afternoon or on weekends. For maximal efficiency, we offered this experience to 7 participants simultaneously over 2 hours requiring a coordinated faculty presence. In lieu of professional SPs, trainees participated enthusiastically with minimal impact on participants and benefited from an opportunity for mentorship with a faculty member. Participants found this exercise to be valuable, informative, anxiolytic, and an accurate model of actual examination content and process. Feedback from participants is an important component of any SOSCE program and should be used as a metric for maintaining fidelity to the evolving APPLIED exam.

Appendix A

Appendix A. Stems and Assessment Tools

Simulated OSCE

Name: ________________________

Evaluator: __________________________

SCENARIO 1: PRACTICE-BASED LEARNING & IMPROVEMENT

One of your nurse colleagues in the PACU is working on a patient safety project to reduce medication errors on the unit. She has asked if you would meet with her to discuss how to design a multi-disciplinary patient safety project to address the topic.

Your task is to address a recent medical error with a concerned nurse manager and determine a plan for improving patient safety.

SCENARIO 1: PRACTICE-BASED LEARNING & IMPROVEMENT

• Y / N Did the resident greet the nurse in a friendly and collegial fashion (shake hands and/or friendly conversation)?

• Y / N Did the resident maintain a professional demeanor throughout the encounter (looks colleague in the eye, sits and engages in conversation, not rushed and generally interested in helping)?

• Y / N Did the resident ask about baseline information such as needs assessment and definition of the problem (seeks clarity, asks clarifying questions)?

• Y / N Did the resident help determine stakeholders and stakeholders' view (asks who will be involved, how many will help propose solutions)?

• Y / N Did the resident help the colleague identify root causes and barriers to change (asks where the obstacles to change are and what other information is needed if not already known from needs assessment and problem definition)?

• Y / N Did the resident discuss how to establish and implement a plan of action (offers ideas about how to make a plan or prompts colleague on plan)?

• Y / N Did the resident help the colleague determine how to measure outcome (asks about desired outcome and possible metrics for measurement)?

GLOBAL ASSESSMENT:

Should this candidate pass this station? Yes / No / Maybe

COMMENTS:

SCENARIO 2: INFORMED CONSENT

PCA vs PCEA

Ms. B is an anxious 67-year-old woman with hypertension and COPD presenting this morning for a large ventral hernia repair. You have elicited a history and performed a physical exam, deemed her an acceptable risk candidate for surgery and obtained consent for general anesthesia. The patient asks you about epidural versus IV postoperative pain management. Please explain the risks and benefits of epidural and/or IV PCA for postoperative pain control and, if patient requests an epidural, obtain informed consent for placement.

Your task is to obtain informed consent from the patient for a method of postoperative pain control.

SCENARIO 2: INFORMED CONSENT

• Y / N Explains conduct of procedure in lay terms

• Y / N Explains treatment options for postoperative pain control and presents alternatives, if appropriate

• Y / N Explains benefits and risks of an epidural placement, including both less severe/more common and more severe/less common material risks

• Y / N Identifies all persons who will participate in the procedure

• Y / N Elicits questions from examinee and responds appropriately in lay terms

• Y / N Elicits affirmative consent without coercion

• Y / N Confirms final decision regarding epidural placement with patient

• Y / N Demonstrates understanding and concern for the patient

• Y / N Acknowledges patient's anxiety and shows empathy

GLOBAL ASSESSMENT:

Should this candidate pass this station? Yes / No / Maybe

COMMENTS:

SCENARIO 3: PERI-PROCEDURAL COMPLICATIONS

POSTPARTUM HEADACHE

Mrs. T is a healthy 35-year-old G2P2 woman with gestational diabetes who underwent a forceps assisted vaginal delivery after prolonged pushing in the labor and delivery room. Her baby weighed 9 lb 5 oz at birth and is doing well in the nursery. For labor analgesia, the patient received a combined spinal epidural. She is 24 hours postdelivery and feeling well except for a severe positional headache that occurs when standing and is relieved when laying down. The patient is very concerned because her headache is impeding her ability to care for her newborn. Review of the anesthetic record reveals no apparent complications during CSE placement. You are called by the nurse before morning rounds to evaluate Mrs. T.

Your task is to evaluate a patient with postpartum headache and determine a plan of care to address her concerns.

SCENARIO 3: PERI-PROCEDURAL COMPLICATIONS

• Y / N Knocks on door, washes hands, and introduces self

• Y / N Elicits history relevant to postdural puncture headache

• Y / N Performs focused physical evaluation

• Y / N Discusses potential causes of headache

• Y / N Discusses most common course and outcome of post-dural puncture headache

• Y / N Discusses conservative vs. definitive treatment

• Y / N Presents plan to follow up with patient & further evaluation if deemed appropriate

• Y / N Elicits questions and responds appropriately

• Y / N Demonstrates understanding and concern for the patient

GLOBAL ASSESSMENT:

Should this candidate pass this station? Yes / No / Maybe

COMMENTS:

SCENARIO 4: ETHICS

JEHOVAH'S WITNESS FOR SPINE SURGERY

Ms. B is a 28-year-old female who is scheduled for scoliosis surgery today. You are doing the pre-anesthesia assessment prior to going to the operating room for surgery. She has a Cobb's angle of 33 degrees. The surgery will be a T5-S1 Laminectomy and Fusion.

She is a physical therapy student in her senior year. Her neurological exam has been stable until last year when she started having a left foot drop that now affects her ability to work. She has no other significant medical history but had a prolonged recovery from a wisdom tooth extraction with prolonged oozing from the surgical site. Workup for bleeding disorders was unrevealing. Ms. B is a Jehovah's Witness.

EKG: NSR

CXR: lung fields clear. Scoliosis evident.

Labs:

Hb: 13.1, Hct: 33, Platelets: 185, WBC: 4.2

Na: 139, K: 4.1

BUN: 18, Creatinine: 0.9

Glucose: 92

PT: PTT: INR 1.1

Your task is to assess the patient's wishes regarding blood products and establish a safe and mutually agreed upon plan for intraoperative resuscitation.

SCENARIO 4: ETHICS

• Y / N Actively explore patient's wishes

• Y / N Assures patient conversation is kept confidential

• Y / N Facilitates the process of informed decision-making

• Y / N Establishes a cooperative and respectful relationship with patient

• Y / N Communicates in a clear and professional manner

• Y / N Prioritizes communication of information relevant to this patient's care

GLOBAL ASSESSMENT:

Should this candidate pass this station? Yes / No / Maybe

COMMENTS:

SCENARIO 5:

COMMUNICATION WITH OTHER PROFESSIONALS

Dr. Davida Keller is an orthopedic surgeon who is posting the following case: a 45-year-old patient requires an ORIF for a closed tibia and fibular fracture that he sustained after falling off his bike. The patient is hemodynamically stable, GCS 15, has no significant past medical history, and had a full meal approximately 2 hours ago. The surgeon would like to proceed with the surgery now as it is late in the day and she says she has obligations later in the evening. As the anesthesiologist, please explain to the surgeon why this case should not proceed immediately.

Your task is to discuss with the surgeon the risks, benefits, and alternatives to proceeding with this case immediately.

SCENARIO 5: COMMUNICATION WITH OTHER PROFESSIONALS

• Y / N Introduces himself/herself, offers to shake hands

• Y / N Sits down with surgeon

• Y / N Makes eye contact

• Y / N Listens actively without interrupting

• Y / N Stays calm and professional

• Y / N Effectively summarizes patient's situation

• Y / N Explains why the case should wait in calm, clear manner

• Y / N Acknowledges concerns of surgeon respectfully

• Y / N Proposes options (delay case, list as urgent, discuss with patient)

• Y / N Offers to consult a colleague

GLOBAL ASSESSMENT:

Should this candidate pass this station? Yes / No / Maybe

COMMENTS:

SCENARIO 6: TECHNICAL SKILLS I

ULTRASONOGRAPHY

This OSCE scenario will focus on ultrasonography skills.

While this is a technical examination, please maintain a physician and patient interaction throughout. When it is time, please enter the examination room. Using the correct probe and orientation, you will be asked to identify the following structures (please note, you may be asked to use a pointer to demonstrate what approach you might use for intravenous catheter and / or nerve block placement):

Your task is to identify structures and answer questions related to several sites.

SCENARIO 6: TECHNICAL SKILLS I (ULTRASONOGRAPHY)

• Y / N Did the resident introduce himself or herself to the patient?

• Y / N Did the resident wash his or her hands or use hand sanitizer prior to their introduction?

• Y / N Did the resident maintain a professional demeanor throughout the encounter?

• Y / N Did the resident successfully identify the brachial plexus using the supraclavicular approach?

• Y / N Did the resident appropriately position the needle as if he or she would perform the block?

• Y / N Did the resident correctly identify the right internal jugular vein?

• Y / N Did the resident appropriately position the needle as if he or she would place a central line?

• Y / N Did the resident successfully identify the right carotid artery?

• Y / N Did the resident successfully identify the radial artery?

• Y / N Did the resident successfully identify a vein appropriate for IV catheter placement?

GLOBAL ASSESSMENT:

Should this candidate pass this station? Yes / No / Maybe

COMMENTS:

SCENARIO 6: TECHNICAL SKILLS II

ECHOCARDIOGRAMS

Your task is to evaluate Trans-Esophageal Echocardiogram images and answer the clinically related questions.

SCENARIO 7: TECHNICAL SKILLS II (ECHOCARDIO-GRAMS)

Image Identification (45 seconds)

• Y / N Selects the proper name of the view from a list of standard views.

• Y / N Identifies 3 labeled structures of TEE image shown.

  1. ___________
  2. ___________
  3. ___________

Short Case (2 minutes)

• Y / N Selects the proper name of the view from a list of standard views.

• Y / N Provides the best diagnosis based on the echocardiographic findings.

• Long Case (3 minutes with short case scenario)

• Y / N Selects the proper name of the view from a list of standard views.

• Y / N Provides the best diagnosis based on the echocardiographic findings.

• Y / N Explains 2 hemodynamic goals in terms of intraoperative management for this patient.

  1. ___________
  2. ___________

GLOBAL ASSESSMENT:

Should this candidate pass this station? Yes / No / Maybe

COMMENTS:

Appendix B

Appendix B. Faculty Recommendations

ABA OSCE Best Practices: Pearls & Pitfalls

Michael Banks, Steve Beaudry, MaryBeth Brady, Gillian Isaac, Megan Kostibas, Christina Miller, Adam Schiavi, Deb Schwengel, Tina Tran, Jed Wolpaw, David Berman, Katy Norgaard, Alyson Russo

This exam, much like the traditional oral boards, is as much about acting and communicating in a clear, calm, and organized manner as it is about what facts you know. You must perform.

EXECUTIVE SUMMARY: These suggestions apply to every station:

WASH YOUR HANDS (you are on camera and they can see what you do outside the room)

KNOCK

INTRODUCE YOURSELF

SHAKE HANDS and SMILE, if appropriate

SHOW EMPATHY. Use the phrase “I know this must be really X for you” where X is difficult, painful, frustrating, etc. You can use this with patients “I know this must be really difficult for you” or with surgeons “I know this situation must be really frustrating for you.” Sit down at the same level as your patient/surgeon. Make eye contact, nod, look interested.

ADJUST YOUR LANGUAGE TO SUIT YOUR AUDIENCE. When you are talking to a patient, make sure you use lay terms. When you are talking to a surgeon or nurse manager, use professional medical language or the lingo of quality improvement.

SOLICIT QUESTIONS. Ask “What questions do you have for me?”

PLAN TO FOLLOW UP. With patients, “I will follow up with you later, or tomorrow, or I will have my colleague check in with you.” With the surgeon, “I will discuss this with my department so that we can make sure we can avoid delays whenever possible.”

STATION-SPECIFIC COMMENTS

1. Informed Consent (Pain Management Options – PCA vs PCEA)

• - ask what the patient wants; do not assume that they want or do not want an epidural

• - confirm final decision after all options have been discussed and weighed

• - specifics may be helpful in terms of risk assessment, ie, say “There is a 1 in 100,000 risk of nerve damage” rather than “very rare”

• - do not use the term NARCOTIC - the correct term is OPIOID

• - listing specific side effects from opioids (itching, sleepiness, constipation, nausea) is helpful in weighing risks/benefits

2. Peri-procedural Complications (Postdural Puncture Headache)

• - demonstrate empathy (“this is distressing, unfortunate,” “I know this must be difficult for you,” etc)

• - develop patient rapport; asks about new baby and congratulate patient

• - focus on getting the patient better and redirect away from negative thoughts

• - name specific conservative treatment options - Tylenol, ibuprofen, PO and IV fluids, caffeine, etc

• - give expected time frame for recovery given conservative vs blood patch management

• - several residents drew a picture to illustrate the problem which was noted to be helpful and effective

• - do not assume that PDPH is the ONLY possible diagnosis; inquire about other symptoms and check for gross neurologic deficits by checking pupils, cranial nerves, extremity exam - could also be SAH, meningitis, migraine, etc - R/O other more serious things or need for further work-up

almost no one did a physical exam which was a required element for this scenario; it was crucial to have a plan to follow up with the patient

3. Quality Improvement (this one seems to be all about using the correct lingo)

• - engage with the professional and accept that they are concerned about a legitimate problem

• - shake hands, make eye contact, nod and show concern

• - define specific stakeholders (and used the term) to include in the discussion - pharmacy, EMR, hospital administrators

• - identify root causes and barriers/obstacles to change; talk about things like buy-in and changing culture with data

• - provide specific examples of what to study - plan of action

• - determine how to measure outcomes and provide specific metrics; perhaps prospectively looking at number of errors (or near misses) documented, or surveying participants as to effectiveness of plan

• - offer to follow up or meet again to discuss progress

several residents missed the root cause/barrier and outcome measure/metrics portion; discussions tended to be more preliminary and empathetic without generating a game plan

...but don't forget the empathy and enthusiasm

4. Ethics (Jehovah's Witness Patient for Major Spine Surgery)

• - remember this is about the ethical issue and don't lose a lot of time focusing on medical history except as it pertains to risk of greater than usual blood loss and the urgency with which surgery must proceed to preserve nerve function

• - assure patient that conversation is confidential - from family & community; this was a big concern

most residents identified what the relationship of the family member is to the patient but did NOT ask if it was okay to discuss medical decisions in front of them

only a few residents ensured confidentiality from family and/or asked the guest to step out

• - attempt to solicit preferences of the patient despite interference from family; this can be done by directly stating to the family member that you need to understand what the patient wants, and if interference continues, you may need to ask them to step out so you talk to the patient alone; for example, “I need to ask you to step out for a moment so I can complete my ‘physical exam,’” is a smooth way of getting them out of the room without being offensive or dismissive of the patient's relationship/trust of their family member

• - do NOT refer to Hopkins-specific resources (ie bloodless medicine program, DART) on the OSCE

• - do not minimize the risk of death or defer decision to a special team; take responsibility for a potentially serious situation

• - offer to involve surgeon in discussion to determine safe stopping points and possible staging if blood loss is too great

• - albumin is a HUMAN product; it is purified but it is not recombinant and is extracted from human blood; despite this JW patients often accept it, but to say it is not from blood is misleading

A JEHOVAH'S WITNESS DIVERSION

• - the discussion is confidential and at least part of it must be had in isolation from family members; if I don't overtly ask the family to leave for the discussion, I will pause as we are rolling back to the OR and confirm, asking if there is anything else they want to tell me in confidence before I put them to sleep

• - I frame the discussion in this way (feel free to plagiarize):

It is my job to defend your wishes in the OR after I put you to sleep and you no longer have the ability to participate in the conversation.

I will carry out whatever you decide, but I need to know your preferences in detail so that I can make these decisions on your behalf.

When there is life-threatening bleeding, our default is to give blood. If this is not what you want, I need to defend your right to refuse blood to the rest of the staff in the OR who, with the best intentions, will want to save your life (I use the same argument when patients wish to keep DNRs active in the OR... which by the way would be another GREAT ethics OSCE scenario).

I will not give you blood unless it is a life-threatening emergency and if you tell me you would rather die than receive blood, I will not give you blood, even in that situation. I need you to tell me that you would rather die than receive blood products, and then I will honor this wish.

The current JW thinking categorizes blood into major fractions (there are 4: red cells, white cells, plasma, and platelets) and minor fractions (things like cryoprecipitate, albumin, individual factor concentrates from a human source).

Every single Jehovah's Witness has a different interpretation of what they will and won't accept so you have to ask specifics. However, the Watchtower forbids major fractions and most will refuse these, but has a less adamant stance on minor fractions leaving that decision up to the individual so they may be more likely to accept these. Most JW patients will accept recombinant factors (activated FVII for instance, which do not come from a human blood source).

If your patient is struggling with the specifics, I use a chicken soup analogy - major fractions are like chicken, vegetables, noodles and broth. Minor fractions are like salt, pepper or parsley - a tiny component extracted from a large pool and highly concentrated, but still related to the whole.

Most JW patients will not bank their own blood ahead of time -once it leaves the body, even if it is their own, they won't accept it back. Many will be willing to accept cell-saver, hemo-dilution (dialysis etc) techniques but may have specific requirements about the blood always being in continuity with them in a closed circuit; don't promise anything you can't deliver.

One more note - our Bloodless Medicine Program has heavy influence and support from the Jehovah's Witness community so when you get a preop report that lists what the patient is willing to accept, assume that this has been done under the influence of scrutiny from JW community members; you must review with the patient what their preferences are. This is why this frequently changes on the day of surgery from the expectations that you were given initially.

5. Communication with Professionals (Impatient Surgeon wants to Violate NPO Guidelines)

• - stay calm, do not get irritated or respond unprofessionally

• - invite surgeon to sit down with you

• - use empathetic phrases “I understand,” “I acknowledge”

• - emphasize that the case is not emergent, not limb or life-threatening; reframe it as a patient safety issue and weigh the benefits vs risks of proceeding such as aspiration, sepsis, and prolonged intubation

• - offer to help the patient and support the team; assist with pain management, regional block, consult another colleague, and/or take responsibility for doing case later after appropriate fasting

Many residents volunteered to offer pain management support or regional for pain management

Only a few residents offered to consult another anesthesiologist colleague or ask for a second opinion on whether it was appropriate to proceed

6A. Technical Skills I (Ultrasound)

• - indicate needle position and trajectory for block or catheter placement without prompting once optimal image is obtained; it probably doesn't hurt to point out structures you are trying to avoid and how you would accomplish that as well

• - make sure that you are still engaging the model professionally and respecting their modesty; communication and professionalism counts here as well, not just identifying the correct structure

7. Technical Skills II (Transesophageal Echocardiography)

• - Know the 11 named views. You will not have to write these out, but be able to circle these from a list

• - Pay attention to the omni plane on the TEE image to help guide you on which view it is

• - Know the anatomy cold

• - There are only a handful of diagnoses they can ask about, no other crazy diagnoses. Likely, there will not be more than 1 diagnosis on the screen.

• - Here are the ONLY pathologies they can ask:

  1. Biventricular function and wall motion
  2. Presence or absence of an atrial septal defect
  3. Volume status assessment – hypovolemia and response to volume therapy
  4. Pulmonary embolus
  5. Air embolus
  6. Basic valvular lesions
  7. Pericardial effusion
  8. Aortic dissection

• - You will have 30 seconds to look at views and are not able to replay it afterwards. Take all 30 seconds to look at the video.

GOOD LUCK EVERYONE!