The Nightmares Course: A Longitudinal, Multidisciplinary, Simulation-Based Curriculum to Train and Assess Resident Competence in Resuscitation

Abstract

Background 

Postgraduate medical education programs would benefit from a robust process for training and assessment of competence in resuscitation early in residency.

Objective 

To describe and evaluate the Nightmares Course, a novel, competency-based, transitional curriculum and assessment program in resuscitation medicine at Queen's University in Kingston, Ontario, Canada.

Methods 

First-year residents participated in the longitudinal Nightmares Course at Queen's University during the 2015–2016 academic year. An expert working group developed the entrustable professional activity and curricular design for the course. Formative feedback was provided following each simulation-based session, and we employed a summative objective structured clinical examination (OSCE) utilizing a modified Queen's Simulation Assessment Tool. A generalizability study and resident surveys were performed to evaluate the course and assessment process.

Results 

A total of 40 residents participated in the course, and 23 (58%) participated in the OSCE. Eight of 23 (35%) did not meet the predetermined competency threshold and required remediation. The OSCE demonstrated an acceptable phi coefficient of 0.73. The approximate costs were $240 per Nightmares session, $10,560 for the entire 44-session curriculum, and $3,900 for the summative OSCE.

Conclusions 

The Nightmares Course demonstrated feasibility and acceptability, and is applicable to a broad array of postgraduate medical education programs. The entrustment-based assessment detected several residents not meeting a minimum competency threshold, and directed them to additional training.

What was known and gap

Patient resuscitation requires specific skills and can be a daunting task for junior residents who often provide the first line of inpatient care.

What is new

A novel, competency-based, resuscitation course and assessment program distributed over an academic year.

Limitations

Assessment bias due to voluntary participation; relatively high implementation costs.

Bottom line

The course demonstrated feasibility and acceptance by residents, and the assessment identified residents in need of additional training.

Introduction

Due to increased clinical responsibilities and expectations, the transition from undergraduate to postgraduate medical education can be a challenging time for resident physicians. Junior residents are often the first physicians to attend to inpatients with acute illnesses. With the movement toward competency-based medical education, there is an increasing mandate to ensure that residents demonstrate the knowledge, skills, and attitudes required to progress through the stages of training.¹ Patient resuscitation requires specific skills and can be a daunting task for junior residents. A recent meta-analysis of 182 publications endorsed the utility of simulation in resuscitation curricula.² A second meta-analysis found that 67% of transition to residency curricula provided formative feedback to residents, but no programs provided a summative assessment.³ It would be beneficial for postgraduate training programs to establish a robust process for training in and assessment of minimum competencies in resuscitation early in residency.

This article describes and evaluates the Nightmares Course at Queen's University in Kingston, Ontario, Canada. The course is a novel, competency-based, transitional curriculum in resuscitation medicine that uses distributed and deliberate practice facilitated by high-fidelity simulation over a 6-month period. It also provides a summative assessment in the form of an entrustment decision⁴ on the course entrustable professional activity (EPA) common to junior postgraduate residents from several specialties, along with remediation for residents not meeting the competency threshold.

Methods

Setting and Participants

Course participants in the 2015–2016 academic year included residents at the foundations of discipline level of training at an academic teaching hospital. This stage roughly corresponds to the level of residents in their first or second postgraduate year, where a breadth of foundational abilities is acquired prior to advancing to increasingly specialized training.¹ Residents from internal medicine, orthopedic surgery, general surgery, anesthesiology, obstetrics and gynecology, urology, ophthalmology, and neurology participated in the course.

Curricular Design

An EPA common to several specialty training programs was defined with expert consensus input from 2 physicians with expertise in resuscitation (A.K.H. and T.C.; Table 1).

Table 1.

Nightmares Course Entrustable Professional Activity for Critical Interventions in Acutely Unwell Patient

Nightmares Course sessions were run on a weekly basis from August 2015 through February 2016. Over this period, residents self-scheduled 1 session every 4 weeks using an online system. Each session was held in the simulation laboratory at the Kingston Resuscitation Institute within Kingston General Hospital, and 2 simulation mannequins were used: SimMan 3G and SimMan Essential (Laerdal, Toronto, Canada).

Each 90-minute session was attended by 4 to 6 residents and involved 3 simulated scenarios. Topics for each session were chosen based on collected data from the most common calls to our hospital's Rapid Assessment of Critical Events team for acutely unwell admitted patients. A structured debrief⁵ with formative feedback followed each scenario. The sessions were taught by 8 attending physicians with an interest in resuscitation medicine and simulation-based instruction from a variety of specialties (emergency medicine, critical care medicine, anesthesiology, and general internal medicine). Faculty participation was rewarded with acknowledgment via a points-based reward structure from our department of postgraduate medicine.

Approval to perform this study was obtained through the Research Ethics Board at Kingston General Hospital.

Assessment Process

We used a multi-station objective structured clinical examination (OSCE) on March 1, 2016. The OSCE included 4 independent simulation scenarios assessed by 4 attending physician examiners from a variety of specialties (emergency medicine, anesthesiology, general surgery, and general internal medicine). The examiners were not blinded to the identity of the residents because all were already instructors for the Nightmares Course. The residency programs encouraged all residents to participate in the OSCE, but it was not mandatory. The total OSCE time per resident was 60 minutes, about 10 minutes per station and 3 minutes to transition. Residents completed a brief questionnaire following the OSCE.

Stations were mapped by the study authors (L.M., A.K.H., T.C.) to the core 18 scenarios of the Nightmares Course via a blueprinting process⁶ (provided as online supplemental material), and 7 experts with emergency medicine and critical care medicine backgrounds reviewed the stations for content validity. All stations were rehearsed with nursing staff and simulation technicians 1 day prior to the OSCE to ensure a standardized experience.

The Queen's Simulation Assessment Tool (QSAT), an anchored global assessment scoring tool, was modified for use in this setting (provided as online supplemental material). A validity argument for the use of the QSAT in simulated resuscitations has been previously presented.^7,8 Modifications for the Nightmares Course OSCE included adding an EPA to anchor examiners and using an entrustment decision⁴ instead of the previously utilized global assessment scale. All examiners underwent a 30-minute training session led by 1 study investigator (A.K.H.).

Considering the expected level of independence of first-year residents, the competency threshold was set at 3 out of 5 on all stations. This indicates an entrustment decision of indirect supervision required (Table 2), and was determined a priori by expert consensus with input from 2 authors (A.K.H. and T.C.). While individual QSAT domain scores were completed by examiners, these scores did not contribute to the summative competency decision.

Table 2.

Description of Entrustment Decisions^a

A generalizability (G) study was conducted to determine the stability of residents' entrustment scores and the magnitude of the identified sources of variance. With separate examiners for each station, our analysis was limited to a 2-facet, person-by-item design, with item denoting station/examiner. The G-study was performed using SPSS version 22 (IBM Corp, Armonk, NY), and confirmed independently by a second researcher using GENOVA (generalized analysis of variance). Phi coefficients were calculated as a measure of score dependability, and a decision study was conducted.

Results

A total of 40 residents participated in the Nightmares Course in the 2015–2016 academic year. Our hospital's Rapid Assessment of Critical Events team database identified the following as the most common reasons for calls to inpatient units: decreased level of consciousness, shortness of breath or hypoxia, tachycardia, hypotension, and clinical concern on the part of the patient's nurse. After reviewing this list, study authors (L.M. and T.C.) generated 6 session topics that formed the Nightmares Course curriculum (provided as online supplemental material).

Six 4-week sessions were held between July 2015 and March 2016. The majority of residents participated in at least 4 of 6 sessions prior to the OSCE in March (mean = 4.25; range, 0–6). All program directors supported the Nightmares Course and considered it protected academic time for their residents. The primary reason for absence was out-of-town clinical rotations.

A total of 23 of 40 residents (58%) participating in the Nightmares Course completed the OSCE. The frequencies of entrustment scores were calculated for each OSCE station (Figure). Person variance was found to be greater than item variance, but a large portion (51%) of the variance remained unaccounted for (person-by-item interaction and error; Table 3). The use of 4 stations/examiners yielded acceptable dependability (phi = 0.733).

Figure.

Frequency of Entrustment Scores Achieved by Resident Physicians

Abbreviations: VT, ventricular tachycardia; STEMI, ST-elevation myocardial infarction; LOC, level of consciousness

Table 3.

Estimated Variance Components and Decision Study

Of the 23 residents, 8 (35%) scored less than 3 on at least 1 OSCE station and therefore did not meet the competency threshold. These 8 residents participated in the subsequent remediation sessions that ran from March through June 2016. The response rate of the post-OSCE questionnaire was 23 of 23 (100%). The comments were uniformly positive; residents indicated that they found the examination to be fair, a valuable learning experience, and representative of their clinical ability.

Each weekly session cost approximately $240 for human resources, and there were a total of 44 weekly sessions, yielding a total curriculum cost of $10,560. The multi-station OSCE required 4 lab technicians and 6 registered nurses for a full 9-hour day, as well as a half-day preparatory session, contributing to a total approximate cost of $3,900 for the OSCE (Table 4).

Table 4.

Financing for the Nightmares Course and Objective Structured Clinical Examination (OSCE)

Discussion

The Nightmares Course was valued by residents and faculty from all programs as it provided an opportunity to teach and assess residents using direct observation, frequent formative feedback, and a summative OSCE. Most importantly, this program identified residents requiring additional remediation and directed their further training.

Most of the literature pertaining to transitional curricula (or boot camps) in postgraduate medical education describes the preparation of residents with basic technical skills required to begin surgical programs.^9–11 Nishisaki et al¹² described a resuscitation-focused boot camp for 22 pediatrics critical care fellows, employing a 2.5-day course with both didactic and simulation-based methods, with no assessment of residents. Pliego et al¹³ described a boot camp for 23 obstetrics and gynecology residents involving 4 high-fidelity simulation scenarios with no assessment, demonstrating improved perceived confidence. There are several examples of crisis resource management curricula; however, these courses are discipline specific, provide mainly formative feedback, and are not described as transitional curricula.^14,15

Several attributes make the Nightmares Course a novel intervention. It is a longitudinal course that takes place over the academic year, and it reiterates a standard approach to acutely unwell patients. Repetitive practice distributed over time is recognized as an important ingredient for learning.¹⁶ Most transitional curricula are short in duration, with the longest reported boot camp in a recent meta-analysis taking place over a 7-week period.³ The course also provides a summative assessment in the form of an entrustment decision on the course EPA, which is not reported in any of the published curricula as previously mentioned.³ The course is interdisciplinary and addresses an EPA common to residents providing inpatient care. Finally, remediation is built into the curriculum by design, with the infrastructure in place to continue weekly remedial sessions following the summative OSCE.

The G-study results for the entrustment scores in our OSCE demonstrate that resident variance (41%) was much larger than station/examiner variance (8%), indicating that the scores were sensitive in differentiating resident performance. However, there was a large portion of unaccounted variance, suggesting that additional factors were contributing errors to the performance rating. Separating examiners from stations to develop a fully crossed design would help to account for a larger percentage of the observed variance.

The course was well attended and rated highly by residents. With 8 faculty contributing to weekly sessions, the teaching burden was manageable, and all faculty volunteered to continue teaching the course in the upcoming year.

There are several limitations. The cost for the Nightmares Course and the OSCE was substantial, estimated at $14,760. Participation in the OSCE was optional and low, and this may have biased the results in favor of competent performance; the small number of OSCE stations was also a limitation. The use of nonblinded examiners may have resulted in potential assessment bias. Finally, the use of a single entrustment score to inform the summative assessment decision, instead of individual domain scores, could be viewed as a limitation in the assessment process.

Future work will be required to ensure that competency is translated into the hospital setting and, ultimately, improved patient-level outcomes. Moving forward, we will also evaluate the implementation and effectiveness of similar curricular designs at other centers for similar EPAs.

Conclusion

The Nightmares Course is a novel, competency-based, educational intervention and assessment program in resuscitation medicine applicable to a majority of postgraduate medical training programs. This course was feasible and supported by residents and faculty at our institution. The entrustment-based assessment detected several residents not meeting a minimum competency threshold, and directed their further remediation.