Simulation-Based Assessments and Graduating Neurology Residents' Milestones: Status Epilepticus Milestones

Abstract

Background

The American Board of Psychiatry and Neurology and the Accreditation Council for Graduate Medical Education (ACGME) developed Milestones that provide a framework for residents' assessment. However, Milestones do not provide a description for how programs should perform assessments.

Objectives

We evaluated graduating residents' status epilepticus (SE) identification and management skills and how they correlate with ACGME Milestones reported for epilepsy and management/treatment by their program's clinical competency committee (CCC).

Methods

We performed a cohort study of graduating neurology residents from 3 academic medical centers in Chicago in 2018. We evaluated residents' skills identifying and managing SE using a simulation-based assessment (26-item checklist). Simulation-based assessment scores were compared to experience (number of SE cases each resident reported identifying and managing during residency), self-confidence in identifying and managing these cases, and their end of residency Milestones assigned by a CCC based on end-of-rotation evaluations.

Results

Sixteen of 21 (76%) eligible residents participated in the study. Average SE checklist score was 15.6 of 26 checklist items correct (60%, SD 12.2%). There were no significant correlations between resident checklist performance and experience or self-confidence. The average participant's level of Milestone for epilepsy and management/treatment was high at 4.3 of 5 (SD 0.4) and 4.4 of 5 (SD 0.4), respectively. There were no significant associations between checklist skills performance and level of Milestone assigned.

Conclusions

Simulated SE skills performance of graduating neurology residents was poor. Our study suggests that end-of-rotation evaluations alone are inadequate for assigning Milestones for high-stakes clinical skills such as identification and management of SE.

Objectives

An objective simulation-based assessment was used to evaluate graduating residents' status epilepticus (SE) identification and management skills and how they correlate with Accreditation Council for Graduate Medical Education (ACGME) Milestones reported for epilepsy and management/treatment by their program's clinical competency committee.

Findings

There were no associations between skills performance and ACGME Milestones for epilepsy and treatment/management assigned at graduation.

Limitations

The checklist used for assessment was designed based on the protocols at one institution which may be different from those learned at the participants' institutions; additionally, it is possible that participating residents did not understand the expectations resulting in incorrect decisions during the simulation-based assessments.

Bottom Line

End-of-rotation evaluations are inadequate as the sole data source for assigning Milestones for high-stakes clinical skills; formal simulation assessments should be considered to support graduation level decisions on the Milestones.

Introduction

Residency programs must ensure their trainees are capable of handling a number of medical emergencies before graduation. Status epilepticus (SE) is a neurological emergency with an incidence of up to 41 per 100 000 patients and a 20% mortality rate.¹ Timely and appropriate treatment of SE improves morbidity and mortality.² Yet, a review of multiple studies demonstrated deficiencies in the treatment of SE by neurologists in academic medical centers.³ Treatment is often hindered due to delays in recognition, administration of incorrect first- and second-line anti-seizure drugs (ASDs), and suboptimal doses of correct ASDs.^3–5 Depending on the hospital setting, SE is often managed by physicians from various specialties, including neurology, emergency medicine, internal medicine, family medicine, and intensive care. Therefore, it is imperative that residency programs focus efforts on teaching residents about SE diagnosis and management beyond lecture series and chance clinical experiences.^6,7

In 2015, the American Board of Psychiatry and Neurology formed a joint initiative with the Accreditation Council for Graduate Medical Education (ACGME) to develop Milestones that provided a framework for the assessment of neurology residents.⁸ The Milestones are scored on a 5-point behaviorally based developmental rating scale, with level 1 roughly equivalent to a novice to advanced beginner in neurology and level 5 as an aspirational goal (someone who has been in practice for several years). Level 4 is the targeted level for residency graduation, although this is not a requirement for graduation. Specific Milestones address neurology residents' ability to diagnosis and manage patients with epilepsy and other neurological emergencies. Neurology residents achieve a level 3 for the epilepsy Milestone if they can diagnose and manage convulsive and non-convulsive SE. Residents reach level 4 for the management/treatment Milestone when they can independently direct management of patients with neurological emergencies (such as SE).

Despite guidance on what needs to be assessed at each level of residency, the Milestones do not provide a description for how residency programs should perform assessments. Residency programs use Milestones to evaluate the progression and advancement of their residents over time. They often use end-of-rotation or shift evaluations as a guide to help assign Milestone levels. End-of-rotation evaluations may be unreliable due to variation in opinions by supervising faculty.^9–13 Accurate assignment of Milestone levels require a sizeable number of direct observations by faculty of a vast array of clinical experiences.^14,15 However, this is problematic because end-of-rotation/shift evaluations often rely on chance clinical experiences and lack direct observations of infrequent clinical scenarios.^6,7,14–16 Milestone evaluations of high-stakes clinical skills such as diagnosis and treatment of SE and other neurological emergencies should not be based solely on subjective evaluations made during chance clinical experiences.

We previously developed an SE simulation-based mastery learning (SBML) curriculum at Northwestern University Feinberg School of Medicine.¹⁷ During the SE SBML curriculum, residents are assessed rigorously with a skills checklist on SE diagnosis and management and required to meet or exceed a minimum passing standard (MPS). The checklist helps guide the neurology program director in assessing a resident's ability to achieve the levels 3 and 4 Milestones in epilepsy and management/treatment. The current study has 2 aims: first, to evaluate the simulated SE identification and management skills of graduating (postgraduate year [PGY]-4) neurology residents and how their performance correlated with clinical experience and self-confidence, and second, to compare residents' simulated SE skills with their ACGME level of Milestones attained for epilepsy and management/treatment.

Methods

Settings and Participants

Twenty-one graduating PGY-4 neurology residents were eligible to participate in the study from the 3 neurology residency programs. We assured them that the identity of participants and programs would remain anonymous. Residents were recruited by the program director at each of the 3 sites. None of the programs had provided their graduating residents with prior SE simulation training. All participants were required to provide written informed consent. They received feedback and extra training by participating in these modules.

Resident identifiers were only known by one investigator (Y.M.D.) who provided participant information to the ACGME so they could provide the final level of Milestone assigned to each resident by their program director. Data were deidentified by the ACGME when returned to the study team for analysis.

Procedure

We performed a cohort study of graduating PGY-4 neurology residents from 3 tertiary care academic medical centers in Chicago from May to June 2018. We evaluated residents' skills identifying and managing SE using a simulation-based assessment. Simulation-based assessment performance scores were compared to clinical experience (the number of SE cases each resident reported identifying and managing during residency), their self-confidence in identifying and managing these cases, and their end of residency level of Milestone for epilepsy and management/treatment (assigned by the clinical competency committee [CCC], composed of 3–4 core faculty members and the program director).

Graduating neurology residents came to the Northwestern Simulation Laboratory to perform SE assessments that were previously developed and used in our SE SBML curriculum.¹⁷ The rooms used for the assessment were identical in their setting and were equipped with supplies identical to hospital rooms. The same exact clinical scenario was used for all participants. Participants provided demographic and clinical data, including age, sex, race, number of cases they had personally experienced identifying and managing convulsive and non-convulsive SE, and their self-confidence identifying and managing convulsive and non-convulsive SE.

After briefing the residents on the simulation scenario, including how the Laerdal SimMan 3G high-fidelity simulator worked, room equipment, and expectations, the assessment began. The scenario involved a patient having 2 generalized tonic clinic seizures without return to baseline, and a nurse confederate was in the room to assist. Residents were evaluated by 1 of 2 authors (Y.M.D. or E.G.) on their identification and management of SE using a previously developed 26-item skills checklist.¹⁷ Both raters were trained and calibrated to use the instrument during pilot testing of the curriculum with 7 PGY-3 neurology residents from our institution. The learners were observed and evaluated in real time using a paper printout of the 26-item checklist. At the end of the scenario, residents were provided feedback on their performance, and knowledge gaps were discussed.

The ACGME was then given a list of resident participants and provided the level of Milestones assigned to each resident by their CCCs for epilepsy and management/treatment at the time of graduation. We subsequently compared performance on the simulated scenario, cases performed during residency, self-confidence, and the level of ACGME Milestones assigned.

Measurement

The 26-item checklist was graded dichotomously (1, done correctly; 0, not done/done incorrectly). The order of completion of checklist items needed to make sense clinically. The development and validation of the checklist are described in detail elsewhere.¹⁷ The 2 authors (Y.M.D. and E.G.) who scored checklists for this study had perfect interrater reliability based on checklist data from our previous study (Cohen's kappa = 1.0).¹⁷ The MPS was previously set at 88% (23 of 26) checklist items correct for the SE SBML curriculum.¹⁷ Participant self-confidence identifying and managing convulsive and non-convulsive SE were measured on a 10-point scale (0, not confident, to 10, very confident). Each of the 3 participating neurology residency CCCs assigned a graduating level of Milestone based on end-of-rotation evaluations filled out by faculty and staff. They had no access to the simulated SE assessments scores at the time of the Milestones assignment.

Statistical Analyses

Demographic and clinical data are described using means, standard deviations, and frequency counts. We used Pearson's correlations to compare mean skills checklist performance with the number of cases reported and self-confidence. We used a one-way analysis of variance to compare mean SE skills checklist scores of the participants by level of Milestone reported for epilepsy and management/treatment. Statistical analyses were performed using SPSS 26 software (IBM Corp, Chicago, IL).

The Northwestern University Institutional Review Board has determined that the study meets the criteria for exemption.

Results

Sixteen of 21 (76%) eligible residents participated in the study (4 of 6 eligible residents from site 1; 6 of 7 eligible residents from site 2; and 6 of 8 eligible residents from site 3). Eighteen residents came for the simulated sessions. One declined to participate (from site 1), and one other was excluded because he had already participated in the design of the study (from site 2). Demographic, clinical experience, and self-confidence of residents can be found in Table 1. The majority of graduating residents experienced 8 or fewer clinical episodes where they identified and managed convulsive or non-convulsive SE; one graduating resident experienced no clinical episodes.

Table 1.

Demographics, Clinical Experience, and Self-Confidence of Neurology Resident Participants

Characteristic	N = 16a
Male, No. (%)	9 (56%)
Age, mean (SD)	30.53 (1.51)
Race, n (%)
African American	1 (6)
Asian	6 (38)
White	9 (56)
Experience, No. (%)
Identification of convulsive status epilepticusa
0	1 (6)
1–3	4 (25)
4–8	7 (44)
9+	2 (12)
Management of convulsive status epilepticusa
0	1 (6)
1–3	5 (31)
4–8	6 (38)
9+	1 (6)
Identification of non-convulsive status epilepticusa
0–3	0
4–8	7 (44)
9–14	2 (12)
15+	4 (25)
Management of non-convulsive status epilepticusa
0–3	1 (6)
4–8	7 (44)
9–14	2 (12)
15+	4 (25)
Self-confidence (0, not confident; 10, very confident), mean (SD)
Identification of convulsive status epilepticus	7.88 (1.59)
Management of convulsive status epilepticus	6.88 (1.89)
Identification of non-convulsive status epilepticus	5.94 (1.44)
Management of non-convulsive status epilepticus	6.19 (1.56)

^aN varies in the clinical experience category as some residents elected to not fill all entries of the clinical experience form. All other categories were completed by all residents.

No graduating neurology residents were able to meet or exceed the MPS during testing. The average SE checklist score was 15.6 of 26 checklist items correct (60%, SD 12.2). There were no significant correlations between resident checklist performance and number of cases performed during residency and self-confidence (Table 2). The average participant's level of Milestones for epilepsy and management/treatment were 4.3 of 5 (SD 0.4) and 4.4 of 5 (SD 0.4), respectively. Only one resident achieved a score of 3.5 (below the residency graduation threshold). There were no significant associations between checklist skills performance and level of Milestones assigned at graduation (Table 3). The Figure shows checklist performance of residents at each institution.

Table 2.

Pearson's Correlation Coefficients Between Simulation-Based Skills Checklist Performance and Experience and Self-Confidence

Skills	Correlation Coefficient (rho)	P Value
Experience
Identification of convulsive status epilepticus	0.25	.39
Management of convulsive status epilepticus	0.43	.15
Identification of non-convulsive status epilepticus	-0.34	.26
Management of non-convulsive status epilepticus	-0.30	.32
Self-confidence
Identification of convulsive status epilepticus	-0.22	.42
Management of convulsive status epilepticus	-0.17	.53
Identification of non-convulsive status epilepticus	-0.21	.44
Management of non-convulsive status epilepticus	-0.20	.46

Table 3.

Neurology Residents Achieving Epilepsy and Management/Treatment Level of Milestones at Graduation and Average Simulation-Based Status Epilepticus Skills Checklist Score by Milestone Level (N = 16)

Milestone	Milestone Level				P Value
	3.5	4	4.5	5
Epilepsy
N (%)	1 (6)	6 (38)	7 (44)	2 (13)
Average checklist score (SD)	61.5% (N/A)	60.1% (14%)	57.1% (13%)	69.2% (0%)	.71
Management/treatment-patient care
N (%)	0	6 (38)	6 (38)	4 (25)
Average checklist score (SD)	…	63.5% (10%)	59.6% (14%)	55.5% (14%)	.63

Figure.

Performance on the 26-Item Status Epilepticus Skills Checklist

Abbreviation: MPS, minimum passing standard.

Note: Performance (percent correct) on the 26-item status epilepticus skills checklist by graduating residents (postgraduate year 4) at each institution. Each circle represents an individual resident.

Table 4 shows the performance on each checklist item by graduating residents. Only 11 of 16 (69%) residents properly identified the possibility of non-convulsive SE occurring after convulsive SE. While all residents were properly able to identify a first-line ASD, less than half properly ordered a second-line ASD and only half ordered relevant lab work. No residents ordered a post-ASD administration drug level.

Table 4.

Abbreviated Status Epilepticus Checklist¹⁷ and Number (%) of Each Graduating Neurology Resident Getting Each Specific Item Correct

Checklist Item	No. (%), N = 16
1. Obtains a history	1 (6)
2. States out loud the patient's level of alertness	9 (56)
3. Examines eyes	10 (63)
4. Examines for any focal weakness	12 (75)
5. States that the patient is having a seizure	16 (100)
6. States out loud time of seizure onset	6 (38)
7. Calls for help	13 (81)
8. Repositions patient onto the side	4 (25)
9. Evaluates patient's airway: suction patient	4 (25)
10. Places pulse oximeter	15 (94)
11. Asks RN to provide oxygen if the patient is hypoxic	13 (81)
12. Asks RN to check blood pressure	16 (100)
13. Asks RN to initiate cardiac tele monitoring	13 (81)
14. Performs a brief neurological exam: check if any eye deviation	9 (56)
15. Ensures that the patient has a working IV access	9 (56)
16. Orders first-line ASD (must be given within 5 minutes of seizure onset)	16 (100)
17. Asks RN to check finger stick blood glucose	7 (47)
18. Orders appropriate labs	8 (50)
19. Orders an appropriately dosed second ASD (ordered within 5 minutes from the first ASD)	7 (44)
20. Calls pharmacy or asks RN to communicate the emergent need for ASDs	9 (56)
21. Orders a stat computed tomography scan of the brain	13 (81)
22. Communicates with attending physician/fellow on call to staff the case	6 (38)
23. States out loud the concern for non-convulsive status epilepticus	11 (69)
24. Orders a post-load ASD level	0 (0)
25. Orders an emergent EEG	11 (69)
26. Makes appropriate decision regarding disposition/level of care	11 (69)

Abbreviations: RN, registered nurse; ASD, antiseizure drugs; EEG, electroencephalogram.

Note: The detailed checklist is published elsewhere.¹⁷

Discussion

We identified a significant gap in the skills of graduating residents consistent with prior work showing identification and management of SE is less than ideal.^3–5 To our knowledge, this is the first study to objectively evaluate graduating residents' skills using simulation compared to final level of Milestones assigned at graduation. Our findings are also consistent with our previous work showing graduating neurology residents performed lumbar puncture with highly variable skills.¹⁸ Additionally, and maybe more importantly, we revealed that despite the gap in SE identification and management skills as measured by the checklist, participating residents still received “ready to graduate” level Milestones for epilepsy and management/treatment on end-of-rotation evaluations.

Our study suggests that neurology training programs should not rely solely on end-of-rotation assessments submitted by faculty and staff to ensure residents' competency in high-stakes clinical events such as SE. End-of-rotation evaluations are often based on chance experiences, and it is unlikely that residents are observed frequently in their management of neurological emergencies such as SE.^6,7,14–16 This is evidenced by the variable number of self-reported graduating resident experiences identifying and managing convulsive and non-convulsive SE (Table 3), which did not correlate with their performance measured objectively. This lack of correlation between experience and measured performance is also consistent with many other studies using objective assessments of less commonly observed clinical skills.^19–23 Instead, we recommend that residency programs use competency-based assessment tools for direct observation using simulation for difficult to observe high-stakes events such as SE. Anesthesiology and emergency medicine residency programs have already successfully used objective structured clinical examinations or simulation-based assessments to help assess level of Milestones for clinical events that may be difficult to observe by faculty.^24–27 Additionally, information gathered from the simulated sessions helps programs identify and address gaps in trainee skills and areas for improvement in existing curricula and clinical workflows.

Equally problematic is the relatively moderate to high reported self-confidence identifying and managing SE, which did not correlate with the performance on the simulated assessment. This is also a common finding among studies comparing self-reported confidence and performance measured objectively.^{19–23,28,29} Because learners cannot reliably self-identify gaps in their knowledge, it is unlikely that they will seek further training or clinical experiences. Although we did not directly assess the effect of SBML on metacognitive skills, we believe the process of deliberate practice (a key component of SBML) improves residents' metacognition.³⁰

At Northwestern University Feinberg School of Medicine, we have successfully used SBML to help internal medicine, neurology, and surgical residency program directors guide Milestone assessments for clinical skills, including advanced cardiac life support,³¹ paracentesis,²² thoracentesis,²³ central venous catheter insertion,³² lumbar puncture,¹⁸ and laparoscopic surgery.³³ We also used SE SBML to help evaluate neurology residents in epilepsy and management/treatment Milestones.¹⁷ Similar SBML curriculum can be developed to help evaluate residents' performance in other high-stakes scenarios such as stroke codes, intracerebral hemorrhage, gastrointestinal bleeding, sepsis, and anaphylaxis.

Our study has several limitations. First, it was performed in a relatively small cohort of residents from only 3 academic neurology programs. Therefore, our study was underpowered to detect significant differences for small correlation coefficients. In a post-hoc power analysis, our study had 80% power to detect an rho of 0.65 with 16 participants (P < .05). We would have needed 47 participants to detect an rho of 0.40. However, this should not take away from the finding that no residents were able to meet or exceed the MPS on the skills checklist at residency graduation despite getting high Milestone-level ratings. We did not do an a priori power calculation because we used the entire cohort of graduating residents available for our study. Second, it is possible that participating neurology residents did not understand the expectations resulting in incorrect decisions during the simulation-based assessments. Additionally, the checklist was designed based on the protocols at one institution, which may be different from those learned at the participants' institutions. Although this may affect performance for some skills checklist items, this does not take away from the fact that several graduating residents did not properly identify non-convulsive SE, send appropriate labs, or correctly order second-line ASDs in the management of SE. Finally, residents were only provided one assessment (observation). We recommend using multiple assessments to determine final Milestone levels for residents. It is important to note that Milestones were not designed to be used as assessment tools, but rather to help residency programs evaluate the progress of their residents over time.³⁴

Conclusions

In conclusion, our study suggests that end-of-rotation evaluations are inadequate as the sole data source for assigning level of Milestones for high-stakes clinical skills such as identification and management of SE.