• Education • Simulation-based assessment for the emergency medicine milestones: a national survey of simulation experts and program directors

The focus of outcome-based education has shifted from the process of training to the end “product” of education. This has necessitated a reframing of the teaching, learning, and assessment. Therefore, in 2013, the Accreditation Council for Graduate Medical Education (ACGME) implemented the educational milestones for all accredited residencies and fellowship programs.[1] The milestones have been described as “developmentally based, specialty-specific achievements that residents are expected to demonstrate at established intervals as they progress through training.”[1] Milestones are based on the six core competencies established by the ACGME and American Board of Medical Specialties (ABMS), which consist of medical knowledge (MK), patient care (PC), interpersonal and communication skills (ICS), practice-based learning and improvement (PBLI), professionalism (PROF), and systems-based practice (SBP).[2,3] ACGME, in conjunction with the American Board of Emergency Medicine (ABEM), drafted 23 detailed subcompetencies relevant to emergency medicine (EM).[4]

To address ongoing concerns regarding inconsistencies in the assessment of subcompetencies across specialties, “Milestones 2.0” was introduced in 2021.[3] These changes focused on elaboration or description of some of the vague elements, which represented a planned iterative development process.[3] The current EM milestones (2.0) are meant to “increase transparency of performance requirements”, “encourage informed self-assessment and self-directed learning”, and more importantly, “facilitate better feedback to learner.”[5] Most programs have continued to use the same assessment methods, neglecting the few subcompetencies[6] that are difficult to evaluate using traditional assessment methods.[7] An attractive solution is simulation.[8]

ACGME and ABEM have previously recommended the use of simulation in the assessment of milestones.[9] Simulation has been described as, “a technique- not a technology- to replace or amplify real experiences with guided experiences that evoke or replicate substantial aspects of the real world in a fully interactive manner.”[2,10,11] Simulation-based medical education provides meaningful feedback to learners and measures outcomes based on observational ratings.[12] In addition, simulation provides the opportunity for formative and summative assessment.[13]

Given the importance of accurate and feasible assessment of the milestones, identifying current challenges in resident evaluation will provide opportunities for innovative assessment methods. This study’s primary aim is to identify the EM milestones that are the most difficult to assess using traditional methods and those most suitable for simulation assessment.

The survey was initially drafted based on the original EM milestones and updated to the revised milestones 2.0.[9,14] The survey was piloted on a focus group of core education leadership at our institute, and some of the questions were rephrased for clarity based on their feedback. The Institutional Review Board at University of Maryland School of Medicine determined the study to be exempt (HP-00082381).

The web-based survey for this cross-sectional study contained a total of 12 questions (Supplementary file). Background information regarding the demographics of the affiliated residency program, role of the respondent, and simulation usage in the program was collected. The two key questions focused on selecting the five milestones most difficult to assess using traditional methods and the five milestones most suitable for simulation-based assessments.

EM residency programs with affiliated simulation fellowships were selected as participants for the study. Twenty eight such programs were identified through the Society of Academic Emergency Medicine (SAEM) and Emergency Medicine Residents’ Association (EMRA) websites. [15,16] The residency program and simulation leadership were identified from each department’s website. Contact information was obtained via the department’s official website, social media, mutual contacts, and residency coordinators. The survey was conducted using Survey Monkey (www.surveymonkey.com) and was sent weekly for 6 weeks between May and June 2021.

Demographic data and the total number of votes for each of the 22 EM milestones subcompetencies for each question were summarized using descriptive statistics. The number of votes for “most difficult to assess” and “most suitable for simulation assessment” were counted from both the simulation and education experts. Five subcompetencies were prioritized for each question by virtue of earning the greatest number of votes. All analyses were conducted using Microsoft Excel.

Thirty-eight of the 115 invited respondents completed the survey, for a response rate of 33%. Of the 115 survey invitations that were sent, 12 were not delivered and 1 participant opted out. The geographic location of the respondents’ residency programs was well matched with the national distribution of residency programs, and our respondents were more likely to be from larger programs.[15,16] Simulation-based education was utilized frequently (>5 times per year) by 92.11% of the residencies.[17] The majority of programs (89.47%) reported using simulation for formative assessment, with only one-third (34.21%) using simulation for summative assessment.

The milestone subcompetencies viewed as “most difficult to assess” were: SBP, with its subcompetencies of quality improvements, physician role in healthcare, system navigation for patient-centered care, and patient safety, as well as reflective practice and commitment to personal growth, a subcompetency of PBLI. The milestone subcompetencies viewed as “most suitable for simulation assessment” were emergency stabilization, the performance of a focused history and physical exam and procedures, subcompetencies of the PC. Other subcompetencies include interprofessional and team communication and patient and family-centered communication of ICS (Table 1). A complete list of the ranked subcompetencies is in Supplementary Tables 1 and 2.

Table 1.

Emergency milestone subcompetencies most difficult to assess using traditional non-simulation methods and most suitable to assess using simulation

Our survey of EM residency education experts identified the 5 milestone subcompetencies most frequently viewed as difficult to assess using traditional non-simulation methods, and the 5 most suitable for assessment using a simulation-based tool, with no overlap between the two categories.

Several PC and ICS subcompetencies were identified as suitable for assessment via simulation. These same subcompetencies have been identified previously as suitable for assessment using simulation by EM simulation experts, albeit utilizing the original version of the milestones.[8] Simulation cases with scenario-specific checklists have been successfully utilized for these subcompetencies to evaluate EM residents at different stages of training, with good to excellent inter-rater reliability and item discrimination.[8,18]

The entire SBP milestone was identified as “difficult to assess using traditional method”. ACGME defines SBP as “demonstrating an awareness of and responsiveness to the larger context and system of healthcare, as well as the ability to call effectively on other resources in the system to provide optimal health care.”[19] This systems-based approach requires healthcare clinicians to work collaboratively to provide high-quality patient-centered care while minimizing the associated costs and risks.[20] However, minimal importance is given to understanding the complex healthcare system during training.[21] Simulation cases that emphasize SBP issues such as out-of-hospital care, appropriate consultation and disposition, and multi-tasking/team management, accompanied by checklists of SBP subcompetency-specific criteria to guide the debriefing can play a pivotal role.[18,22] Case-based scenarios which pose questions to dissect the aforementioned constructs and identify barriers, supports, drivers, and constraints are another form of simulation that has been described for teaching systems-based thinking.[20]

The element of SBP that is identified as most difficult to assess is quality improvement (QI). Experiential learning has been identified as a best practice feature for high-quality QI education.[23,24] The Quality Improvement Knowledge Application Tool (QIKAT-R) has been found to be useful in assessing QI knowledge.[25,26] Simulation scenarios developed for teaching various aspects of QI, such as that published by Worsham et al,[27] can be utilized.

Patient safety is another important focus of the SBP milestone, which encompasses knowledge of and participation in the analysis of patient safety events and the development of prevention strategies. Purposefully engaging residents in existing departmental reporting structures and processes to address patient safety provides them with valuable learning and subsequent evaluation opportunities.[28] A systematic review focused on quality improvement and patient safety curricula for medical students and resident physicians, with some of the included studies publishing their assessment methods as well.[18,29] It is surprising that patient safety was not frequently identified as “suitable for simulation assessment.” by our surveyed EM residency education experts; the suggestion for using simulation is present in the description of the subcompetency, with elements relating to reporting of safety events through institutional reporting systems as well as participating in and leading safety event disclosures to patients and families.[14] Simulation methods, such as error disclosure and situational awareness, have been extensively described and used for patient safety teaching.^[30–34]

System-navigation for patient-centered care, another SBP subcompetency, focuses on care coordination, transition of care, and health care inequity. Care coordination is simply organizing the patients’ care by interacting with interprofessional teams. Interprofessional simulations have been used to assess and improve collaborative learning and practice among various healthcare workers.^[35–37] Transition of care involves an effective handoff report, which may be assessed by direct observation using the Handoff Communication Assessment tool or the PREP-4Cs.[38,39] Establishment of a longitudinal social EM curriculum can be effective in bridging the education and assessment gap in healthcare inequity.[40] Simulation has been used effectively in educating and assessing health equity via direct observation, critical action checklists, and facilitated debriefing.[41]

The last subcompetency of SBP is physician role in health care systems, which focuses on navigating the components of the healthcare system to provide efficient patient care, patient payment models, and EM business. Navigating the components of the healthcare system is challenging to teach due to the ever-changing nature of the working environment in the ED. This ranges from unavailability of resources throughout the week, to working with different teaching and practice styles of the supervising emergency physician.^[20,42–44] Nevertheless, educators have attempted to include this discussion of navigating the system in previously published simulation scenarios.[18] Emergency Medical Treatment and Labor Act (EMTALA) protects patients’ right to treatment irrespective of patient payment models, which limits patient payment model discussions in clinical practice.[45] Approaches to teaching this subcompetency are not well described.[20,42] The business of EM, is not adequately taught and may be addressed by implementing an administrative educational track and elective.[46,47]

In addition to SBP, reflective practice and commitment to personal growth was identified as “difficult to assess using traditional methods”. Simulation followed by debriefing using the advocacy-inquiry model can provide opportunities for reflective practice for learners.[48] This model can be utilized to provide feedback and gauge insight after directly observing a patient care interaction. ACGME mandates individual bi-annual resident meetings with program leadership which may provide a reasonable opportunity for assessment of a trainee’s reflective practice. Additionally, reflective writing with guidance and evaluation from faculty using a reflective writing rubric can provide opportunities for reflection and growth.[49]

Moreover, subcompetencies that are difficult to assess using traditional assessment methods may be assessed at least partially utilizing simulation. However, this opportunity was not identified by the surveyed EM residency education experts. This finding is in line with previous literature showing that simulation in EM education is most often used for resuscitation-based teaching and high-risk procedures.[50] Creating and utilizing simulation scenarios that incorporate non-knowledge difficult skills in addition to patient-care components would be the first step in utilizing simulation more frequently in the assessment of the milestones. It is essential to adopt an innovative view on the assessment of residents in regard to the milestone subcompetencies, with a focus on tools that are reliable, feasible, and deliver high-yield feedback to the learner.

Our study has several limitations. One of the primary limitations is the response rate, which is a common drawback of the anonymous survey methodology. However, our sample was diverse and broadly representative of the different geographical regions of the country. Another limitation is that our purposeful sampling focused on residency programs affiliated with a simulation fellowship to engage education experts who are more knowledgeable about simulation, which may not be representative of all training programs. The slight under-representation of simulation experts among our respondents may have led to differences in the selection of milestones most suitable for simulation. Selection bias, which is inherent to voluntary surveys, is another limitation.

SBP and reflective practices and commitment to personal growth are difficult to assess using traditional methods. Our current simulation method does not address the challenges faced when evaluating these subcompetencies. Incorporation of innovative simulation views as well as non-traditional evaluation methods could help. Further research on the ability of these assessment methods to accurately identify knowledge gaps and predict clinical competence is needed.