Abstract
Background
Since 2015, development of competencies by emergency medical services (EMS) fellows have been evaluated using the EMS Milestones 1.0 developed by a working group consisting of relevant stakeholders convened by the Accreditation Council for Graduate Medical Education (ACGME). Feedback from users and data collected from the milestones assessments in the interim indicated a need for revision of the original milestones. In May 2021, the Milestones 2.0 working group was convened for the purpose of revising this specialty‐specific assessment tool.
Methods
A working group consisting of representatives from American Board of Emergency Medicine, the Review Committee for Emergency Medicine, and volunteers selected by the ACGME Milestones Committee, chaired by the ACGME vice president for milestones development, was convened using a virtual platform to revise the milestones and develop a supplemental guide for use along with the Milestones 2.0. There were no in‐person meetings of this working group due to the COVID‐19 pandemic.
Results
Data from milestones reporting, discussion within the working group, stakeholder input, and public commentary were used to revise the original milestones. A new supplemental guide to enhance milestone usability and provide recommended resource materials was also developed for use alongside the milestones.
Discussion
The EMS Milestones 2.0 and accompanying supplemental guide provide an updated framework for fellowship programs to use as a guide for developing the competencies necessary for independent practice as EMS physicians and in the formal, competency‐based evaluation of trainees as required by the ACGME.
Keywords: competency‐based assessment, emergency medical services, emergency medical services fellowship, graduate medical education, milestones
INTRODUCTION
The Milestones 1.0 project
The Accreditation Council for Graduate Medical Education (ACGME) began the process that has led to the Milestones 2.0 that are introduced in this paper nearly 30 years ago. In the late 1990s, the ACGME initiated the transition to the assessment of training outcomes of physicians in graduate medical education training programs. This led to the adoption of the six core competencies of patient care (PC), medical knowledge (MK), systems‐based practice (SBP), practice‐based learning and improvement (PBLI), professionalism (PROF), and interpersonal and communication skills (ICS) in 1999. 1 , 2 In 2001 the Outcomes Project was launched, and in 2008 Dr. Thomas Nasca, the ACGME CEO, wrote about “the next step in the outcomes‐based accreditation process” in the ACGME Bulletin. 3 During 2013 the ACGME initiated the next accreditation system (NAS), which included the first version of the milestones, marking the shift from a process‐based assessment to one that is competency based. The milestones grade competence along a five‐stage model that is consistent with the Dreyfus framework of expertise development and consists of “novice,” “advanced beginner,” “competent,” “proficient,” and “expert.” 4 Achieving a “proficient” level is considered a graduation goal, not a requirement, and achieving a level of “expert” is considered aspirational. 1 , 4
As part of the milestones process, each specialty created their own subcompetencies within the six core competency domains 5 . MK and PC will necessarily have considerable variability across training programs in various medical specialties. However, a high degree of variability was also found across the specialties with respect to the remaining milestones of ICS, PBLI, PROF, and SBP. The first set of milestones for the emergency medical services (EMS) subspecialty was adopted in 2015. 6
At the inception of the milestone process, it was understood that this would be an “iterative process, and that a periodic review and revision of the milestones would be beneficial, as experience and research evidence accrued.” 7 As a part of this “iterative process” the ACGME milestones staff conducted more than 200 meetings and visitations to accredited programs to gather feedback on the implementation of the milestones. Information was gathered from stakeholders, focus groups, and the ACGME second milestones summit held in 2016. The feedback on the milestones gathered in these settings and milestones reporting by programs, served as the basis for the development of the Milestones 2.0 revision process.
Challenges and limitations of Milestones 1.0
Based on the information and data collected during this process, Edgar et al. 7 conducted a review of the ICS, PBLI, PROF, and SBP competencies. They found that stakeholders frequently commented on the variability across the specialties and on the significant overlap seen in core competency domains. As an example of this variability, they found that PROF, ICS, PBLI, and SBP were described in 230, 176, 171, and 122 different ways respectively across the 26 specialties leading to American Board of Medical Specialties certification and the transitional year. For instance, there is no fundamental reason that the ICS assessed in an EMS fellowship program should be any different than those assessed in any other fellowship program. Stakeholders reported that this variability was dissatisfying and an impediment to the development of more consistent assessment tools and faculty development programs that could be shared across multiple specialties. 7
To reduce this variability across the milestones in these four competencies, the ACGME brought together multispecialty groups of medical educators who developed two or three subcompetencies for each of the ICS, PBLI, PROF, and SBP categories. These subcompetencies were further refined, including a period of public comment followed by revision to reflect consideration of these concerns. The result of this process is the “harmonized milestones” that served as the starting point for the Milestones 2.0 project. 7 An excellent explanation of this process is presented by McLean et al. 8
METHODS
The Milestones 2.0 process
In May 2021 the ACGME convened a working group to revise the EMS Milestones 1.0 based on updated specialty information, including the 2019 EMS core content, data reported to the ACGME and collected via a survey to program directors, and feedback that had been collected since their inception. This group consisted of 12 members selected by ACEP, the ACGME review committe, and volunteers chosen by the ACGME for the project. The authors were all members of this working group. All meetings were held on a virtual platform. Additional meetings of four‐person groups were also held between the larger meetings to work on tasks such as revision of specific subcompetencies and the development of the corresponding sections of the supplemental guide (described below). Figure 1 outlines the steps in the revision process leading to the revised milestones.
FIGURE 1.
Revision process for the Milestones 2.0 project. Figure adapted from the slideshow “Welcome to the Milestones: An Introduction.” Copyright 2022 ACGME, Chicago, IL.
More specifically, the group participated in a modified Delphi process to identify knowledge, skills, and behaviors that would be important to a graduate in 2025. Through multiple rounds of review, the group identified four PC, two MK, and one SBP subcompetencies specific to EMS. These were added to the harmonized milestones used by all specialties. Once the subcompetencies were identified, the group had facilitated discussions to draft the language of the developmental trajectories within each subcompetency. Additional edits were made in small groups that met asynchronously and were then reviewed and approved by the larger group.
The supplemental guide was developed in the same manner: work on the supplemental guide was originally facilitated with the entire group developing the various components, followed by small‐group work, and finalized via large‐group review and approval. Once the milestones were revised and the supplemental guide had been written, these documents were posted for a period of public comment. Eighteen responses to the public comment survey were received with few specific comments. These responses were considered during a full‐group final review of the documents and did lead to minor changes to the original version. The final version of the Milestones 2.0 and the accompanying supplemental guide were published for use effective July 1, 2022.
RESULTS
What has changed in Milestones 2.0?
The six core competencies of MK, PC, ICS, PBLI, PROF, and SBP remain unchanged in Milestones 2.0. However, new subcompetencies have been added, while others have been removed or combined to reduce redundancy (Table 1). There are now 19 subcompetencies compared to the 14 in Milestones 1.0. In the 1.0 version, there were many milestones that were not part of a complete trajectory, therefore not developmental, and were often only in one or two levels. For Milestones 2.0, those that did not have a complete trajectory across competency levels have been eliminated or combined with others and each learning trajectory now spans from Level 1 to Level 4 or 5, reflecting the development of fellow from a novice, or entry‐level fellow (Level 1), to one who is proficient (Level 4), or an expert (Level 5). Language has been changed to be affirmative, emphasizing what our fellows can do and not focusing on those milestones that have not yet been achieved. A comparison of the Milestones 1.0 and 2.0 for PC3 is shown as an example in Figure 2.
TABLE 1.
Comparison of the Milestones 1.0 and 2.0 subcompetencies
| Milestones 1.0 | Milestones 2.0 |
|---|---|
| PC1: Procedures Performed in the Pre‐Hospital Environment | PC1: Procedures Performed in the Pre‐Hospital Environment |
| PC2: Pre‐Hospital Recognition and Stabilization of Time/Life‐Critical Conditions | PC2: Pre‐Hospital Recognition and Stabilization of Time/Life‐Critical Conditions |
| PC3: Recognition and Treatment of Pre‐Hospital Conditions | PC3: Recognition and Treatment of Pre‐Hospital Conditions |
| PC4: Mass Casualty and Disaster Management | PC4: General Special Operations in Emergency Medical Services (EMS): Mass Casualty and Disaster Management |
| PC5: General Special Operations in Emergency Medical Services (EMS) | PC4: General Special Operations in Emergency Medical Services (EMS): Mass Casualty and Disaster Management |
| MK1: Medical Oversight | MK1: Medical Oversight |
| MK2: Special Teams | |
| SBP1: EMS Personnel (Supervision and Training) | SPB5: EMS Personnel (Supervision and Training) |
|
SBP2: Systems Management |
SBP1: Patient Safety SBP4: Physician Role in Health Care Systems |
| SBP3: System Navigation for Patient‐Centered Care | |
| PBLI1: Quality Management | SBP2: Quality Improvement |
| PBLI2: Research | PBLI1: Evidence‐Based and Informed Practice |
| PBLI2: Reflective Practice and Commitment to Personal Growth | |
| PROF1: Ethics and Professional Behavior | PROF1: Professional Behavior and Ethical Principles |
| PROF2: Accountability | PROF2: Accountability/Conscientiousness |
| PROF3: Self‐Awareness and Well‐Being | |
| ICS1: Team Communications and Management | ICS2: Interprofessional and Team Communication |
| ICS2: Patient‐centered Communications | ICS1: Patient‐ and Family‐Centered Communication |
| ICS3: Communication within Health Care Systems |
Note: From “The Emergency Medical Services Supplemental Guide,” March 2022. Copyright ACGME, Chicago, IL. Used with permission of the Accreditation Council on Graduate Medical Education.
FIGURE 2.
Comparison of PC3 between Milestone 1.0 (blue) and Milestone 2.0 (green). Milestones with one or two levels have been eliminated and trajectories have been revised to carry through to at least Level 4. Language has also been revised to be affirmative in nature. The levels also now relate to the proceeding and following levels as well. From “The Emergency Medical Services Milestones,” July 2022. Copyright ACGME, Chicago, IL. Figure used with permission of the Accreditation Council on Graduate Medical Education.
After revision, PC has been reduced from five to four subcompetencies. The original PC4 and PC5 have been combined into PC4, which now covers general special operations and mass casualty/disaster topics. PC1, 2, and 3 are essentially unchanged from Milestones 1.0.
MK has seen the addition of MK2, special teams. The committee felt that we should separate out these specialized teams that our fellows interact with, such as tactical, high angle, water, confined space, and vehicle rescue teams as very specialized knowledge is needed to function as a member, or to provide medical direction to, these teams. Exposure to these teams also varies widely across fellowship programs.
SBP has undergone significant revision as part of the harmonization process (see below). The former SBP1 and 2 have now been expanded to five subcompetencies. SBP1 is essentially unchanged except that it is now SBP5. SBP2, system management, is now covered by SBP1 and 4, patient safety and the physician role in health care systems. SBP3 delineates milestones for the evaluation of the patient through the health care system. The new SBP2, quality improvement, is now aligned with PBLI1, quality management.
PBLI remains with two subcompetencies in this revision of the milestones although the focus of the subcompetencies has changed. The old PBLI1, quality management, is now replaced by SBP2, quality improvement, as mentioned above. The new PBLI1 now examines evidence‐based and informed practice. PBLI2, formerly research, is now reflective practice and commitment to personal growth.
PROF has been expanded to three subcompetencies in Milestones 2.0. The new subcompetency, PROF3, self‐awareness and well‐being, was added. As is noted in the description of the subcompetency, the intent is not to assess the well‐being of the fellow but to ensure that the fellow has “the fundamental knowledge of factors that affect well‐being, the mechanisms by which those factors affect well‐being, and available resources and tools to improve well‐being”(from the ACGME Milestones 2.0 description).
Finally, ICS has been expanded from two to three subcompetencies. ICS1 and 2 are essentially unchanged while ICS3, communication within health care systems, has been added with the intent of ensuring that fellows are able to communicate effectively using a variety of methods.
The supplemental guide
An integral part of the Milestones 2.0 project has been the development of the accompanying supplemental guide. These guides were developed by the milestones work groups and are meant to serve as a resource document for residency and fellowship leadership, faculty, and members of the clinical competency committees who are not necessarily faculty or physicians. The supplemental guide contains the intent for each of the subcompetencies as well as practical examples of an activity that meets the particular milestone. The guide also provides assessment models or tools and additional notes or resources that can be used by the faculty or members of the program clinical competency committee for further explanation. Figure 3 shows an example of the supplemental guide for MK2, special teams.
FIGURE 3.
An example from the supplemental guide showing the explanation of MK2 special teams. From “The Emergency Medical Services Supplemental Guide,” March 2022. Copyright ACGME, Chicago, IL. Figure used with permission of the Accreditation Council on Graduate Medical Education.
The supplemental guide is designed to assist program leadership and faculty in the development of a framework for assessment of their learners that is applicable to their specific program. As each fellowship program has different opportunities for their respective fellows it is sometimes difficult to determine if the fellow has met a particular milestone. For instance, MK2, special teams, provides examples of a number of these special teams that the EMS fellow could interact with to fulfill this milestone as each fellowship may have access to only a few of these resources.
Given the unique experiences and opportunities of each program it will be necessary for the leadership, faculty, and the members of the clinical competency committee to all review and discuss the new milestones and decide how the fellow will be evaluated within this new framework. The supplemental guide should be used as a resource during these discussions to ensure that the intent of the subcompetency is being met even if the example that is offered does not precisely align with the opportunities offered within the program.
DISCUSSION
EMS, like many other medical specialties, is in a period of rapid change. External pressures, such as the COVID‐19 pandemic, have caused unprecedented stress not only on EMS systems but also on the entire health care system. This is additive to the many preexisting stressors within the prehospital system such as a shrinking workforce, inadequate pay, and the psychological strain of working in a potentially dangerous environment. 9 , 10 , 11 , 12 Taken together, these pressures have accelerated change in the traditional EMS systems that we knew only a decade ago. Alternative care destinations, community paramedicine, and provider health and wellness have become more prominent topics of discussion recently among providers and EMS leadership.
How then does this upheaval relate to our fellows and this revision of the milestones? The goal of a training program is to prepare graduates to not only perform highly complex technical skills but to also understand the systems within which care is delivered and how EMS integrates into and enhances those systems. In our subspecialty, we interact regularly with prehospital professionals who provide care on the streets and in the homes within the communities they serve. Our fellows work alongside these providers during their training. Many of us do the same in our roles as EMS medical directors or members of specialized teams in our communities. This new version of milestones emphasizes the continuity of care of patients from the activation of EMS to disposition from the emergency department and the role of our trainees in that process as well as how factors such as environment and interdisciplinary collaboration affect the provision and supervision of prehospital care. Also included are tools to assess the competence of our fellows in educating providers and performing quality improvement audits and their familiarity with prehospital care that exists outside of traditional 911 activation such as mobile integrated health, disaster planning, and mass gathering types of events. Functioning within systems and the transitions of care that take place during a patient's health care experience have been emphasized. The ability of the fellow to assess factors affecting wellness and professional well‐being has now been included. The milestones have been refined to allow us to more easily assess the journey of our fellows along that pathway from new learners to EMS physicians ready for the independent practice of prehospital medicine. As training in medicine has begun its journey to a competency‐based education model, the milestones are a first step toward identifying what those competencies may look like. These milestones are another step along that pathway.
With change comes uncertainty, and the new milestones will undoubtedly create challenges. The faculty and clinical competency committee will need to thoroughly review the new milestones and the supplemental guide to ensure that they are operating from the same shared mental model. As it is a new document, the supplemental guide will require more extensive review to understand the intent of the harmonized milestones and of the edited specialty‐specific PC and MK competencies. There will certainly be questioning of why some subcompetencies were included and others were not by those using the milestones and this is unavoidable. However, the supplemental guide is intended to specifically address the intent of those subcompetencies that were included and possible means of assessment as a starting point for programs to ultimately decide on how they will assess the fellows within the unique environment of that program. Future versions of the milestones must take into account the changing role of the physician in the provision of prehospital care and continue to provide a competency‐based framework within which we are able to assess the progression of our fellows through this complex system within our individual training programs. Involvement of stakeholders, learners, the public, and teaching faculty in a continuous process of review and improvement of our assessment methods will help to ensure that we are training fellows who are ready to meet the challenges of providing high‐quality, patient‐centered care whenever and wherever we are called upon to do so.
LIMITATIONS
As was seen with the development of the initial milestones, there were limitations that the working group has sought to address with these new milestones. Heterogeneity in the six core competencies and the subcompetencies was seen as a major limitation of the Milestones 1.0. The ACGME and work group has sought to address this as described above but, undoubtedly, new inconsistencies will arise.
Any document produced by a working group is necessarily the product of the makeup of the group as well as the give and take of that group of participants at that particular time. The ACGME sought to ensure that the working group was representative in that members were recruited from many sources and that nonphysician members were included in the group, though no group of this size could be perfectly representative of the field of EMS. Additionally, public comments were sought before the publication of the final document. Although the number of public comments received was small, minor changes were made to the final document based on these comments.
CONCLUSIONS
The field of emergency medical services medicine is continually evolving and this new version of the milestones provides program directors, clinical competency committees, core faculty, and associated fellowship faculty an updated competency‐based framework that will be used to evaluate progress of fellows on the path from new learners to proficient or expert emergency medical services physicians. Emergency medical services medicine will continue to change as the health care system itself changes, pushed and pulled by internal and external forces. In a few years, Milestones 2.0 itself will come under revision and will serve as the basis for a new system of evaluation of our fellows. As we use this new tool we will find the good and the not so good and use it to determine more effective and meaningful processes for the evaluation of our fellows as the field of emergency medical services medicine and the delivery of prehospital care continue to evolve.
AUTHOR CONTRIBUTIONS
All authors contributed to the review and revision of the milestones over the course of multiple meetings and additional small‐group meetings. Chadd E. Nesbit drafted the original manuscript. Joshua D. Mastenbrook, Matthew T. Ball, Kathy J. Rinnert, and Laura Edgar provided critical revisions of the manuscript through multiple drafts. Laura Edgar also provided additional technical and administrative support for the project.
CONFLICT OF INTEREST
Laura Edgar is employed by the Accreditation Council for Graduate Medical Education. The other authors declare no potential conflict of interest.
ACKNOWLEDGMENTS
The authors thank Sydney McLean and Braden Harsy for their excellent administrative support of the workgroup during the entire revision process. We also thank Sydney McLean for additional support during the preparation of the manuscript. We also wish to express our gratitude to our colleagues on the Milestones 2.0 workgroup for the generous contribution of their time and expertise during this process.
Nesbit CE, Mastenbrook JD, Ball MT, Rinnert KJ, Edgar L. Emergency medical services Milestones 2.0: What has changed? AEM Educ Train. 2022;6:e10821. doi: 10.1002/aet2.10821
Supervising Editor: Dr. Teresa Chan
REFERENCES
- 1. Edgar L, McLean S, Hogan SO, Hamstra S, and Holmboe ES. The Milestones Guidebook, version 2020. Copyright 2020, Accreditation Council on Graduate Medical Education, Chicago, IL. Accessed 8/4/2022 at https://www.acgme.org/globalassets/milestonesguidebook.pdf
- 2. Swing SR. The ACGME outcome project: retrospective and prospective. Med Teach. 2007;29(7):648‐654. doi: 10.1080/01421590701392903 [DOI] [PubMed] [Google Scholar]
- 3. Nasca TJ. The CEO's first column–the next step in the outcomes‐based accreditation project. ACGME Bull. 2008;5:2‐4. [Google Scholar]
- 4. Batalden P, Leach D, Swing S, Dreyfus H, Dreyfus S. General competencies and accreditation in graduate medical education. Health Aff (Millwood). 2002;21(5):103‐111. doi: 10.1377/hlthaff.21.5.103 [DOI] [PubMed] [Google Scholar]
- 5. Swing SR, Beeson MS, Carraccio C, et al. Educational milestone development in the first 7 specialties to enter the next accreditation system. J Grad Med Educ. 2013;5(1):98‐106. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Accreditation Council for Graduate Medical Education , Emergency Medical Services Milestone Pilot, July 15, 2015. L. Edgar, personal communication, Accessed May 5, 2021.
- 7. Edgar L, Roberts S, Holmboe E. Milestones 2.0: a step forward. J Grad Med Educ. 2018;10(3):367‐369. doi: 10.4300/JGME-D-18-00372.1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. McLean S, Edgar L, Harsy B. The milestones 2.0 development process. J Grad Med Educ. 2021;13(2 Suppl):4‐7. doi: 10.4300/JGME-D-20-00909.1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Edgar L, Roberts S, Yaghmour NA, et al. Competency crosswalk: a multispecialty review of the accreditation Council for Graduate Medical Education Milestones across Four Competency Domains. Acad Med. 2018;93(7):1035‐1041. doi: 10.1097/ACM.0000000000002059 [DOI] [PubMed] [Google Scholar]
- 10. National Association of Emergency Medical Technicians . How COVID‐19 has impacted our nation's EMS agencies. National Association of Emergency Medical Technicians. Accessed July 15, 2022. https://www.naemt.org/docs/default‐source/covid‐19/covid‐impact‐survey‐06‐03‐2021.pdf?sfvrsn=c62fea93_4.
- 11. Mountfort S, Wilson J. EMS provider health and wellness. StatPearls [Internet]. StatPearls Publishing; 2022. [PubMed] [Google Scholar]
- 12. Isakov A, Carr M, Munjal KG, Kumar L, Gausche‐Hill M. EMS agenda 2050 meets the COVID‐19 pandemic. Health Secur. 2022;20(S1):S97‐S106. doi: 10.1089/hs.2021.0179 [DOI] [PubMed] [Google Scholar]