Introduction
In the Next Accreditation System, resident performance and progression assessed through educational Milestones will be used as 1 of a number of quality indicators to evaluate the educational effectiveness of residency programs. The Anesthesiology Milestones will be used to measure the success of individual anesthesiology residents in achieving expected goals of training, and the effectiveness of the residency program will in part be evaluated based on aggregated Milestone performance data.
Box Members of the Anesthesiology Milestone Working Group
Deborah J. Culley, MD, Brigham and Women's Hospital, Chair
Neal Cohen, MD, MPH, MS, University of California, San Francisco
Steven Hall, MD, FAAP, Northwestern University
Catherine Kuhn, MD, Duke University
Loraine Lewis, EdD, RD, Executive Director, Anesthesiology RRC
Linda Mason, MD, Loma Linda University
Steven P. Nestler, PhD, Accreditation Council for Graduate Medical Education Consultant
Rita M. Patel, MD, University of Pittsburgh
Scott A. Schartel, DO, Temple University
Brian Waldschmidt, MD, Resident Member, University of California, San Francisco
Mark Warner, MD, Mayo Clinic
Margaret Wood, MD, Columbia University College of Physicians and Surgeons
Milestone Development History
The members of the Anesthesiology Milestone Working Group were selected by the Residency Review Committee (RRC) for Anesthesiology, with input from the American Board of Anesthesiology (ABA), the Society for Education in Anesthesia, and program directors. The working group included 9 physician anesthesiologists, the executive director of the Anesthesiology RRC, and an Accreditation Council for Graduate Medical Education (ACGME) consultant. Four of the physician members were representatives from the Anesthesiology RRC (1 also served as the designated institutional official of a university-based medical center), 2 were former directors of the ABA, 2 were anesthesiology program directors, and 1 was an anesthesiology resident in the final year of training. An anesthesiologist who did not participate in the development of the Anesthesiology Milestones was appointed to the multidisciplinary expert panel that developed cross-specialty Milestones for interpersonal and communication skills, practice-based learning and improvement, professionalism, and systems-based practice suggested for use by all specialties. Working group members were provided access to Milestone documents developed by several specialties that were a part of the initial stage in Milestone development. These models were used as templates to approach the task of developing anesthesiology-specific Milestones.
The Anesthesiology Milestone Working Group held its first face-to-face meeting in February 2012, at which the ACGME consultant, who had experience working with prior Milestone working groups, provided an overview of the approach ACGME envisioned for the development of specialty-specific Milestones and shared lessons learned from Milestone development in the Phase 1 specialties. Based on the ACGME's recommendation that each Milestone should contain 5 levels of assessment (corresponding to the range from beginner to expert), initial discussions focused on determining if Milestone levels could be correlated with levels of residency training (clinical base year, clinical anesthesia years 1–3, and graduation). After recognizing that in their final form the levels would be independent of strictly time-based markers, the group made the decision that each Milestone level would represent expectations at the end of a point on the educational continuum. As a part of that continuum Level 1 Milestones were set as the expectations at the end of clinical base training. Level 2 Milestones were set to represent training expectations at the end of foundational (nonsubspecialty) clinical anesthesia training. Level 3 Milestones were set to represent the expectations for a resident at the midpoint of training after having been exposed to all subspecialty areas within anesthesiology. Level 4 Milestones were set to represent the expectations for a physician ready to practice anesthesiology independently without supervision. Level 5 Milestones were set to represent advanced characteristics of independently practicing physicians and were considered aspirational goals (a few of which might be achieved by a subgroup of exceptional residents during their training).
After completion of the first draft of the Anesthesiology Milestones, the committee solicited feedback from members of the Association of Anesthesiology Core Program Directors (AACPD) and the Society for Education in Anesthesiology during their 2012 annual meetings. At the AACPD meeting feedback was provided formally through an audience response system that allowed program directors to respond to questions of relevance, importance, and measurability of each Milestone. Following these meetings, the working group edited the Milestones based upon the feedback received. This revised draft was then distributed to all program directors who were asked to provide constructive feedback through an electronic survey tool allowing confidential, individualized feedback from the program directors to the working group. The feedback was reviewed by the committee and used to make additional modifications to the Anesthesiology Milestones.
General Features of the Anesthesiology Milestones
The working group developed Milestones for each of the 6 ACGME competencies over the course of multiple face-to-face meetings and teleconferences. The patient care and medical knowledge Milestones, which are unique to each specialty, were developed first, and generic Milestones developed by a multispecialty panel were used as a starting point for the development of Milestones for the competencies of interpersonal and communications skills, problem-based learning and improvement, professionalism, and systems-based practice.
The working group recognized the potential increase in workload that Milestone evaluation will place on program directors and Clinical Competency Committees (CCCs), and refined the number of Milestones to ensure appropriate assessment of competencies at each level of training, while not making the Milestones so granular that the evaluation process would be too burdensome. In the final version, the number of Milestones was reduced to 25. The working group believed this number provided an appropriate balance between comprehensiveness and practicality. The distribution of the Anesthesiology Milestones across the 6 competencies is shown in the table in this Introduction.
TABLE.
Distribution of Anesthesiology Milestones
Establishing Milestone Validity, Utility, and Practicality
While the working group sought input from the anesthesiology educational community to ensure that the Milestones were appropriate for assessing anesthesiology resident progression, the working group also recognized that the value of the proposed Milestones would benefit from an initial assessment using a pilot study in selected programs. A representative group of 20 anesthesiology programs has been selected to participate in the pilot study. The pilot will use the CCC in each program to assess the residents using the current Milestones. An evaluation tool has been developed for the program directors to provide the working group with feedback on each Milestone. The working group anticipates that the current version of the Anesthesiology Milestones will require modification over time as the specialty gains experience with this evaluation process. Once the final version of the Milestones is incorporated into training program evaluation, the RRC and other stakeholders will need to monitor and modify the Milestones based on experience so that they provide meaningful assessment of the residents, allowing anesthesiology training programs to foster excellence and optimize the educational experience for each trainee.
Envisioned Practical Use of the Anesthesiology Milestones in Evaluating Residents
The goal of the working group's effort was to produce Milestones that will provide program directors and CCC members with a progressive resident assessment tool that more readily identifies areas of strength and areas in need of improvement for each trainee. As Milestone data is accumulated in a department, the performance of an individual resident can be readily compared to peers and the course of progress of individual residents and residency levels in all 6 competencies domains can be evaluated. In addition, the program will have information about resident performance across the entire residency complement that will allow it to identify areas in which to focus educational efforts.
Recommendations for Competency Committee Composition and Functioning
The Next Accreditation System requires that accredited programs appoint CCCs to monitor residents' progress. The CCC is not new to anesthesiology training programs; the ABA has required submission of CCC reports for more than 25 years. The working group recognized that the Milestones represent a different and important approach to assessment of resident performance and progression. Although CCCs have been instrumental in assessing residents in anesthesiology programs, incorporation of Milestones into the assessment process will require modifications of the assessment process, faculty development to understand the Milestones and their implications and changes in the assessment, and communication with residents regarding performance.
Based on the addition of Milestones in the assessment process, anesthesiology programs may have to modify the composition of the CCCs to ensure representation of all subspecialties and faculty involved in all levels of training. The ideal size of the CCC will likely vary by program based on the number of residents and faculty in the program. Regardless of the size of the CCC, faculty development will be the cornerstone in the transition to Milestones as a method for assessing clinical performance and progression through training. The working group recognized that as CCCs gain experience with incorporating Milestone assessments into their decisions, modifications of assessment strategies may be required. In addition, some Milestones may be difficult to assess in the traditional clinical environment because of the changing clinical environment and case distribution. Not all residents will be presented with the same clinical case experiences; therefore, a resident may not have an opportunity within the clinical environment to demonstrate some of the characteristics encompassed within the Milestones. The use of simulation, standardized patients, and other nonclinical assessments will likely grow in importance as methods for evaluating resident performance in some areas. The Milestone process will be helpful in assessing resident progression during training and the need for remediation when a resident is not progressing through a Milestone. At the same time, the Milestone process may be helpful in evaluating the effectiveness of and role for these alternative training methods in identifying areas of weakness.
The adoption of Milestones in resident assessment will advance the journey toward competency-based graduate medical education that began with the Outcomes Project. Yet with this advance comes additional challenges. Once well-defined Milestones are fully integrated into residency education, it will be necessary to reconcile competency-based end points of training with the variable rate at which Level 4 Milestones will be reached by individual residents. Resolving this dilemma will provide opportunities for innovation in future training paradigms.