Graduate medical education (GME) in psychiatry, like other medical specialties, has been transitioning to competency-based training and assessment. Competency-based medical education was born from a desire to certify physicians based on training outcomes, rather than training inputs such as the amount of time one spends in training [1]. The transition to a focus on training outcomes has been at least 25 years in the making. In 1994, The Accreditation Council for Graduate Medical Education (ACGME) began efforts to determine the expected competencies of physicians-in-training and to introduce methods of assessing training outcomes [2]. From an initial list of thirteen competency domains, the ACGME outcome project advisory committee identified six general core competencies thought to be common to physicians training across all specialties [3]. The core competencies of patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice were approved by the ACGME board in 1999 [4].
The next phase of competency-based GME was to involve “translation of the core competencies into specialty-based competencies” [5]. Following this, residency programs were expected “to develop instructional and assessment methods for integrating the competencies (into) their curricula” [5]. In order to assist programs with this integration, the ACGME developed an assessment “toolbox” of online evaluation methodologies [6]. A portfolio comprised of documents that chronicled resident transition from novice to competent physician was seen as particularly promising. However, as reported by Lurie et al., these portfolios were not straightforward to interpret and peer-reviewed literature found no evidence that assessment tools could directly assess the six core competencies [7].
It was against this backdrop that the ACGME began to focus on the developmental aspects of the acquisition of knowledge and skills by physicians-in-training, a project named the Milestones. The ACGME defines Milestones as “specific behaviors, attributes, or outcomes in the six general competency domains to be demonstrated by residents during residency” [5]. The Milestones are an integral part of ACGME’s Next Accreditation System (NAS) and one way in which the ACGME is trying to redevelop the standards for GME [8]. Under this system, the residency program’s Clinical Competency Committee (CCC) is charged with determining each resident’s level of performance on each subcompetency, often by using resident evaluations or other assessment tools to generate semi-annual reports of Milestone attainment. In other words, the CCC and program director first decide how the Milestones are assessed and then, through a “data-anchored, conversation-based iterative process,” assign subcompetency scores [9].
The original Milestones for general psychiatry residency programs were developed over a two-year period by representatives from the American Board of Psychiatry and Neurology (ABPN), the American Association of Directors of Psychiatric Residency Training (AADPRT), American Psychiatric Association (APA), and the ACGME. This group drew upon many documents, including the Core Competencies for Psychiatric Practice, the ACGME Psychiatry Program Requirements, and the content outlines for the ABPN Certification Examination in Psychiatry [10]. In the original Milestones, the six core competencies were further subdivided into 22 subcompetencies. Individual milestones within each subcompetency were organized using a five-level framework, with level 1 representing skills and attributes expected of a beginning resident, levels 2 and 3 as intermediate stages, level 4 those of a graduating resident, and level 5 representing aspirational achievements [10].
The Milestones themselves were not intended to exist as a static description of the maturation of a physician’s abilities throughout training. Rather, in 2013 the ACGME committed to reviewing and revising the milestones every three to five years as part of a continuous quality improvement process [11]. The ACGME planned that the Milestones would adapt and evolve as competency-based medical education and assessment research accumulated. In addition, it was anticipated the Milestones would need additional updating as CCC’s developed more practical experience assigning subcompetency scores. Therefore, in anticipation of reviewing the original Milestones, the ACGME Milestones staff and executive staff attended program director meetings and visited programs across all accredited specialties to obtain feedback on the first version of the Milestones [11]. The ACGME also held focus groups, interviewed CCC’s and residents, and held Milestone Summits to obtain as much feedback as possible from those stakeholders who were directly using the Milestones.
The consensus feedback among stakeholders was that the Milestones were too complex [11]. More specifically, the descriptions of the individual milestones were overly lengthy and there were too many subcompetencies. The subcompetencies often described a variety of differing skills across levels of attainment, rather than describing the typical maturation from beginning to advanced resident. In addition, feedback to the Milestones staff suggested dissatisfaction with the lack of uniformity of the Milestones and subcompetencies across specialties. The ACGME Milestones staff concluded from these feedback sessions that the complexity of the Milestones “complicated efforts to share assessment tools across programs and provide comprehensive faculty development across specialties” [12]. In addition, different accredited specialties were taking unique approaches to assessing resident skill development, even when the desired skills were similar in nature. This prevented comparisons across specialties and diminished opportunities for multidisciplinary resident assessment and collaborative faculty professional development.
For psychiatry, a Milestones revision group was convened to develop the Psychiatry Milestones 2.0. The group was charged with the following tasks: (1) to simplify the Milestones by reducing unnecessary subcompetencies; (2) to reduce the complexity of each milestone; (3) to incorporate standardized (Harmonized) Milestones across specialties for professionalism (PROF), interpersonal and communication skills (ICS), systems-based practice (SBP), and practice-based learning and improvement (PBLI) which would allow for the development of cross-specialty assessment tools and comprehensive faculty development; and (4) to develop a Supplemental Guide to provide insights into the intent of the subcompetencies. This paper describes the resulting changes in the Psychiatry Milestones based on this charge, the limitations the group faced with making the changes, and offers recommendations for Psychiatry Milestones 3.0.
The Process for Updating the Milestones
One of the first tasks of the Milestones revision process was for the ACGME to understand data collected regarding the first version of the Milestones. Analysis of non-patient care (PC) and non-medical knowledge (MK) Milestone domains demonstrated wide variation among the 26 specialties, demonstrating more than 230 different ways of describing professionalism (PROF), 171 for practice-based learning and improvement (PBLI), 176 for interpersonal and communication skills (ICS), and 122 for systems-based practice (SBP) [12]. The concerns regarding differences in the non-PC and non-MK related milestones across specialties resulted in the ACGME’s convening four groups of multispecialty and interprofessional content experts to develop Harmonized Milestones that can be used across specialties for ICS, PBLI, PROF, and SBP [13]. The Harmonized Milestones were subsequently made available to the specialty-specific Milestone revision groups.
After reviewing data from Milestones 1.0, the ACGME appointed the Milestones 2.0 Working Group. In order to appoint the working group, the ACGME consulted with key organizations including the AADPRT, the APA, the ABPN, and the AOA. The goal of the working group was to update the Milestones to be consistent with current and foreseeable future psychiatric practice as well as current developments in neuroscience. The working group had broad representation of eleven physicians from academic and community psychiatry residency programs across the country. The group also included one resident physician, one public member, and ACGME staff and executive staff. The makeup of the group provided a blend of expertise and fresh points of view, as only three working group members had participated in the initial Psychiatry Milestones process.
The Psychiatry Milestones 2.0 Working Group met three times over the course of two years, with additional conference calls as needed. The vision of the working group was to ensure the Milestones continued to represent realistic progression of knowledge, skills, and behaviors; meaningfully discriminate between levels of competency; and allow for effective assessment of the subcompetencies. The working group revised each subcompetency with the goal of improving clarity, decreasing redundancy, and decreasing the overall number of subcompetencies. The working group revised all PC and MK subcompetencies and edited the Harmonized Milestones to make them more specific to psychiatry.
Teams of two to three working group members updated each of the subcompetencies to ensure each “developmental” thread across each subcompetency could be more easily interpreted. The entire working group then reviewed and discussed all draft subcompetencies to ensure consensus. In similar fashion, the group divided into teams of two or three members to create the Supplemental Guide. The goal of the Supplemental Guide is to improve clarity by providing precise descriptions and example observable behaviors representing each subcompetency across each developmental level. The working group held conference calls to review progress on the draft Milestones and the Supplemental Guide. The draft of the new Psychiatry 2.0 Milestones was posted for public comment on August 13, 2019 [13]. Additionally, AADPRT conducted their own survey of program directors asking for feedback on the proposed Milestones. The group also sought input from AADPRT, ACGME, the Academy of Consultation-Liaison Psychiatry, and the AADPRT Neuroscience Committee. Further revisions were made based on the feedback received. The final go-live date for Milestones 2.0 was delayed from July of 2020 to July of 2021 because of the disruption caused by the COVID-19 pandemic.
Summary of Changes from Milestones 1.0 to 2.0
Although the overall number of subcompetencies remains similar (21 subcompetencies for version 2.0 versus 22 subcompetencies for version 1.0), program directors will notice several changes intended to increase the utility of the Milestones, as well as their ease of use (Table 1). First, all subcompetencies are now limited to no more than 3 threads, and some have fewer. Second, there are no longer any “orphan” milestones or incomplete threads. In other words, all threads must carry through from level 2 to level 4 or 5. Merging of threads is allowed at level 5 if such merging is a natural progression of the subcompetency’s concepts. Level 1 of a thread is now intended to represent the skillset of an incoming resident. Most, but not all, threads contain a level 1. When a thread lacks a level 1, it is because an incoming resident would not be expected to develop those skills until after training has commenced.
Table 1.
An outline of the macroscopic differences between Milestones 1.0 and Milestones 2.0
| Milestones 1.0 | Milestones 2.0 |
|---|---|
| 22 total subcompetencies | 21 total subcompetencies |
| More than 3 total threads per subcompetency | No more than 3 total threads per subcompetency |
| Incomplete threads that did not carry through from level 1 to 4 | All threads must carry through from level 1 to 4 |
| Footnotes throughout the document; no Supplemental Guide | Footnotes eliminated; addition of the Supplemental Guide |
| Only specialty-specific subcompetencies | Harmonized Milestones standardized across specialties for ICS, SBP, PBL, and PROF |
In addition to the changes to the threads, footnotes are no longer part of the Milestones document. The Milestones are intended to stand alone and be easily understandable without the need for footnotes. Instead, the footnotes from Milestones 1.0 were replaced with the Supplemental Guide in Milestones 2.0. However, the Supplemental Guide is intended to be more than a repository for footnotes. In addition to providing examples of demonstrated subcompetencies, the Supplemental Guide includes a list of potential assessment tools that could be utilized to evaluate each subcompetency, as well as possible curricular resources. The intention behind this change is to aid in faculty and program development while moving toward the creation of a shared mental model of training.
The Psychiatry Milestones 2.0 Working Group was allowed to make adjustments to the Harmonized Milestones to better fit the goals and objectives of psychiatric practice. However, the overarching concept behind the Harmonized Milestones is that the skills inherent to interpersonal and communication skills, professionalism, practice-based learning and improvement, and systems-based practice are largely similar between specialties.
One product of the Harmonized Milestones that drew scrutiny during the period of public commentary was Professionalism 3, the subcompetency on well-being (Table 2). AADPRT, which solicited input via a membership survey, provided the most detailed feedback. Both AADPRT membership and leadership expressed three concerns: (1) that the original draft depicted well-being as solely the responsibility of the resident, rather than reflecting the widely held consensus that well-being is also the responsibility of systems; (2) as a result of the subcompetency, well-being would be “graded” as an educational achievement rather than a measure of variable stress; and (3) that assigning a milestones level on well-being would disincentivize residents from seeking help.
Table 2.
Mapping Milestones 1.0 to Milestones 2.0
By mapping, the table indicates where the subcompetencies are similar between versions. While not necessarily exact matches, the mapped areas include some of the same elements. Not all subcompetencies map between versions 1.0 and 2.0. Adapted with permission from the ACGME
AADPRT’s preference was that the subcompetency on well-being be removed altogether. However, the other medical subspecialties who provided feedback to the ACGME wanted the well-being subcompetency included in order to have grounds to advocate for wellness initiatives in their programs. Thus, AADPRT and the ACGME worked together to change the language so that an individual resident’s responsibility incorporates reasonable measures within that resident’s domain and control. In other words, the individual bears responsibility for recognizing the factors that both positively and negatively affect well-being and for being aware of programs designed to decrease burnout or to effect systemic change [14, 15].
Another subcompetency that drew scrutiny during the period of public commentary was Professionalism 4, the subcompetency on Professional Identity and Development (Table 2). This subcompetency was added by the Psychiatry Milestones 2.0 Working Group in order to recognize the unique professional identity of a psychiatrist. However, public feedback demonstrated that most individuals did not feel this subcompetency was necessary or useful in the assessment of residents, so it was removed.
AADPRT leadership also solicited feedback from its organizational components with content expertise: the Psychotherapy Committee, the Neuroscience Committee, and the Child Caucus—to review the draft of Milestones 2.0 and propose changes. The Child Caucus recommended no significant changes to the original draft of Milestones 2.0. The Neuroscience Committee recommended substantive changes, which were largely adopted for MK3, Clinical Neuroscience, in both the subcompetency and the Supplemental Guide. The Psychotherapy Committee proposed changes to MK4 and PC4, Psychotherapy, which were adopted after some editing for conciseness. Finally, the Education Committee of the Academy of Consultation-Liaison Psychiatry provided feedback on PC6, Clinical Consultation in terms of both developmental progression and wording, as well as on SBP3 (Physician Role in Health Care Systems) and MK2 (Psychopathology). All of the feedback was considered, and some integrated to develop a new draft which was then further edited by the working group and ACGME staff, including the addition of examples to the Supplemental Guide, ensuring that both were up to date and included best practices for teaching and clinical care
Limitations of the Milestones
Given the decision to restrict each subcompetency to three threads with milestones across at least four levels, Milestones 2.0 is significantly narrower in scope than Milestones 1.0. While a limitation, this narrower scope acknowledges that not all content is optimally measured by the Milestones, which are based on demonstrated developmental behaviors. In other words, not everything learned during residency training is part of a developmental progression. ACGME emphasizes that the Milestones are one of several measures that programs should use to assess residents, opening the door for the use of existing tools, like those outlined in the Supplemental Guide, and the development of new ones.
Per ACGME’s intention, the Milestones 2.0 were not developed by content experts and the process did not begin with a comprehensive literature search. Rather the working group was comprised of selected representatives of organizations such as AADPRT and the Psychiatry RRC, and volunteers from a range of programs, including one public member, one addictions fellow, and three individuals who also had certification in child and adolescent psychiatry. They were deemed to represent a broad range of experiences and approaches rather than the potentially vested interest of content experts. Using only content experts also would have limited the diversity of opinions present on the working group. A few members experienced writing Milestones; most did not. This process distinguishes the Milestones from, for example, consensus practice or diagnostic guideline development. These factors meant that for content areas generally taught by experts, for example Psychotherapy and Neuroscience, the working group would have benefitted from earlier consultation with national experts.
What might Milestones 3.0 bring? Experience with the new Supplemental Guide and the Harmonized Milestones will likely result in recommendations for future improvements. The Milestones themselves are likely to reflect changes in health care delivery and the practice of psychiatry, such as the increased use of telepsychiatry and integrated/collaborative care. The evolving evidence base and practice standards will also need to be integrated. However, a challenge with developing Milestones 3.0 will continue to be the lack of well-validated assessment methods to collect data regarding resident performance.
In conclusion, the Milestones 2.0 and 1.0 before them represent a step in the right direction as medicine continues adapting a system of competency-based assessments along a developmental trajectory. Strengths of the Milestones 2.0 include simpler language, less redundancy, fewer subcompetency threads, and more uniformity in how each subcompetency thread progresses along a developmental trajectory. The Supplemental Guide, in particular, will be helpful to programs in understanding the intention behind each subcompetency and gives specific examples of the same.
The process of rewriting the Milestones represented a balance between the perspective of educators who use the Milestones in their programs and that of content experts. For the future, involving content experts earlier in the process might result in a more balanced document to submit for public commentary. Involving leading experts in each area of psychiatric assessment, diagnosis and treatment might also prove helpful. In addition, while data comparing subcompetency assessments across specialties were examined, most of the feedback that led to the changes in Milestones 2.0 was based on consensus feedback from focus group meetings or professional organizations. Although the authors recognize the importance of consensus feedback, the process may have benefitted from more systematically collected and codified data. Finally, developing Harmonized Milestones across medical specialties before the Psychiatry Working Group was formed limited the ability of the working group to have a more comprehensive impact on all of the subcompetencies, raising the question of whether this is the optimal approach. AADPRT and other educational organizations in psychiatry may do well to establish clear structures and processes for feedback on these and other issues as the Milestones 2.0 are implemented.
Acknowledgments
The authors would like to acknowledge Marisa Jones (the University of Kansas) and Sydney Mclean (ACGME) for administrative support.
Funding Information
The work that produced the Milestones 2.0 was fully funded by the Accreditation Council for Graduate Medical Education. This included travel costs for the working group members.
Compliance with Ethical Standards
Disclosures
Jeffrey Aronowitz, D.O., was funded by the ACGME to travel to Milestones 2.0 Working Group Meetings. Adrienne Bentman, M.D., is serving a term as Vice-Chair of the ACGME Psychiatry Review Committee from 2020 to 2022. Lynneice Bowen, M.D., reports non-financial support from the ACGME during her participation on the Milestones 2.0 Working Group. Deborah Cowley, M.D., was funded by the ACGME to travel to Milestones 2.0 Working Group Meetings. Sandra DeJong, M.D., was funded by the ACGME to travel to Milestones 2.0 Working Group Meetings. Lillian Houston, M.D., reports non-financial support from the ACGME during her participation on the Milestones 2.0 Working Group. George Keepers, M.D., served on the American Board of Psychiatry and Neurology Board of Directors from 2011 to 2018, the ACGME Psychiatry Residency Review Committee, the ACGME Psychiatry Milestones Task Force 1.0, and the ACGME Psychiatry Milestones Task Force 2.0. Matthew Macaluso, D.O., has conducted clinical trials research as principal investigator for the following pharmaceutical companies over the last twelve months: (1) Acadia, (2) Allergan, (3) Alkermes, (4) AssureRx/Myriad, (5) Eisai, (6) Lundbeck, (7) Janssen, (8) Neurim, (9) SAGE pharmaceuticals, (10) Suven. All clinical trial and study contracts were with and payments made to the Kansas University Medical Center Research Institute, a research institute affiliated with Kansas University School of Medicine Wichita (KUSM-W). Dr. Macaluso is also a member of the speaker bureau for Janssen pharmaceuticals (esketamine). Dr. Macaluso does not believe these relationships pose a conflict relative to this work.
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Cate O, Billett S. Competency-based medical education: origins, perspectives and potentialities. Med Educ. 2014;48:325–332. doi: 10.1111/medu.12355. [DOI] [PubMed] [Google Scholar]
- 2.Batalden P, Leach D, Swing S, Dreyfus H, Dreyfus S. General competencies and accreditation in graduate medical education. Health Aff (Millwood) 2002;21:103–111. doi: 10.1377/hlthaff.21.5.103. [DOI] [PubMed] [Google Scholar]
- 3.Swing SR. The ACGME outcome project: retrospective and prospective. Med Teach. 2007;29:648–654. doi: 10.1080/01421590701392903. [DOI] [PubMed] [Google Scholar]
- 4.Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. Shifting paradigms: from Flexner to competencies. Acad Med. 2002;77:361–367. doi: 10.1097/00001888-200205000-00003. [DOI] [PubMed] [Google Scholar]
- 5.Ling L, Derstine P, Cohen N. Implementing milestones and clinical competency committees. 2013; 2014
- 6.Green ML, Holmboe E. Perspective: The ACGME toolbox: half empty or half full? Acad Med. 2010;85:787–790. doi: 10.1097/ACM.0b013e3181d737a6. [DOI] [PubMed] [Google Scholar]
- 7.Lurie SJ, Mooney CJ, Lyness JM. Measurement of the general competencies of the accreditation council for graduate medical education: a systematic review. Acad Med. 2009;84:301–309. doi: 10.1097/ACM.0b013e3181971f08. [DOI] [PubMed] [Google Scholar]
- 8.Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system—rationale and benefits. N Engl J Med. 2012;366:1051–1056. doi: 10.1056/NEJMsr1200117. [DOI] [PubMed] [Google Scholar]
- 9.Carter WA., Jr Milestone myths and misperceptions. J Grad Med Educ. 2014;6:18–20. doi: 10.4300/JGME-D-13-00438.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Thomas CR, Keepers G. The milestones for general psychiatry residency training. Acad Psychiatry. 2014;38:255–260. doi: 10.1007/s40596-014-0102-2. [DOI] [PubMed] [Google Scholar]
- 11.Edgar L, Roberts S, Holmboe E. Milestones 2.0: a step forward. J Grad Med Educ. 2018;10(3):367–369. doi: 10.4300/JGME-D-18-00372.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Edgar L, Roberts S, Yaghmour NA, Hunderfund AL, Hamstra SJ, Conforti L, Holmboe ES. Competency crosswalk: a multispecialty review of the Accreditation Council for Graduate Medical Education milestones across four competency domains. Acad Med. 2018;93:1035–1041. doi: 10.1097/ACM.0000000000002059. [DOI] [PubMed] [Google Scholar]
- 13.ACGME Psychiatry Milestones 2.0 Survey, August 2019. Retrieved from: https://www.surveymonkey.com/r/MDTVXYN. Last accessed on 22 Apr 2020.
- 14.Kroth PJ, Morioka-Douglas N, Veres S, et al. Association of electronic health record design and use factors with clinician stress and burnout. JAMA Netw Open. 2019. 10.1001/jamanetworkopen.2019.9609. [DOI] [PMC free article] [PubMed]
- 15.Wolpaw JT. It is time to prioritize education and well-being over workforce needs in residency training. Acad Med. 2019;94:1640–1642. doi: 10.1097/ACM.0000000000002949. [DOI] [PubMed] [Google Scholar]