Psychiatry Milestones 2.0: Using the Supplemental Guide to Create a Shared Model of the Development of Professional Identity and Expertise

In 2013, the Accreditation Council for Graduate Medical Education (ACGME) launched the Milestones for all accredited specialties and subspecialties, with the first use in psychiatry during the 2014–2015 academic year. The Milestones were intended to provide a shared mental model, or common understanding, of the six ACGME core competencies of Patient Care (PC), Medical Knowledge (MK), Systems-Based Practice (SBP), Practice-Based Learning and Improvement (PBLI), Professionalism (PROF), and Interpersonal and Communication Skills (ICS) across and within specialties [1]. Within the core competencies and specialty-specific subcompetencies, individual milestones described knowledge, skills, and attitudes on a developmental trajectory from program entry to graduation, as well as aspirational goals [1, 2]. Thus, the Milestones were intended to provide a developmental progression of competencies, or outcomes, expected of residents or fellows and a basis for assessing trainees as part of competency-based medical education [1–3]. Milestones data, aggregated at the national level, could also assist ACGME Residency Review Committees in specialty-specific quality improvement [4]. The ACGME introduced the Milestones with the intent to review and revise them in 3–5 years as part of a continuous quality improvement process [1].

As the original Milestones were implemented, several concerns were raised. Primarily, they did not meet the intent of a shared mental model across and within programs. Users reported that there were too many milestones and that they were complex, vague, and difficult to rate [1, 5–8]. The inclusion of too many “threads,” or sets of milestones progressing developmentally from level 1 to level 5, within individual subcompetencies caused problems assigning a single Milestone rating. A lack of sufficient faculty development and trainee education led to difficulty transitioning to a culture of grading according to developmental milestone achievement. Faculty members were tethered to Likert scale measures of excellence and their trend toward “grade inflation.” There was confusion differentiating milestone level from postgraduate year of training. Other incentives for “grade inflation,” and specifically for rating graduating trainees at or above level 4, included the definition of level 4 as graduation level, and concerns about how Milestones data might reflect on the program. Despite expected common elements of SBP, PBLI, PROF, and ICS across medical disciplines, milestones differed considerably across specialties [1].

In developing the next iteration of the Milestones, Milestones 2.0, the ACGME had several goals. These included increasing clarity and readability, decreasing the number of milestones and threads within each subcompetency and eliminating “orphan” milestones that were not part of a developmental progression. The ACGME convened four separate committees to develop common, or harmonized, subcompetencies and milestones across specialties for SBP, PBLI, PROF, and ICS, although the Psychiatry Milestones Workgroup could and did modify these for psychiatry. These issues were addressed within the Milestones themselves and changes made to the Psychiatry Milestones 2.0 to address these goals are discussed elsewhere [9]. Further goals were to provide specific examples of milestone achievement to foster development of a shared mental model within each specialty and program and to provide tools and resources to aid implementation [1]. A new secondary document, the Supplemental Guide, was created by each specialty’s Milestones 2.0 development group to provide such examples and resources. In this paper, we discuss the contents of the Psychiatry Supplemental Guide, ways in which psychiatry programs can use and customize this Guide, and possible future directions for the Supplemental Guide.

Contents of the Supplemental Guide

Two versions of the Supplemental Guide are available on the ACGME website. The first is a locked PDF which represents the “official” version of the Guide approved by the ACGME and members of the Milestones 2.0 Working Group [10]. The second version is a “template.” This version of the Guide is identical in content but is an editable Word document. It is intended for programs to use as a living document that can be tailored in a dynamic and ongoing way to the individual needs of their program.

Within the Guide, each subcompetency receives its own table (see Table 1 for an example). Under the name of the subcompetency is a brief description of its overall intent, i.e., the knowledge or skill to be assessed. Next, the table lists all thread content from the subcompetency by level with accompanying examples of resident behavior illustrating achievement at each level of each thread. These examples are not intended to be exhaustive or prescriptive; rather, they are intended to provide tangible examples to aid resident and faculty understanding and Clinical Competency Committee (CCC) discussion and decision-making. In the Word version of the Guide, programs may choose to add examples that better reflect the clinical experience and specific rotations of their program.

Table 1.

Content example, Supplemental Guide for Psychiatry

A. Patient care 1: Psychiatric evaluation B. Overall intent: To gather and organize findings from the patient interview; mental status and cognitive exams; targeted physical and neurologic exams; data from collateral sources including information gathered from the medical record, family members, other treaters; and laboratory and imaging results; to screen for risk and integrate risk assessment into the patient evaluation
Milestones	Examples
C	Patient is referred to the emergency room by his or her primary care provider. The patient’s partner is present, and the patient reports feeling overwhelmed and anxious. At the conclusion of the assessment, the patient is found to have an alcohol use disorder and to be the victim of interpersonal violence (Vignette written for levels 1–4)
Level 1 Collects general medical and psychiatric history and completes a mental status examination	•Uses a template to obtain thorough psychiatric and medical history and completes a mental status and cognitive exams
Collects relevant information from collateral sources	•Contacts primary care provider of a patient who said, “I don’t think I can go on like this,” during a visit
Screens for risk of harm to self, to others, or by others	•Asks the patient if the patient is feeling suicidal
D. Assessment Models or Tools	•American Board of Psychiatry and Neurology Clinical Skills Verification (ABPN CSV) •Case-based discussion •Clinical skills exam •Direct observation •Medical record (chart) audit •Simulation or standardized patients
E. Curriculum mapping
F. Notes or resources	•This Milestone set refers to psychiatric evaluations in all clinical settings (e.g., emergency, inpatient, outpatient, consultation) and with patients throughout the lifespan •Collateral includes information from family members, friends, caregivers, other providers, past medical records •Case presentation and documentation is included in interpersonal and communication skills •American Association of Directors of Psychiatric Residency Training. Virtual Training Office. https://www.aadprt.org/training-directors/virtual-training-office. 2019 •Columbia suicide severity rating scale •American Psychiatric Association. The American Psychiatric Association Practice Guidelines for the Psychiatric Evaluation of Adults. 3rd ed. Arlington, VA: American Psychiatric Publishing; 2016. https://psychiatryonline.org/doi/book/10.1176/appi.books.9780890426760. 2019

A, subcompetency name; B, overall intent of subcompetency; C, behavioral examples of each level of each thread; D, assessment models or tools that could be used to assess the subcompetency; E, curriculum mapping (for internal program use); F, notes or resources that may be of use in learning more about the subcompetency. Adapted from: Accreditation Council for Graduate Medical Education. Supplemental Guide: Psychiatry, pages 5-7. https://www.acgme.org/Portals/0/PDFs/Milestones/PsychiatrySupplementalGuide.pdf?ver=2020-03-10-161139-047. Accessed March 7, 2021

Following the examples are suggestions for possible assessment models or tools to assess the subcompetency. Again, these are intended to serve as a starting point for the program, not an exhaustive list of every possible assessment method for the subcompetency. Suggested methods vary depending on the subcompetency, for example, multiple-choice exams for medical knowledge or direct observation for patient care skills. Programs can add the assessment tools that they use in their program.

The next section, curriculum mapping, is intended solely for the use of the program. Here, the program can list elements of their curriculum (seminars and rotations) that relate directly to the teaching of the subcompetency. For example, the program’s interviewing skills curriculum would map to the patient care subcompetency of psychiatric evaluation. Curriculum mapping can serve as a useful tool for ensuring that all subcompetencies are being appropriately taught within a program.

The final segment of each table is notes and resources. This section is devoted to additional footnotes regarding subcompetency content and potential references, such as textbooks, journal articles, and screening or rating tools. This list contains items deemed to be of potential use in learning more about the skills being assessed or in developing internal curricula or assessment tools.

At the end of the Supplemental Guide is a table that maps the 1.0 Milestones to the 2.0 Milestones. This is helpful for tracking concepts across both versions of the Milestones and for quickly identifying skills that are newly incorporated into the Milestone framework, such as PROF3: Well-Being.

Using the Supplemental Guide

One of the main goals of creating the Supplemental Guide was to help programs create a shared mental model of resident professional development. Highly coordinated, effective teams make sound decisions even in complex situations based on team members’ ability to communicate information effectively using shared concepts and language. This common understanding is the shared (or team) mental model [11]. In sports, players on good teams anticipate the next move a teammate will make and act accordingly. Team members simultaneously, but independently, interpret game information the same way, develop similar causal accounts for a situation, and share expectations concerning future events.

The development of ACGME’s Milestone Project, with its iterations of the specialty Milestones and now the Supplemental Guide, is an attempt to help residency programs develop a similar shared mental model of professional development. The Supplemental Guide expands on the Milestones to allow programs to better describe, explain, and predict their residents’ progress. It helps programs to describe residents’ current professional development level by providing detailed examples of knowledge, skills, and attitudes at different levels within each subcompetency. It explains the intent of each subcompetency and describes how resident milestone determinations can be made by providing possible assessment tools. In addition, it provides descriptions of how residents’ knowledge, skills, and attitudes evolve across their professional development as a psychiatrist, thereby outlining a pathway of expectations for a resident’s future development.

Because the Supplemental Guide is editable and programs are encouraged to create their own examples of resident attainment, individual residencies can create a customized Guide that portrays a highly localized professional development model. The Supplemental Guide, in its original or customized form, can be useful to the many constituents of a residency program, including program directors, residents, faculty, CCCs, and Program Evaluation Committees (PECs).

With the development and first use of the Milestones, our experience as program directors was that residents had trouble understanding how their subcompetency scores were derived and how they could be used to guide their ongoing professional development. The scores seemed arbitrary and residents did not routinely utilize Milestones to create goals for the next phase of their professional development. Studies from other medical specialties indicate variable concordance between a resident’s self-assessment on the Milestones and ratings by faculty or CCCs [12–14]. The Supplemental Guide, especially one edited to include details relevant to the local program, can offer transparent linkages from individual evaluations to subcompetency scores. It can also be used in resident meetings with program directors to describe the pathway to becoming an independent psychiatrist. Essentially, the Supplemental Guide can provide a narrative behind the numbers.

Reliable scoring between supervisors is difficult without a shared mental model. Some faculty members have had difficulty transitioning to competency-based assessments; to them, great trainees deserve the top score of a perceived Likert scale. Using the Guide to create rotation-specific descriptions of different subcompetency levels allows both residents and faculty to have confidence in the reliability of resident assessment. The written descriptions and clinical vignettes in the Supplemental Guide can be used for faculty development across the residency and best practices can be shared. Faculty development focused on the Milestones may also allow programs to engage community faculty more intentionally in resident education while enabling them to participate more meaningfully in resident advancement [15].

For CCCs, it can be challenging to distill disparate assessments into a single subcompetency determination. Having a shared mental model is important for effective functioning of CCCs [16, 17]. Suggestions for developing a shared mental model include clear orientation to the purpose of the group and the criteria for assessment, group discussion of specific examples, and having individual CCC members take ownership of particular competencies or subcompetencies, educating other members about them [16, 17]. The Supplemental Guide can aid in this process by explaining the intent of each subcompetency and providing specific examples of achievement on each milestone. The CCC can edit and add to the Guide based on examples they discuss and their growing areas of consensus. Another challenge for CCCs is the number, types, and validity of assessment measures. A multisite study of psychiatry CCCs found that PC and MK milestones were the easiest to rate and that a greater diversity and number of assessments, beyond end-of-rotation assessments, was needed to assess other competencies [18]. The Supplemental Guide takes a first step in suggesting assessment methods for each subcompetency, although the further development of useful, reliable, and valid assessment tools remains a challenge for implementation of competency-based medical education.

The Supplemental Guide can be helpful to residency PECs in aligning individual curriculum elements, such as seminars and clinical rotations, with the program’s overall educational goals and desired milestone and competency outcomes. The Milestones constitute a core component of competency-based medical education [1–3]. They can be used to define expected developmental progression throughout training and expected resident and program outcomes. PECs can use the Supplemental Guide to map their curriculum to the Milestones and identify curricular and assessment gaps as part of a continuous quality improvement process.

Finally, the Supplemental Guide can serve as an outline of best practices in training in psychiatry. Previously, psychiatry programs did not have a formal way of sharing professional development models of graduate medical education. The Milestones themselves were a step in that direction. Although Milestones development committees included psychiatry educators from around the country, members were generally not content experts in many subcompetencies. The Supplemental Guide creates an opportunity for content experts to develop improved resident performance descriptions, assessment tools, and guidance toward coaching and remediating resident performance for specific subcompetencies. The Guide has the potential to become an evidence-based specialty-wide shared mental model of professional development within psychiatry.

Discussion

The Psychiatry Milestones 2.0 Workgroup created a Supplemental Guide to provide further explanation of the Milestones, examples of performance at each level within each subcompetency, potential assessment methods, and resources for improving the understanding and usefulness of the Milestones for residents, faculty, and programs.

There are several limitations to the Supplemental Guide. First, it is a guide for understanding and assessment of Milestones. However, the Milestones do not include everything to be learned during psychiatry residency, but rather only selected knowledge, skills, and attitudes acquired in a developmental progression over the course of training. The Supplemental Guide also does not comprehensively address resident evaluation, program evaluation, or quality improvement. Secondly, the examples and assessment methods included are limited by the lack of specific content expertise of members of the Psychiatry Milestones 2.0 Working Group and the validity and reliability of existing assessments. In addition, there are limitations to the extent to which programs and faculty can achieve a truly “shared mental model” of the Milestones and milestone achievement. For example, faculty from different training backgrounds, generations, and theoretical perspectives are likely to have different mental models of the satisfactory achievement of particular skillsets or areas of knowledge. However, through use of the Supplemental Guide and additional program-specific examples, the hope is to increase common understanding and shared rating practices.

Programs can use the Supplemental Guide template to create customized guides reflecting the program’s shared mental model of the Milestones with “institution/program-specific examples, assessment tools used by the program, and curricular components” [10]. Because psychiatric care delivery depends on local social and cultural factors, education and assessment of psychiatrists in training must adapt as well. For example, the clinical environment in regions with shorter hospital stays will involve differences in patient assessment, care modalities, involvement of family and social supports, and arrangement of follow-up care compared to areas where longer stays are the norm. With the advent of the COVID-19 pandemic, many programs began delivering care primarily through telepsychiatry and integrated care models. The Supplemental Guide template could be adjusted to reflect the evolving mental model of Milestone attainment based on such care delivery changes.

Nationally, the Supplemental Guide can form the basis for research and improvement in assessment in psychiatry training and in graduate medical education generally. Medical student and resident educator associations could utilize their content expert and assessment committees to draft more extensive examples of milestone achievement and suggest appropriate assessment tools. Organizations such as the American Association of Directors of Psychiatric Residency Training (AADPRT) could help programs share best practices for use of the Supplemental Guide. Because of psychiatry’s special role in professionalism and interpersonal and communication skills, additional content regarding best practices in these core competencies could be useful for Milestone assessment in all specialties.

Overall, the Supplemental Guide was constructed to provide better understanding of the next version of the Psychiatry Milestones for programs, faculty, and trainees. It has the potential to serve as both a tool for sharing a local program’s mental model of Milestone attainment and as a guide for the development of best practices in Milestone assessment. We invite programs and organizations to help meet that potential.

Declarations

Disclosures

Dr. Bentman reports non-financial support from Accreditation Council for Graduate Medical Education (ACGME) - Psychiatry Milestones 2.0 Work Group and non-financial support from ACGME/Psychiatry Review Committee - member, now Vice-Chair, outside the submitted work. Dr. DeJong reports she was Past President of AADPRT when Milestones 2.0 for Psychiatry were developed. Drs. Cowley, Houston, and Kinzie report non-financial support from the Accreditation Council for Graduate Medical Education. Dr. Macaluso has conducted clinical trials research as principal investigator for the following pharmaceutical companies over the last 12 months: (1) Acadia, (2) Allergan, (3) Alkermes, (4) AssureRx/Myriad, (5) Eisai, (6) Lundbeck, (7) Liva Nova, (8) Janssen, (9) Neurim, (10) Otsuka, (11) SAGE pharmaceuticals, and (12) Suven. All clinical trial and study contracts were with and payments made to either the University of Alabama at Birmingham Medical Center or the Kansas University Medical Center Research Institute. From April 2019 to June of 2020, Dr. Macaluso was a member of the speaker bureau for Janssen pharmaceuticals (Spravato/esketamine). Drs. Aoun, Bowen, Edgar, Janssen, and Keepers have nothing to disclose.