Miles to Go Before We Sleep. Reforming the Pulmonary and Critical Care Milestones to Improve Trainee Assessment

Abstract

The Accreditation Council for Graduate Medical Education (ACGME) Milestones are a systematic assessment framework for medical trainees within the six core competencies of practice. Their use by internal medicine subspecialties, including semiannual reports to the ACGME, was mandated beginning in 2014. The Milestones, which were based on specific, observable behaviors, improved upon the prior subjective, global comparisons of each fellow with an “average” fellow in his or her field and served the goals of competency-based medical education. However, the original set of Milestones has proven challenging to apply and interpret. Part of the challenge stems from the use of identical Milestones across all medicine subspecialties, which led to unclear relevance of the patient care and medical knowledge domains to the practice of pulmonary and critical care. This also precluded their use for individualized feedback or development of a learning plan for fellows. In addition, verbose behavioral descriptors, which were designed to provide specificity, ultimately led to rater fatigue among assessors and clinical competency committees. Therefore, the ACGME convened committees for each of the medical subspecialties to revise the original Milestones in an effort to improve subspecialty relevance, minimize educational jargon, and simplify the current iteration. New patient care and medical knowledge Milestones were created to be subspecialty specific and improve utility. The remaining four Milestones were developed as a common set of shorter Milestones, harmonized across specialties. For pulmonary, critical care, and combined fellowship programs, the resulting Milestones 2.0 aims to simplify the use, implementation, and interpretation of this framework for program directors, trainees, and society.

The woods are lovely, dark and deep, But I have promises to keep, And miles to go before I sleep, And miles to go before I sleep.

—Robert Frost, “Stopping by Woods on a Snowy Evening”

Systems-based practice, medical knowledge, patient care, professionalism, interpersonal and communication skills, practice-based learning and improvement: in the two decades since the Accreditation Council for Graduate Medical Education (ACGME) embraced the concept of competency-based medical education, training program directors, faculty, and trainees have grappled with these six “core competencies” of medical practice. How can the myriad tasks of a physician, such as consulting on patients, providing longitudinal care, performing procedures, and serving on hospital committees, be parsed into these six reductionist phrases? Medical educators at the bedside and in the clinic have struggled to assess trainees with respect to these competencies, which often seemed ill defined and overlapping.

Thus, soon after their introduction, the need for clear, shared definitions and a structured assessment system quickly became apparent. The abstruse language of the core competencies defied a common set of expectations for trainees, their teachers, and society. Moreover, an assessment framework that was fair, concrete, behavior based, and less obscured by educational jargon was lacking. To address this need, in 2013 the ACGME introduced the Milestones, a novel criterion-based framework for assessing Graduate Medical Education (GME) competency (1–3). This assessment framework encompassed the six core competencies but also provided a trajectory-based metric to demonstrate a trainee’s progression toward competence and, ultimately, readiness for unsupervised practice (4–7).

In general terms, the Milestones are analogous to childhood developmental milestones, described in terms of behaviors that can be observed during medical training (3, 8). The Milestones posit that, like childhood development, trainees progress with regard to the six competencies at rates and in temporal patterns that can vary within and among individuals (Figure 1). They provide trainees and their instructors with a standardized map of expectations that can help inform a learning plan. Beyond this, they provide a benchmark for minimal competency, and thus an assurance that program graduates can practice independently even while they continue to improve their skills.

Figure 1.

Conceptual depiction of the Milestones. Each line could represent the overall progress of three different fellows or the progress of one fellow in three different domains. Some arrive more advanced and progress at faster or slower rates.

To operationalize the vision of the Milestones, the ACGME convened working groups in each of the core fields of medicine (internal medicine [IM], pediatrics, surgery, etc.) in 2009 (6). These committees were tasked with parsing the six core competencies into discrete, measurable “subcompetencies” amenable to assessment. Each of these subcompetencies was then coupled to sets of observable, behavioral anchors spanning a five-point scale. For IM, the levels ranged from “critical deficiencies” (level 1) to “ready for independent practice” (level 4) and, finally, to a level deemed “aspirational” (level 5). In 2013, this novel assessment system was launched for all core residency programs (3, 4, 6).

Soon thereafter, work began on establishing Milestones for the subspecialties of IM. For this project, the ACGME assembled a single, large working group with representation from the major professional societies and associations of program directors for all of the subspecialties, the American Board of Internal Medicine, the American College of Physicians, and other stakeholders. Although pulmonary and critical care medicine (PCCM) was well represented by members from the American Thoracic Society, the American College of Chest Physicians, the Society for Critical Care Medicine, and the Association of Pulmonary and Critical Care Medicine Program Directors, these members comprised a small fraction of the larger 85-member committee (9).

One early intention was to use the Milestones as a metric to compare the different subspecialties. To promote generalizability across IM subspecialties, the committee was encouraged to adopt a uniform set of Milestones for all specialties, using the IM core Milestones as a template. In the subcompetencies for professionalism, interpersonal and communication skills, systems-based practice, and practice-based learning and improvement, the IM Milestones were adopted virtually verbatim. In contrast, the Milestones for patient care and medical knowledge were modified more extensively to capture the more advanced expectations of fellowship training. This included an added subcompetency separating assessments for invasive and noninvasive procedures, and a new subcompetency for scholarship, broadly defined. Finally, in consideration of the more discontinuous clinical training in many fellowships with long periods of research or academic time, a checkbox labeled “Not yet assessable” was added to each subcompetency (Figure 2A) (9).

Figure 2.

(A) Milestones 1.0 table for invasive procedural skills. Reprinted by permission from Reference 9. (B) Milestones 2.0 table for procedural skills, both invasive and noninvasive. Reprinted by permission from Reference 17. PC = patient care.

The Milestones were implemented nationally across all IM subspecialties in July 2014. To ensure they met the needs of physicians, trainees, and society at large, the ACGME committed to ongoing data collection and review, with the opportunity for future revisions. Feedback from educators, program directors, and trainees, with additional input from qualitative and quantitative research, highlighted the benefits and challenges of the original Milestones within the subspecialties (3, 5, 10, 11). In PCCM and other fields, this system appeared to achieve construct validity based on the year-by-year upward trajectory of milestone ratings among fellows (Figure 3) (12). However, national implementation revealed several practical shortcomings.

Figure 3.

Example of a year-to-year trajectory of the 2019 national cohort of all pulmonary and critical care medicine fellows (n = 1,903) in four of the patient care subcompetencies (12). The box represents the 25th–75th percentile and the diamond is the median for all evaluated fellows during the 2018–2019 academic year. (Note that the Accreditation Council for Graduate Medical Education refers to trainees in any year, both fellows and residents, as “residents.”) Reprinted by permission from Reference 12. PC = patient care.

For trainees, Milestones summary reports lack meaningful, specific, actionable feedback with clear relevance to PCCM.

One of these shortcomings involved the lengthy descriptive anchors used for each Milestone. The Milestones were written to provide multiple concrete examples for each row and column that could be compared with observed behaviors. However, this led to each subcompetency table being quite verbose (Figure 2A). The voluminous reading required for each subcompetency made assigning Milestones to trainees rather onerous, especially for large programs. The potential consequences of this included survey fatigue and inattention, which may have contributed to “straight-lining” of the ratings for individual learners (i.e., filling in the same level on every question). Such straight-lining of ratings was seen for 13.9%, 13.1%, and 25.6% of first-, second-, and third-year fellows in PCCM, respectively, during the 2018–2019 academic year.

The use of the same set of Milestones for all medical subspecialties has also had some untoward consequences for both programs and trainees (13). Although the anchors are based on observable behaviors, they are described in subspecialty-independent, context-free language. This has required substantial faculty development to understand how these broad Milestones are to be applied within each specialty and each program. For trainees, Milestones summary reports lack meaningful, specific, actionable feedback with clear relevance to PCCM.

Furthermore, the expected value of comparing Milestones among different IM subspecialties has proven illusory. The temporal trajectories are very similar among the subspecialties (12), and were there any differences, interpretation of these differences would be quite speculative.

Finally, the labeling of the columns, intended as a helpful guide for raters, may have had an unexpected influence on the selection of Milestones. For example, the damning “Critical Deficiency” column is infrequently selected. Among all 24 subcompetency Milestones evaluations completed in January and June for all first-year PCCM fellows in academic year 2018–2019, level 1 was assigned just 0.28% of the time. On the other hand, “Ready for unsupervised practice” (level 4 or above) strongly implies that a fellow must reach this level on all subcompetencies before completion of the fellowship and is nearly universally selected in all subcompetencies for graduating fellows (12). “Aspirational” (Milestones level 4.5 or 5), which was intended for rare use for fellows who truly exceed program expectations for a graduate, was selected for 45% of all subcompetencies for senior fellows in their final 2019 evaluation (Kenji Yamazaki, Ph.D., written communication, October 11, 2019). Overall, these implementation issues highlighted the need for a revised version of the Milestones for subspecialty programs.

Milestones 2.0

The Milestones 2.0 project aims to address these shortcomings, using the lessons learned in the original iteration to improve the Milestones across core and subspecialty fields (14). A national survey of IM subspecialty program directors revealed a widespread preference for separate Milestones for each field to improve relevance, so each subspecialty was tasked with making specialty-specific revisions of the original Milestones through different working groups. Therefore, for PCCM-specific Milestones planning, the major specialty societies (the American Thoracic Society, American College of Chest Physicians, Society for Critical Care Medicine, and Association of Pulmonary and Critical Care Medicine Program Directors) and fellowship program directors were invited to provide working group participants. In addition to program and society leadership, a fellow, a member of the public, and a member of the ACGME staff yielded a committee of 16 members, representing pulmonary, critical care, and PCCM combined programs, who shared knowledge of the field and experience with program management and education. These members represented a variety of program sizes and types, and regions of the country. The smaller size of this committee facilitated consensus building. Furthermore, the diversity of members and programs provided some measure of generalizability across the wider diversity of all ACGME-accredited pulmonary, critical care, and PCCM subspecialty training programs.

As a result of these changes, we believe that Milestones 2.0, although briefer than the original version, will yield greater specificity for the knowledge and skills relevant to PCCM.

The PCCM Milestones 2.0 Committee was charged with reviewing and revising all current Milestones, with the goal of ensuring that they represented all the pivotal skills necessary to work effectively and safely as a physician within pulmonary, critical care, or combined fields. To achieve this goal, new medical knowledge and patient care Milestones were developed by the committee to align optimally with subspecialty knowledge and skill requirements. In contrast, to provide a more consistent and rational set of non–patient care or medical knowledge competencies across all of the medical specialties, four other committees wrote “harmonized” Milestones for professionalism, interpersonal and communication skills, practice-based learning and improvement, and systems-based practice (15, 16). The harmonized Milestones comprising these four core competencies underwent only modest editing by the PCCM committee.

To address rater fatigue, the committee minimized the number of subcompetencies. The behavioral anchors were reduced to a maximum of three rows, with each row describing progressive competence in a single assessable skill. For this purpose, while preserving the essence of the Milestones, we reduced the number of subcompetencies for PCCM programs from 24 to just 19, and fewer still for stand-alone pulmonary or critical care fellowships (Table 1). Rows of descriptive behavioral anchors were also reduced (from >75 to 40; Figure 2B). Although the free-standing subcompetency for scholarship was eliminated, the requirement for scholarship during fellowship training was retained within evidence-based and evidence-informed practice. We also sought to minimize educational jargon to clarify the intended progression toward expertise for non–program leaders and medical education initiates. We avoided descriptors such as “inconsistently” and “frequently,” which invite varied interpretation. Finally, we eliminated the columnar titles that may have influenced evaluators; these columns are now just labeled levels 1–5, with no implied criticism or expectation.

Table 1.

Comparison of subcompetencies in Milestones 1.0 versus 2.0

	Milestones 1.0		Milestones 2.0
PC1	Gathers and synthesizes essential and accurate information to define each patient’s clinical problem(s).	PC1	History and physical exam
PC2	Develops and achieves comprehensive management plan for each patient	PC2	Disease management in critical care
PC3	Manages patients with progressive responsibility and independence	PC3	Disease management in pulmonary medicine
PC4a	Demonstrates skill in performing and interpreting invasive procedures	PC4	Preprocedure assessment
PC4b	Demonstrates skill in performing and interpreting non-invasive procedures and/or testing	PC5	Procedures (invasive and noninvasive)
PC5	Requests and provides consultative care	—	—
MK1	Possesses clinical knowledge	MK1	Clinical reasoning
MK2	Knowledge of diagnostic testing and procedures	MK2	Scientific knowledge of disease and therapeutics
MK3	Scholarship	—	—
SBP1	Works effectively within an interprofessional team	SBP1	Patient safety and quality improvement
SBP2	Recognizes system error and advocates for system improvement	SBP2	Coordination and transition in care
SBP3	Identifies forces that impact the cost of health care, and advocates for and practices cost-effective care	SBP3	Population health
SBP4	Transitions patients effectively within and across health delivery systems	SBP4	Physician role in healthcare systems
PBLI1	Monitors practice with a goal for improvement	PBLI1	Evidence-based and informed practice
PBLI2	Learns and improves via performance audit	PBLI2	Reflective practice and commitment to personal growth
PBLI3	Learns and improves via feedback	—	—
PBLI4	Learns and improves at the point of care	—	—
PROF1	Has professional and respectful interactions with patients, caregivers, and members of the interprofessional team	PROF1	Professional behavior and ethical principles
PROF2	Accepts responsibility and follows through on tasks	PROF2	Accountability
PROF3	Responds to each patient’s unique characteristics and needs	PROF3	Wellness and resiliency
PROF4	Exhibits integrity and ethical behavior in professional conduct	—	—
ICS1	Communicates effectively with patients and caregivers	ICS1	Patient- and family-centered communication
ICS2	Communicates effectively in interprofessional teams	ICS2	Complex communication around serious illness
ICS3	Appropriate utilization and completion of health records	ICS3	Interprofessional and team communication
—	—	ICS4	Communication within healthcare systems

Definition of abbreviations: ICS = interpersonal and communication skills; MK = medical knowledge; PBLI = practice-based learning and improvement; PC = patient care; PROF = professionalism; SBP = systems-based practice. Fellows in combined pulmonary and critical care programs will be evaluated in all subcompetencies. Fellows in pulmonary programs will be evaluated in all but PC2, and fellows in critical care medicine programs will be evaluated in all but PC3.

As a result of these changes, we believe that Milestones 2.0, although briefer than the original version, will yield greater specificity for the knowledge and skills relevant to PCCM. To help ensure uniform application among evaluators and programs, we have also written a Supplemental Guide, a users’ manual that we hope will provide insight into our thought process by defining terms and providing case examples for clarity. We hope that this revision will improve longitudinal assessments of trainees as they progress to independence while also reducing the burden for programs and their leaders.

Timeline and Future Work

The draft Milestones and Supplemental Guide will be posted for public comments in February 2020. Final versions will become effective as of July 1, 2020. Similarly to implementation of the original Milestones, Milestones 2.0 will require a substantial initial investment by fellowship program leaders and faculty. Current assessment tools that were devised for the original Milestones will need to be reviewed, revised, or replaced, and mapped to the new set of subcompetencies. New assessment tools that reflect the revised Milestones framework and our specialty-specific expectations for trainees will be needed. In addition, clinical competency committees and rater training will need to be adapted to this new model.

Overall, we recognize that Milestones 2.0 is an evolution, not a revolution. We sought to iteratively improve the original Milestones, which presented medical educators with an entirely new paradigm aligned with competency-based medical education. The original Milestones framework was a shift in trainee assessment from global, subjective, and normative referenced (How well does this trainee compare with some hypothetical average trainee?) to specific, objective, and criterion referenced (Has this trainee achieved this discrete step?). Milestones 2.0 maintains this progressive assessment paradigm but simplifies it, improves overall subspecialty relevance, and improves interpretability. These Milestones were written with consideration of their utility to trainees and programs but also with consideration of their documentation burden. We are optimistic that we achieved our intended goals.

Conclusions

The complex task of assessing physicians in training has taken many paths. This latest iteration of the Milestones attempts to improve on its predecessor. Milestones 2.0’s unambiguous pertinence to our practice, minimal education jargon, and streamlined and consistent text should help faculty more accurately judge trainees’ progress toward achieving expertise while also helping the trainees to recognize their strengths and weaknesses. Despite these enhancements, many important questions about the original Milestones remain unanswered. Compared with prior assessment systems, do they improve trainees’ understanding of their strengths and weaknesses? Do they help program directors individualize training better than prior systems? Are they achieving their intended goal of ensuring that program graduates are indeed competent? Although these questions will be no easier to answer with Milestones 2.0, the goals of this project were more modest: to make the Milestones more practical. Although we believe we are on the right path for competency-based assessments in PCCM training, we still have miles to go before we sleep.