Mind the Gap: The Bumpy Transition From Medical School to Residency

In 2008, the Accreditation Council for Graduate Medical Education began development of the Next Accreditation System, an assessment strategy based on residents' achievement of specific milestones within defined competencies as they progress through training.¹ Medical specialties were tasked with creating milestones specific to their training programs. Although similar competency frameworks were used across specialties, these milestones were largely developed in isolation. Not surprisingly, the resulting milestones were “richly diverse.”² In several specialties, Level 1 is considered the novice level, and is assumed to have been reached in medical school.

In this issue of the Journal of Graduate Medical Education, 2 articles report on the use of milestones to assess learners at the transition from undergraduate medical education (UME) to graduate medical education (GME). Clay et al³ described the assessment of fourth-year medical students with Transitional Year Milestones during a 4-week mandatory capstone course at the conclusion of their undergraduate training, while Weizberg and colleagues⁴ measured whether emergency medicine residents had achieved Level 1 Emergency Medicine Milestones at the end of their first month of residency. Although these were neither the same subjects, nor the same set of milestones, one would expect comparable performance on the overall construct of milestone evaluation, given the difference of only 1 month of residency. However, the results of these 2 studies suggest very different levels of performance. For students in the study by Clay et al,³ with a range of possible milestone levels from 1 to 5, the mean milestone levels on self-, peer, and faculty evaluations ranged between 2.2 and 3.6; in fact, no mention was made of anyone failing to achieve at least Level 2.

In contrast, only 20% to 73% of the cohort of residents studied by Weizberg et al⁴ achieved Level 1 milestones on faculty evaluations, and on 2 competencies, a majority of interns rated themselves as not achieving Level 1. Considering the difficulties with self-assessment, it is remarkable that interns admitted they did not achieve entry-level milestones. Depending on the subcompetency, 34% to 92% of interns reported achieving Level 1 on self-evaluations. Can we explain how senior medical school students achieved levels 2 to 3 milestones in one group, while a second group of interns in their first month of residency are, for the most part, not achieving Level 1?

Differences in GME Milestones

Milestones represent a developmental process: the “road” they measure has a beginning and an end. One would think that all paths charted by GME milestones would begin at the same place: the graduating medical student entering a residency program. However, comparisons between GME milestones of various specialties rapidly become complicated. Both of the articles in this issue chose milestones that defined Level 1 as “entry level” skills for an incoming resident. However, in the article by Clay et al,³ the authors were concerned that the bar was set too low; they reported that it was “challenging” to identify the target Transitional Year Milestone level for graduating students, despite defining Level 1 as “expected on entrance into transitional year education.”⁵

Some of this discrepancy is because different specialties have distinct expectations for the entering resident. For example, the emergency medicine subcompetency for diagnosis defines Level 1 as “Constructs a list of potential diagnoses based on chief complaint and initial assessment,”⁶ but an equivalent performance (“Integrates patient-specific information and generates an appropriate differential diagnosis”) earns a Level 2 using the Transitional Year Milestones.⁵ If other specialties are included, the divergence is even greater: Level 1 for all Internal Medicine Milestones is defined as “critical deficiencies.”⁷ Pediatrics intentionally frames their milestones “across the continuum of medical education,” from medical school into practice.⁸

When one looks at the content specificity of GME milestones, the differences continue to grow. For example, internal medicine and pediatrics have written milestones that focus on relatively generic skills across multiple medical problems (“Recognizes urgent and emergent medical conditions”).⁷ Other specialties, such as neurological surgery, have written milestones that focus on the application of those skills to specific problems, as in Surgical Treatment of Epilepsy and Movement Disorders. Emergency medicine includes milestones for “multitasking,” reflecting its unique importance to the specialty.

The rich diversity in both expectations and content does not limit the applicability of GME milestones to assessment within specialties or for medical school to GME transitions. Rather, it reflects the differing values, priorities, and cultures of those specialties, and may even improve the usability of GME milestones as specialty-specific assessment tools. However, it does make GME milestones problematic as a basis for assessment of medical students before they have differentiated into specialized practitioners.

The Challenge of Milestones in Undergraduate Medical Education

Competency-based undergraduate medical education (UME) requires curricula that support the desired level of achievement and evaluations that regularly gauge progress toward these predefined expectations of competence. However, UME and GME appear to be speaking differing dialects of the same assessment language. While GME has invested heavily in developing competency-based milestones, UME, represented by the Association of American Medical Colleges (AAMC), has focused on articulating entrustable professional activities (EPAs) for the graduating student.⁹ EPAs are independently executable, observable, and measurable tasks or responsibilities that can be performed unsupervised once a trainee has attained sufficient specific competence.¹⁰ EPAs can be indirectly linked to milestones via the core competencies,^11,12 but that raises the questions of which milestones, and from which specialties. A shared conceptual framework outlining how EPAs used in UME and GME milestones fit together is critical. One important step would be to clearly define a series of developmental milestones that span undergraduate and graduate education—a daunting task given the interspecialty variation discussed above.

Another step would be to determine which milestones link to entrustment decisions for professional activities; as noted, this may mean achievement of a limited number of different milestones for different specialties. Indeed, achieving the AAMC's core EPAs requires students to perform at or above the level described in many specialties' entry level milestones.

The 2 articles in the current issue highlight key areas of the work needed to align undergraduate and graduate achievement education. The Clay et al³ article, which focuses on the undergraduate side, conclude that waiting until a graduation capstone is too late to assess whether students have achieved a milestone. For many of the milestones assessed in the study, only half of the students achieved a level that the authors deemed acceptable. It is possible that the milestone level they found acceptable was too advanced, given that Level 3 is what is expected from a resident at the conclusion of a transitional year. The findings of Weizberg et al⁴ suggest that some students are graduating without achieving milestones expected of them in their new role as residents.

Altogether, these findings are concerning. For competency-based education to work, there needs to be enough time for meaningful remediation of students who have not attained the desired level of performance. Additionally, the evaluation of milestone achievement needs to occur at regular intervals throughout UME curricula, through reliable and valid assessment methods that are transparent to students and faculty members.

The Road Ahead

The way forward is daunting. Although medical schools may be working toward adopting uniform EPAs at graduation, the creation of consistent milestones spanning the 4 years of medical school will be difficult due to the diversity of curricula and experiences across the United States. For there to be continuity with GME, these diverse means must achieve similar ends. However, the work continues: the UME community is developing competencies and milestones, and the GME community is constructing EPAs.^13–15 A recent AAMC perspective article proposes core principles for an educational continuum that extends all the way from premedical training through postgraduate continuing medical education; this is not quite a blueprint, but a big step in the right direction.¹⁶ It is important that one road ends where the next begins—otherwise, we're in for a bumpy ride. The signs confirm there is road work ahead.