Milestone Level Changes From Residency to Fellowship: A Multicenter Cohort Study

Abstract

Background

A vital element of the Next Accreditation System is measuring and reporting educational Milestones. Little is known about changes in Milestones levels during the transition from residency to fellowship training.

Objective

Evaluate the Accreditation Council for Graduate Medical Education (ACGME) Milestones' ability to provide a linear trajectory of professional development from general pediatrics residency to neonatal-perinatal medicine (NPM) fellowship training.

Methods

We identified 11 subcompetencies that were the same for general pediatrics residency and NPM fellowship. We then extracted the last residency Milestone level and the first fellowship Milestone level for each subcompetency from the ACGME's Accreditation Data System on 89 subjects who started fellowship training between 2014 and 2018 at 6 NPM fellowship programs. Mixed-effects models were used to examine the intra-individual changes in Milestone scores between residency and fellowship after adjusting for the effects of the individual programs.

Results

A total of 1905 subcompetency Milestone levels were analyzed. The average first fellowship Milestone levels were significantly lower than the last residency Milestone levels (residency, mean 3.99 [SD = 0.48] vs fellowship 2.51 [SD = 0.56]; P < .001). Milestone levels decreased by an average of -1.49 (SD = 0.65) from the last residency to the first fellowship evaluation. Significant differences in Milestone levels were seen in both context-dependent subcompetencies (patient care and medical knowledge) and context-independent subcompetencies (professionalism).

Conclusions

Contrary to providing a linear trajectory of professional development, we found that Milestone levels were reset when trainees transitioned from general pediatrics residency to NPM fellowship.

Objectives

We sought to compare the last assigned Milestone levels upon graduation from general pediatrics residency to the first assigned Milestone levels as first-year neonatal-perinatal medicine (NPM) fellows.

Findings

Average first fellowship Milestone levels were significantly lower than the last residency Milestone levels, and differences in Milestone levels were seen in both context-dependent subcompetencies (patient care and medical knowledge) and context-independent subcompetencies (professionalism).

Limitations

The study is limited by a retrospective observational design with a small sample size within a single speciality.

Bottom Line

Our results suggest Milestone levels are reset during the transition from general pediatric residency to NPM fellowship and do not create a logical trajectory of professional development in essential elements of competency.

Introduction

Preparing residents and fellows for independent practice is the primary goal of graduate medical education (GME). Competency-based medical education (CBME), an outcomes-based approach to training, was introduced to achieve this goal.^1–4 In 2013, the Accreditation Council for Graduate Medical Education (ACGME) implemented the Next Accreditation System (NAS) in an effort to advance CBME. A key element of the NAS is measuring and reporting educational Milestones.⁵ Milestones are defined by specialty-specific narrative descriptions of the trajectory of professional development.⁵ The goal of the Milestones framework is to “create a logical trajectory of professional development in essential elements of competency”⁵ and to connect medical education programs from undergraduate to graduate, and into practice and maintenance of certification.⁶ The Milestone levels are numeric ratings based on a developmental framework from “beginning resident” (level 1) to “aspirational” (level 5).^6–12 Level 4 generally denotes a level at which trainees are “ready for unsupervised practice.”^6–12

Kogan et al identified 4 themes influencing how faculty judge and numerically rate residents' clinical skills. The factors include variable frames of reference, a high degree of inference, variable approaches to synthesizing data into numerical ratings, and factors external to resident performance.¹³ According to Kogan and colleagues, these themes highlight the variability in the process of assigning numerical ratings based on clinical observations. They also challenge the internal structure (an important source of validity) of numerically based assessments of clinical skills.^14,15 Despite these concerns, several studies have shown a predictable upward trend of Milestone levels during residency training.^8,9,16–21 Reports from the ACGME suggest a similar upward trend of Milestone levels during subspecialty fellowship training.²² There is an evidence gap, however, regarding changes in Milestone levels during the transition from residency to subspecialty fellowship training.

Our study aim was to compare the last assigned Milestone levels of trainees upon graduation from general pediatrics residency to their first assigned Milestone levels as first-year neonatal-perinatal medicine (NPM) fellows. We hypothesized that Milestone level changes would show a linear trajectory of professional development as the trainees advanced from graduating residents to first-year fellows.

Methods

Setting and Participants

Study participants included a convenience sample of NPM fellows who started training between 2014 and 2018 at 6 NPM fellowship programs. Fellowship programs were selected to represent a range of sizes (small to large) and geographic diversity: Northeast, Midwest, South, and West.

Interventions

The study followed a multicenter retrospective cohort design. From the 21 general pediatrics subcompetencies¹¹ and the 21 pediatrics subspecialty subcompetencies,¹² we identified 11 subcompetencies that were identical between the 2 training contexts (provided as online supplementary data). The 11 subcompetencies included a combination of “context-independent” and “context-dependent” subcompetencies.²³

Context-independent subcompetencies are those in which the assessment of Milestone levels should not change based on the context of the training environment. As noted by Heath et al, the competencies of professionalism and interpersonal and communication skills are examples of context-independent subcompetencies.²³ In contrast, the subcompetencies of patient care and medical knowledge are examples of context-dependent subcompetencies for which the assessment of Milestone levels may change based on the context of the training environment.²³ Other subcompetencies in systems-based practice and practice-based learning and improvement are more ambiguous.²³ There were no identical interpersonal and communication skills subcompetencies between general pediatrics residency and NPM fellowship. Therefore, the only context-independent subcompetencies we were able to analyze were for professionalism.

Outcomes Measured

Each site investigator created a list of all trainees in their NPM fellowship program from 2014 to 2018. They then retrieved each of those fellows' Milestones levels on the 11 studied subcompetencies from the ACGME Accreditation Data System website at 2 time points: last assessment during residency and first assessment during fellowship. Milestone scores were deidentified after extraction and uploaded to a central database for analysis. Data were collected between March and November 2019 and managed using REDCap electronic data capture tools hosted at the University of Washington.^24,25 Subjects who completed only 2 years of pediatrics residency before entering NPM fellowship were excluded from the study.

Analysis of the Outcomes

As suggested by the ACGME, Milestone levels were treated as ordinal data and reported as means and standard deviation (SD).^26,27 Mixed-effects models were used to examine the intra-individual changes in Milestone scores between residency and fellowship after adjusting for the effects of the individual programs. For each individual, a difference in Milestone performance was calculated for each Milestone, and then an average taken across all Milestones. Taking the average difference in Milestones performance for each individual, linear models were used to examine the difference in Milestones performance between residency and fellowship across programs, as well as differences in Milestones performance across years after adjusting for the program as a factor. Upper and lower quartiles were determined based on the top 75th percentile and bottom 25th percentile Milestone levels. Milestones changes were calculated as the numerical change between the last residency Milestone level and the first fellowship Milestone level. For example, if a trainee subcompetency Milestone level was 3.0 on their last residency evaluation and 4.0 on their first fellowship evaluation, the Milestone level change was +1; if a trainee was at level 4.0 on their last residency evaluation and then level 3.0 on their first fellowship evaluation, the change was -1. A P value of < .05 was considered statistically significant. Statistical analyses were conducted using RStudio version 1.2.5033.

The study was approved by the Institutional Review Board at each participating site with a waiver of signed informed consent.

Results

Demographic data on the study subjects and the fellowship programs are provided in Table 1. Last residency and first fellowship Milestone levels were available for 89 trainees. The initial data set contained 1957 subcompetency Milestone levels. Of these, 52 Milestone levels were rated as “not yet assessable.” The final number of subcompetency Milestone levels analyzed was 1905.

Table 1.

Subject (n = 89) and Program (n = 6) Demographics

Demographics	Mean (SD)
New fellows per year	4 (2)
Total fellows in program	13 (6)
NICU beds	89 (30)
Total faculty	35 (10)
No. of fellows from each site	n
Site 1	8
Site 2	8
Site 3	18
Site 4	23
Site 5	14
Site 6	18
Year fellow started fellowship	n (%)
2014	7 (8)
2015	11 (12)
2016	27 (30)
2017	20 (22)
2018	24 (27)
Female	64 (72)
Chief resident prior to fellowship	14 (16)
NICU/newborn hospitalist prior to fellowship	10 (11)
Residency and fellowship at same location	16 (18)

The average last residency Milestone levels were significantly higher than the first fellowship Milestone levels (residency, mean Milestone level 3.99 (SD = 0.48) vs fellowship 2.51 (SD = 0.56); P < .001). Average Milestone levels decreased with a mean of -1.49 (SD = 0.65) from last residency to first fellowship assessment. The Figure shows the differences in average last residency and first fellow Milestone levels in the 5 ACGME core competencies. Table 2 shows the differences in last residency and first fellowship Milestone levels for the 11 subcompetencies analyzed in this study after adjusting for the fellowship training program in the mixed-effects model.

Figure.

Comparison of Average Milestone Rating in Each ACGME Core Competency for 89 Trainees Based on Last Assessment During Resident and First Assessment During Fellowship

Note: Patient care included 3 subcompetencies; medical knowledge included 1 subcompetency; systems-based practice included 2 subcompetencies; practice-based learning and improvement included 2 subcompetencies; professionalism included 3 subcompetencies.

Table 2.

Changes in Milestone Levels From Last Evaluation as Graduating Resident to First Evaluation as Neonatal-Perinatal Medicine Fellow

General Competency	Subcompetency	Milestone Levels, Mean (SD), n = 89
		Last Residency Milestone Level	First Fellowship Milestone Level	Milestone Level Change (95% CI)a	P Value
Patient care	Provide transfer of care that insures seamless transitions	4.03 (0.49)	2.53 (0.54)	-1.49 (-1.64, -1.35)	< .001
	Make informed diagnostic and therapeutic decisions that result in optimal clinical judgment	3.99 (0.39)	2.42 (0.56)	-1.58 (-1.71, -1.44)	< .001
	Develop and carry out management plans	4.12 (0.43)	2.48 (0.56)	-1.65 (-1.97, -1.51)	< .001
Medical knowledge	Locate, appraise, and assimilate evidence from scientific studies related to their patients' health problems	4.00 (0.50)	2.44 (0.54)	-1.56 (-1.69, -1.42)	< .001
Systems-based practice	Coordinate patient care within the health care system relevant to their clinical specialty	3.89 (0.43)	2.45 (0.46)	-1.44 (-1.56, -1.32)	< .001
	Work in interprofessional teams to enhance patient safety and improve patient care quality	3.97 (0.48)	2.46 (0.57)	-1.51 (-1.65, -1.36)	< .001
Practice-based learning and improvement	Identify strengths, deficiencies, and limits in one's knowledge and expertise	3.97 (0.43)	2.59 (0.62)	-1.37 (-1.51, -1.23)	< .001
	Systematically analyze practice using quality improvement methods, and implement changes with the goal of practice improvement	3.78 (0.50)	2.32 (0.56)	-1.47 (-1.62, -1.31)	< .001
Professionalism	Professional conduct: high standards of ethical behavior which includes maintaining appropriate professional boundaries	4.07 (0.5)	2.79 (0.57)	-1.27 (-1.62, -1.12)	< .001
	Trustworthiness that makes colleagues feel secure when one is responsible for the care of patients	4.13 (0.51)	2.66 (0.56)	-1.48 (-1.64, -1.33)	< .001
	Capacity to accept that ambiguity is part of clinical medicine and respond by utilizing appropriate resources in dealing with uncertainty	4.00 (0.48)	2.42 (0.53)	-1.59 (-1.73, -1.45)	< .001

^aAfter adjusting for fellowship training program in the mixed-effects model.

There were some statistically significant differences in last residency and first fellowship Milestone levels across the 6 training programs (Table 3). However, no significant change was noted in Milestone levels over time (Table 4).

Table 3.

Differences in Milestone Ratings Across Programs

Program	Last Resident Milestone Level, Mean (SD)	First Fellowship Milestone Level, Mean (SD)	Difference (95% CI)a	P Value
1 (n = 8)	3.86 (0.47)	2.72 (0.39)	Reference	…
2 (n = 8)	4.14 (0.35)	2.45 (0.39)	-0.54 (-1.00, -0.08)	.020
3 (n = 18)	3.83 (0.54)	2.03 (0.50)	-0.65 (-1.04, -0.25)	.002
4 (n = 23)	4.07 (0.36)	2.72 (0.51)	-0.23 (-0.61, 0.15)	.20
5 (n = 14)	4.12 (0.43)	2.84 (0.48)	-0.09 (-0.50, 0.31)	.70
6 (n = 18)	3.99 (0.54)	2.39 (0.51)	-0.45 (-0.84, -0.06)	.030

^aRelative to program 1 difference (reference), after adjusting for program in the linear model.

Table 4.

Differences in Milestone Ratings Over Time

Year Trainee Started Fellowship, Mean (SD)	Last Resident Milestone Level, Mean (SD)	First Fellowship Milestone Level, Mean (SD)	Difference (95% CI)a	P Value
2014 (n = 7)	3.91 (0.39)	2.53 (0.47)	Reference	…
2015 (n = 11)	4.16 (0.39)	2.80 (0.53)	-0.09 (-0.54, 0.37)	.70
2016 (n = 27)	4.03 (0.51)	2.55 (0.55)	-0.08 (-0.50, 0.33)	.70
2017 (n = 20)	3.92 (0.49)	2.29 (0.56)	-0.25 (-0.66, 0.18)	.30
2018 (n = 24)	3.96 (0.47)	2.49 (0.55)	-0.07 (-0.49, 0.35)	.70

^aRelative to 2014 (reference), after adjusting for program in the linear model.

There was no difference in first fellowship Milestone levels or Milestone level changes between trainees who did residency and fellowship in the same program and those who did not (same program, mean Milestone level 2.52 [SD = 0.54] vs different program 2.51 [SD = 0.57], P = .50; same program, mean Milestone level change -1.48 [SD = 0.49] vs different program -1.49 [SD = 0.66], P = .50).

Fellows who were a chief resident or a NICU/newborn hospitalist before fellowship had significantly lower Milestone levels as fellows and higher Milestone level change as compared to those who were not (first fellowship mean Milestone level for chief/hospitalist, 2.42 [SD = 0.58] vs not 2.53 [SD = 0.55], P = .010; chief/hospitalist, mean Milestone level change -1.68 [SD = 0.69] vs not -1.45 [SD = 0.64], P < .001).

Residents with Milestone levels in the top quartile had a greater decrease in Milestone levels on first fellowship evaluation as compared to residents in the bottom quartile (top 75th percentile, mean decrease -1.88 [SD = 0.59] vs bottom 25th percentile, -1.32 [SD = 0.59], P < .001). There was no clear relationship between having a last residency Milestone level in the top or bottom quartile and having a first fellowship Milestone level in the top or bottom quartile. Of the 20 residents with Milestone levels in the top 75th percentile, only 2 (10%) also had Milestone levels in the top 75th percentile on first fellowship evaluation. Of the 20 residents with Milestone levels in the bottom 25th percentile, 8 (40%) also had Milestone levels in the bottom 25th percentile on first fellow evaluation.

Discussion

We studied the ability of the ACGME Milestone levels to accurately measure professional development along the continuum from residency to subspecialty fellowship training. As opposed to seeing a plateau or increase in Milestone level, we found a significant drop in Milestone levels in all 11 subcompetencies studied during the transition from residency to fellowship. The drop was seen in both context-dependent and context-independent subcompetencies. These data suggest a general “resetting of the bar” when trainees move from pediatrics residency to NPM fellowship.

In a study by Li and colleagues of 2030 pediatrics residents across 47 programs, the investigators found significant variation in end-of-year Milestone levels, but noted that the variability decreased during training.²⁸ They reported that 79% of graduating third-year pediatrics residents received a Milestone level of ≥ 3 in all 21 pediatrics subcompetencies, and 21% received a level of ≥ 4 in all subcompetencies. The authors concluded that most graduating pediatrics residents were still advancing on the Milestone continuum and that a Milestone level of ≥ 3 at graduation was a realistic target for pediatrics residents. The Pediatrics Milestone Project supports a target Milestone level of 3 at graduation.¹¹ In our study, the average last residency Milestone level was 3.99 (SD = 0.48), and the average first fellowship Milestone level was 2.51 (SD = 0.56). Therefore, the average first-year NPM fellow Milestone level was lower than the target resident graduation level. The mean Milestone level decreased by -1.49 (SD = 0.65) from residency graduation to the start of fellowship. This decrease was seen in both context-dependent and context-independent subcompetencies. One could argue that this resetting of the bar is appropriate for the context-dependent core competencies of patient care and medical knowledge. For context-dependent subcompetencies, it is logical that a graduating pediatrics resident would be level 4 in general pediatrics patient care and medical knowledge, but not level 4 in neonatology patient care and medical knowledge. We find it hard to argue, however, that a resetting of the bar is appropriate for the 3 context-independent subcompetencies of professionalism we studied. We cannot explain the drop of 1.27 Milestone levels in professional conduct, 1.48 levels in trustworthiness, and 1.59 levels in capacity to accept that ambiguity as part of clinical medicine. Moreover, we cannot explain why the fellows in our study who were a chief resident or a NICU/newborn hospitalist before starting fellowship had a more significant drop in Milestone levels compared to trainees who went directly into fellowship after residency.

Heath and colleagues evaluated Milestone levels of internal medicine subspecialty fellows in the first 6 months of fellowship training as an indicator of response process validity.²³ The subcompetencies chosen were context-independent and included professionalism and interpersonal and communication skills. The author's assumed that those subcompetencies would not be affected by changes in the evaluation context of residency versus fellowship. The investigators found that 34% of professionalism subcompetencies and 26% of interpersonal and communication skills subcompetencies were scored at less than the resident “graduation target” of 4 during the first 6 months of fellowship training. These findings raise concerns about the validity of the Milestone system in specific medical subspecialties.

Our findings are consistent with those of Heath et al and provide additional evidence that Milestone levels are systematically reset when trainees begin fellowship. Why might clinical competency committies (CCCs) reset Milestone levels at the start of fellowship training? One reason is that CCCs want to allow for an upward progression in Milestone levels during training. In a 3-year NPM fellowship, this means that each fellow needs to start at a level < 4 to progress to the suggested target level of 4 at graduation.¹² This issue is a specific concern with the 11 subcompetencies used in the current study, which were the same for residents and fellows. In those 11 subcompetencies, a “ceiling effect” restricts an upward progression during fellowship. Another reason that CCCs might reset Milestone levels at the start of fellowship is a misapplication of the Milestones. An example of misapplication is to assign levels based on the year of training, rather than individual characteristics of the trainee. In a study of family medicine Milestones by Peabody et al, the authors found that the year of training was the primary factor in assigning Milestone levels.²⁰ This same phenomenon may be present in NPM fellowships. Misunderstanding of the Milestone levels by the CCC is another potential explanation. While each Milestone level we studied had a behavioral anchor, there is some degree of interpretation involved (eg, What is “professional conduct,” “trustworthiness,” and “capacity to accept ambiguity”?) To our knowledge, no psychometric analysis has been done on the pediatrics and pediatrics subspecialty Milestones. However, psychometric analysis of the family medicine Milestones supports the idea that some descriptors cause confusion.²⁰

This study has several limitations. The sample of 89 trainees was small. This small sample size increases the risk of bias. Further studies in larger cohorts of NPM fellows would offer additional validity evidence. Since each medical specialty and subspecialty has unique subcompetencies and Milestone levels, our study results may not be generalizable to other medical specialties and subspecialties. Due to the small number of training programs in this study, we could not perform subgroup analysis regarding the impact of program size or geographic location. Such programmatic factors may influence resident and fellow evaluation and would be good to control for in future studies.

Efforts can be taken at the local level to improve the quality of Milestones data. Kinnear et al examined available evidence and developed a set of practical tips to maximize the value of CCCs. Some tips to improve the CCC output included using assessment data from multiple sources, conducting regular CCC member training, and engaging the committee in continuous quality improvement.²⁹ Efforts to improve Milestone-based assessment are underway at the national level. The ACGME is currently working on ‘‘Milestones 2.0.”³⁰ The Milestones 2.0 process started in 2016 with the creation of Harmonized Milestones for interpersonal and communication skills, practice-based learning and improvement, professionalism, and systems-based practice.³⁰ The ability of Milestones 2.0 to accurately demonstrate professional development from residency to fellowship training remains to be seen.

Conclusions

Our results suggest Milestone levels are reset when trainees transition from general pediatric residency to NPM fellowship training. These findings challenge the idea that the Milestones create a logical trajectory of professional development in essential elements of competency.