Pediatric emergency medicine fellows’ milestone evaluations: Do they all meet the targets for graduation?

Abstract

Background

The ACGME Milestone Project created a competency‐based trainee assessment tool. Subcompetencies (SCs) are scored on a 5‐point scale; level 4 is recommended for graduation. The 2018 Milestones Report found that across subspecialties, not all graduates attain level 4 for every SC.

Objective

The objective was to describe the number of pediatric emergency medicine (PEM) fellows who achieve ≥ level 4 in all 23 SCs at graduation and identify SCs where level 4 is not achieved and factors predictive of not achieving a level 4.

Methods

This is a multicenter, retrospective cohort study of PEM fellows from 2014 to 2018. Program directors provided milestone reports. Descriptive analysis of SC scores was performed. Subanalyses assessed differences in residency graduation scores, first‐year fellowship scores, and the rate of milestone attainment between fellows who did and did not attain ≥ level 4 at graduation.

Results

Data from 392 fellows were obtained. There were no SCs in which all fellows attained ≥ level 4 at graduation; the range of fellows scoring < level 4 per SC was 7% to 39%. A total of 67% of fellows did not attain ≥ level 4 on one or more SC. While some fellows failed to attain ≥ level 4 on up to all 23 SCs, 26% failed to meet level 4 on only one or two. In 19 SCs, residency graduation and/or first year fellow scores were lower for fellows who did not attain ≥ level 4 at graduation compared to those who did (mean difference = 0.74 points). Among 10 SCs, fellows who did not attain ≥ level 4 at graduation had a faster rate of improvement compared to those who did attain ≥ level 4.

Conclusion

In our sample, 67% of PEM fellows did not attain level 4 for one or more of the SCs at graduation. Low scores during residency or early in fellowship may predict difficulty in meeting level 4 by fellowship completion.

INTRODUCTION

In 2013, the Accreditation Council of Graduate Medical Education (ACGME) introduced the Next Accreditation System, a component of which incorporated a better measurement of graduate medical education (GME) trainee performance. This included a shift toward competency‐based education and a focus on program accreditation anchored on clinical outcomes. ¹ , ² , ³  This project, which was subsequently implemented into subspecialty fellowship training in 2015, detailed specialty specific subcompetencies (SCs) within each of the six core domains: patient care (PC), medical knowledge (MK), practice‐based learning and improvement (PBLI), systems‐based practice (SBP), professionalism (PROF), and interpersonal and communication skills (ICS). For each individual SC, narrative milestones were created to demonstrate progression from novice to expert across the continuum of medical education, from fellowship training into independent practice. This competency‐based evaluation framework is currently used by all ACGME fellowship training programs to assess trainee progress over time. All program directors (PDs) are required to submit milestone ratings to the ACGME twice annually.

There are 77 pediatric emergency medicine (PEM) fellowship programs in the United States. PEM accepts trainees from two distinct primary residency specialties: pediatrics and emergency medicine (EM). Pediatric‐trained fellows complete 3 years of fellowship training including a 12‐month requirement for scholarly activity, while EM‐trained fellows may complete training in 2 years. PEM fellows are assessed on 23 PEM SCs developed from a combination pediatric and EM SCs. ⁴ , ⁵ , ⁶ , ⁷

Each SC is scored on a 5‐point ordinal milestone scale, with level 1 reflecting beginner‐level performance and level 5 reflecting expert performance. Each milestone level has a descriptive anchor with advancing skills and behaviors. ² Level 4 is designated as the target for graduation by the ACGME and by many local GME departments, but has not been set as an absolute graduation threshold by any accreditation agency or medical board. ¹ ACGME publications have suggested that decisions about readiness for graduation should remain in the purview of the PD and the program's Clinical Competency Committee (CCC) until sufficient performance data are available to determine whether the milestones should be used for this high‐stakes decision and whether level 4 is the appropriate threshold required for graduation in all SCs. ¹  The 2018 Milestone Report found that across pediatric subspecialties, including PEM, not all graduates are attaining a level 4 for every SC. ⁸ , ⁹ Furthermore, surveys suggest that pediatric PDs are using milestone data in combination with other information to make advancement decisions; many feel that not every SC should be weighted equally. ¹⁰  Work is being done to establish ideal graduation targets for individual SCs, because evidence linking milestone levels at graduation with competency in independent practice is lacking.

Our study objective was to describe the number of PEM fellows achieving a level 4 in all 23 PEM SCs at graduation and to identify the SCs for which a level 4 is not achieved. We also sought to determine factors predictive of not achieving a level 4 at the time of graduation from PEM training.

METHODS

This is a retrospective cohort study of PEM fellows from 2014 to 2018. PEM fellowship PDs were asked to voluntarily share deidentified milestone reports for all of their trainees within the study time frame; these reports are all available for download by PDs through the ACGME WebAds portal, and new reports are available every 6 months. PDs were also asked to share the final graduation milestone reports that they received from a fellow's primary residency program, if available. PDs were recruited during national meetings and via email through the PEM fellowship director's listserv over a 6‐month period by a subgroup of PDs who served as the primary investigators. Trainees receive a unique identification number from the ACGME that was used to track their trainee information over time. Demographic information including gender, type of primary residency program (pediatrics or EM), and year entering fellowship was also collected.

Given that fellows entered and completed their fellowship training at different points during the timeframe of data collection, complete 3‐year data was not available for all individuals. Only the subset of fellows with data available for their final year of their fellowship were included in our analyses; subsets of this cohort had data available for all 3 years of fellowship training and/or from their primary residency program. Descriptive analysis of milestone scores was performed. This included frequencies and percentages for categorical data such as gender and type of residency program. The proportion of fellows meeting or exceeding a level 4 in each of the 23 SCs at the end of their fellowship training was determined. Demographic factors were compared between fellows who did not meet a level 4 on their final graduation assessment for each SC and to those who did attain or exceed this score, using the chi‐square test. Planned subanalyses using Wilcoxon rank‐sum test were performed to assess differences in first‐year subspecialty milestone scores and residency milestone scores. The rate of milestone attainment between fellows who did and did not attain a level 4 milestone score at graduation was also compared using linear mixed‐effects modeling. The model included a random intercept for each subject and main effects of fellowship year, attaining a level 4 milestone and their statistical interaction. ¹¹  When complete 3‐year data was not available, missing data for year 1 were treated as missing at random, which is a reasonable assumption for this type of longitudinal comparison, because outcome data from all available time points for any given subject were used in the estimation of the mean rate of change in scores over time. ¹² To adjust for multiple comparisons, we used the Bonferroni correction, where the domain‐level alpha of 0.05 was divided by the number of SCs used in that domain. The human subjects committee approved this study.

RESULTS

Of the 77 active PEM fellowship programs in the United States by 2018, PDs from 48 programs participated in this study and submitted data for 639 individual fellows (66% of all fellows nationally). ¹³  Data were available at the time of fellowship graduation for 392 fellows (62% of the fellows for whom milestone data were submitted). This study cohort had similar demographics to the total study population; the majority of fellows completed their primary residency in pediatrics (93%), were female (65%), and were enrolled in fellowship programs in the northeast (38%; Table 1). The demographics of the study cohort were similar to the demographics of the total PEM fellows training in the United States at that time (92% completed primary residency in pediatrics and 67% were female). ¹⁴ , ¹⁵ Our cohort included more fellows training in the northeast (38% vs. 31%) and less from the south compared to the total fellows training in that time period (24% vs. 34%). ¹⁵ Among the study cohort, first‐year milestone data were available for 202 individuals and graduation milestone data from the primary residency was available for 45 individuals (40 who trained in pediatrics and five in EM).

TABLE 1.

Demographic information by group

Demographic variable, n (%)	Total (N = 392)	Fellows attaining level 4 on all SCs (n = 132)	Fellows not attaining level 4 on at least one SC (n = 260)	x‐value
Gender a				0.31
Female	237 (65%)	86 (68%)	151 (63%)
Male	129 (35%)	40 (32%)	89 (37%)
Primary residency program b				0.88
Pediatrics	338 (93%)	118 (94%)	220 (93%)
EM	24 (7%)	8 (6%)	16 (7%)
Geographic area				0.07
Northeast	147 (38%)	49 (37%)	98 (38%)
South	96 (24%)	41 (31%)	55 (21%)
Midwest	90 (23%)	29 (22%)	61 (23%)
West	59 (15%)	13 (10%)	46 (18%)

^aGender data are available for 366 study participants.

^bPrimary residency program data is available for 362 participants.

There were no SCs in which all fellows attained a milestone score of level 4 or above at the time of graduation; the range of fellows not meeting at least a level 4 among the 23 SCs was from 7% to 39% (Table 2). Two‐hundred sixty fellows (67%) did not attain a level 4 or above on one or more of the 23 SCs at graduation. A total of 101 fellows (26%) failed to achieve a level 4 score on only one or two SCs; however, some individuals (n = 2, 0.5%) failed to attain a level 4 on all 23 of the SCs (Figure 1). The specific SCs with the largest number of graduates not attaining at least a level 4 included SBP1 (39%) and SBP2 (36%). PBLI and PROF represented the other core domains with the largest number of graduates scoring below level 4.

TABLE 2.

Number of fellows who fell below a level 4 milestone score at the time of graduation

SC	Descriptor	N (%), Total, N = 392
PC1	Gather essential and accurate information about the patient: abstracts current findings in a patient with multiple chronic medical problems and, when appropriate, compares with a prior medical record and identifies significant differences between the current presentation and past presentations	32 (8%)
PC2	Organize and prioritize responsibilities to provide patient care that is safe, effective, and efficient	70 (18%)
PC3	Provide transfer of care that ensures seamless transitions	40 (10%)
PC4	Make informed diagnostic and therapeutic decisions that result in optimal clinical judgment	74 (19%)
PC5	Emergency stabilization: prioritizes critical initial stabilization action and mobilizes hospital support services in the resuscitation of a critically ill or injured patient and reassesses after stabilizing intervention	59 (15%)
PC6	Diagnostic studies: applies the results of diagnostic testing based on the probability of disease and the likelihood of test results altering management	48 (12%)
PC7	Observation and reassessment: reevaluates patients undergoing ED observation (and monitoring) and, using appropriate data and resources, determines the differential diagnosis, treatment plan, and disposition	56 (14%)
PC8	Disposition: establishes and implements a comprehensive disposition plan that uses appropriate consultation resources, provides patient education regarding diagnosis, treatment plan, medications, and time‐ and location‐specific disposition instructions	34 (9%)
PC9	General approach to procedures: performs the indicated procedure on all appropriate patients (including those who are uncooperative, at the extremes of age, or hemodynamically unstable, and those who have multiple comorbidities, poorly defined anatomy, high risk for pain or procedural complications, or sedation requirements), takes steps to avoid potential complications, and recognizes the outcome and/or complications resulting from the procedure	44 (11%)
PC10	Anesthesia and acute pain management: provides safe acute pain management, anesthesia, and procedural sedation to patients of all ages regardless of the clinical situation	28 (7%)
PC11	Provide appropriate supervision (milestones for the supervisor)	47 (12%)
MK1	Demonstrate sufficient knowledge of the basic and clinically supportive sciences appropriate to PEM	27 (7%)
SBP1	Advocate for quality patient care and optimal patient care systems	152 (39%)
SBP2	Participate in identifying system errors and implementing potential systems solutions	140 (36%)
PBL1	Use information technology to optimize learning and care delivery	118 (30%)
PROF1	Self‐awareness of one's own knowledge, skill, and emotional limitations that leads to appropriate help‐seeking behaviors	54 (14%)
PROF2	The capacity to accept that ambiguity is part of clinical medicine and to recognize the need for and to utilize appropriate resources in dealing with uncertainty	78 (20%)
PROF3	Practice flexibility and maturity in adjusting to change with the capacity to alter behavior	124 (32%)
PROF4	Provide leadership skills that enhance team functioning, the learning environment, and/or the health care delivery system/environment with the ultimate intent of improving care of patients	58 (15%)
PROF5	Demonstrate self‐confidence that puts patients, families, and members of the health care team at ease	55 (14%)
ICS1	Communicate effectively with patients, families, and the public, as appropriate, across a broad range of socioeconomic and cultural backgrounds	47 (12%)
ICS2	Demonstrate the insight and understanding into emotion and human response to emotion that allows one to appropriately develop and manage human interactions	73 (19%)
ICS3	Act in a consultative role to other physicians and health professionals	100 (26%)

Abbreviations: ICS, interpersonal and communication skills; MK, medical knowledge; PBLI, practice‐based learning and improvement; PC, patient care; PEM, pediatric emergency medicine; PROF, professionalism; SBP, systems‐based practice.

FIGURE 1.

Number of SCs not being met. SCs, subcompetencies

In comparing individual fellows, there were no differences in gender, primary residency program, or geographic location of training among fellows who failed to attain a level 4 or above in at least one SC and those who attained a level 4 or above in all 23 SCs (Table 1). Similarly, when evaluating individual SCs, there was no difference in gender or primary residency program between those who did and did not attain at least a level 4 milestone score. There were some geographic differences noted among attainment of milestone scores in the SCs, but with no apparent pattern.

Those fellows who attained at least a level 4 milestone score at the time of fellowship graduation tended to have a higher score during their first year of fellowship training when compared with those who did not (mean difference = 0.74 points; Table 3). This difference was also seen in the comparison of milestone scores from residency graduation and the end of fellowship. In general, residency graduation scores were a half step, or 0.5 points, lower per SC among those who did not attain at least a level 4 milestone score at fellowship graduation. There was a statistically significant difference in the rate of attainment of milestone scores during training between groups in 10 of the 23 SCs (PC2, PC4, PC5, PC8, PC10, PC11, ICS3, PROF3, PROF4, and SBP2), where those who did not attain level 4 or above at fellowship graduation had a faster rate of attainment over time (p < 0.03).

TABLE 3.

Differences between mean first year and residency milestone scores in each SC among fellows who did and did not attain at least a level 4 milestone score at the time of graduation

SC	Mean first‐year milestone scores, n = 202		Difference, p‐value	Mean residency milestone scores, n = 40		Difference, p‐value
	Met level 4	Did not meet level 4		Met level 4	Did not meet level 4
PC1	3.3	2.8	0.57, <0.001	4.2	3.5	0.7, <0.001
PC2	3.1	2.6	0.55, <0.001	4.2	3.4	0.85, <0.001
PC3	3.3	2.9	0.49, <0.001	4.2	3.4	0.74, <0.001
PC4	3.2	2.8	0.42, <0.001	4.1	3.4	0.76, <0.001
PC5 a	3.1	2.6	0.5, <0.001	4.3	3.2	1.1, 0.1
PC6 b , a	3.2	2.9	0.3, 0.02	4.3	3.5	0.75, 0.03
PC7 b , a	3.3	3.1	0.22, 0.03	4.3	2.8	1.4, 0.02
PC8 b	3.4	3.1	0.3, 0.03
PC9 b	3.2	2.9	0.27, 0.07	4.1	1.0	3.1, 0.002
PC10 a	3.3	2.6	0.71, <0.001	4.1	3.0	1.1, 0.03
PC11	3.1	2.6	0.48, 0.002
M K1	3.4	2.7	0.68, <0.001	4	2.5	1.5, <0.001
SBP1	3.0	2.7	0.39, 0.004	4.2	3.4	0.85, <0.001
SBP2	3.2	2.7	0.49, <0.001
PBL1	3.4	2.9	0.46, <0.001
PROF1	3.5	2.8	0.67, <0.001	4.2	3.4	0.84, <0.001
PROF2	3.4	2.9	0.49, <0.001	4.1	3.4	0.72, <0.001
PROF3	3.4	2.8	0.54, <0.001
PROF4	3.0	2.5	0.48, <0.001
PROF5 b	3.3	3.0	0.29, 0.04
ICS1	3.5	3.0	0.51, <0.001	4.3	3.4	0.84, <0.001
ICS2	3.5	2.8	0.68, <0.001	4.1	3.3	0.85, <0.001
ICS3	2.9	2.6	0.34, 0.001

Only a subset of SCs overlap between residency and PEM fellowship.

Abbreviations: ICS, interpersonal and communication skills; MK, medical knowledge; PBLI, practice‐based learning and improvement; PC, patient care; PROF, professionalism; SBP, systems‐based practice.

^aDifference in residency to fellowship score did not meet statistical significance after correcting for multiple comparisons.

^bDifference in first year to fellowship score did not meet statistical significance after correcting for multiple comparisons.

DISCUSSION

In this national sample of PEM fellows, we found that a significant number of fellows did not attain the target level 4 for at least one of the 23 PEM SCs by the time of fellowship graduation. Our findings are similar to those in the 2018 ACGME Milestone Report, ⁸ which acknowledges that across specialties, not all graduates reach level 4 by the time of graduation. It presents medians and ranges for each SC; however, it neither presents raw numbers nor does it correlate milestone scores to demographic information or follow trainees over time to evaluate their trajectory of improvement. The report suggests that differences in assessment, rater scoring, and complexity of milestone language may be potential explanations for low scores at the time of graduation, as opposed to being solely reflective of trainee performance. ⁸ PDs may also place varying degrees of emphasis on the milestones; however, both the ACGME report and our study findings suggest that “straight lining,” assigning the same score for one trainee across all SCs, is uncommon. ⁸ In addition, some of the SCs, particularly in the areas of professionalism and interpersonal and communication skills, focus on innate behaviors that are more difficult to teach and more difficult to measure such as emotional intelligence, resilience, self‐awareness, and self‐motivation. Revised ACGME milestone language (Milestone 2.0) ¹⁶ , ¹⁷ will be released in 2022 and should help to address some of these issues.

The ability to correlate target milestone levels to success in independent clinical practice would allow for establishment of more definitive graduation benchmarks for PDs. ¹⁸ A group of family practice physicians developed milestones for faculty based on the ACGME core competencies. ¹⁹ In developing and validating these milestone criteria, they used a combination of levels 3 and 4 from the ACGME SCs as goal level for faculty at the assistant professor level. If faculty are not holding themselves to the ACGME target standards of level 4 for graduation, it seems unrealistic to hold trainees to that standard. Similar studies of PEM faculty have not been carried out across the broad range of SCs to date, but the application of the milestone standards to PEM faculty members would provide valuable information to develop clear, evidenced‐based expectations for trainees.

Furthermore, the 23 SCs on which PEM fellows are assessed include a broad range of skills such as advocacy and quality improvement that PEM specialists in various job settings might or might not be likely to utilize. PEM faculty that develop a particular career focus or “niche” do not need to develop mastery in every domain. For example, a medical educator who is not an expert in quality improvement would not typically be expected to lead a quality improvement team, as is the expectation in SBP1. Additional applications of ACGME milestone scores to faculty have been studied; however, most of these have evaluated faculty in only one domain (i.e., education) and do not mirror the broad content scope and the large numbers of SCs assessed for trainees. ²⁰ , ²¹ , ²²  This could prompt PDs to consider those SCs for which a target level of 4 or greater is appropriate within the current scope and duration of training.

Our data demonstrate that SBP1 and SBP2 are the SCs for which fellows most commonly do not achieve at least a level 4 milestone score. These results are consistent with the 2018 ACGME report and studies that show low minimum scores for both SBP1 and SBP2. ⁸ , ²³  These SCs relate specifically to quality improvement and the implementation of systems improvements and require a trainee to make an impact outside of their institution in order to achieve at least a level 4. Quality improvement requirements in fellowship training are relatively new and mentors or leaders in this area may not be present at every program. The aggregate of these findings raises the questions of whether demonstrating the skills and behaviors required to attain at least a level 4 is realistic for trainees. ²⁴ , ²⁵ , ²⁶ If a level 4 is the ideal target for trainees, and they are consistently failing to meet that target across a wide variety of training programs, an opportunity is presented for the development of a more standard, robust quality improvement curriculum and project requirements for PEM fellows. Given the time limitations of GME training, it may also be appropriate for some specialized SCs, such as SBP1 and SBP2, to accept a level 3 at the time of graduation, with the expectation that graduates are lifelong learners and will continue to develop in these areas in accordance with their career trajectory or a revision of some of the anchors could be considered such that the level 3 expectations become level 4 in Milestones 2.0.

In addition to the SBP SCs, we found that graduates frequently do not attain a level 4 for PBL1, PROF2 and PROF3, and ICS3. This is similar to the findings of the 2018 Milestone Report. ⁸  PBL1 level 4 includes an expectation that trainees utilize the electronic health record to improve the care of populations of patients, which, similar to quality improvement, lies out of the scope of most PEM faculty careers. PROF2 has an expectation that trainees will address patient's wishes using a complex framework including cost considerations. Prior literature has shown that PEM fellows receive little formal education on cost awareness. ²⁷  The language of ICS3 level 4 requires that the trainee self‐identify as an expert with “advanced knowledge and vast experience.” CCC committees may find it difficult attribute these skills to a trainee who is early on in their career. PROF3 demands a high level of emotional intelligence and maturity on the part of the trainee and is difficult to measure via traditional assessment tools. ⁸

Our findings also highlight several important factors related to the struggling learner. We found that fellows completing residency or their first year of fellowship with lower milestone scores tend to graduate fellowship with lower milestone scores. Understanding this trajectory may prompt PDs to initiate early remediation plans and to tailor individual learning plans to those SCs with lower milestone scores. Milestone scores from residency are made available to fellowship PDs during the first few months of fellowship training, but are infrequently accessed. One study found that only 40% of pediatric fellowship PDs download the final residency milestones for their fellows and only 27% of those actually use them for individualized education. ²⁸ Fellows in our study who began subspecialty training with lower milestone scores seemed to progress at faster rates compared to fellows who entered with higher milestone scores. This finding suggests there is significant potential for improvement with particular focus on these areas.

LIMITATIONS

Our study has several limitations. First, our study response rate from PEM programs was approximately 65%. While the geographical representation was inclusive and representative of the distribution of PEM programs nationally, our sample had slightly higher representation from the Northeast and less from the South. However, few geographic differences were found in our analysis and those found were without pattern. In addition, programs not reporting their milestone data may have had different outcomes when compared to those programs that did respond. Given the length of fellowship and the time frame of data collection, we did not have complete longitudinal data for all participants to compare milestone scores at fellowship entry to graduation across the whole cohort. Similarly, end of residency data was only available for a small proportion of trainees.

CONCLUSION

In our national sample of pediatric emergency medicine trainees, 67% of fellows did not attain a milestone score of at least a level 4 for one or more of the 23 pediatric emergency medicine subcompetencies at the time of graduation. Low scores at the end of residency or the first year of fellowship may predict difficulty in achieving a milestone score of at least a level 4 by fellowship completion. Further work is needed to better understand why pediatric emergency medicine fellowship graduates are frequently not meeting suggested milestone targets and if and how specific graduation milestones scores relate to competency in independent practice. As the ACGME develops updated milestone language in Milestone 2.0, consideration should be given to realistic goals for trainees, accounting for the broad scope of expertise of practicing physicians. Future studies should also determine for which subcompetencies it is essential that every graduate attain a level 4 to begin independent clinical practice.

CONFLICT OF INTEREST

The authors have no potential conflicts to disclose.

AUTHOR CONTRIBUTIONS

Study concept and design: Noel S. Zuckerbraun, Kathryn Leonard, Maybelle Kou, Jerri A. Rose, Cindy G. Roskind, Tien Vu, Aline Baghdassarian, Kelly Levasseur, Melissa L. Langhan. Acquisition of the data: Noel S. Zuckerbraun, Kathryn Leonard, Maybelle Kou, Jerri A. Rose, Cindy G. Roskind, Tien Vu, Aline Baghdassarian, Kelly Levasseur, Melissa L. Langhan. Analysis and interpretation of the data: Noel S. Zuckerbraun, Kathryn Leonard, Maybelle Kou, Jerri A. Rose, Cindy G. Roskind, Tien Vu, Aline Baghdassarian, Kelly Levasseur, Melissa L. Langhan. Drafting of the manuscript: Cindy G. Roskind, Kathryn Leonard, Melissa L. Langhan. Critical revision of the manuscript for important intellectual content: Noel S. Zuckerbraun, Kathryn Leonard, Maybelle Kou, Jerri A. Rose, Cindy G. Roskind, Tien Vu, Aline Baghdassarian, Kelly Levasseur, Veronika Shabanova, Melissa L. Langhan. Statistical expertise: Veronika Shabanova. Obtained funding: No funding. Administrative, technical, or material support: Melissa L. Langhan. Study supervision: Melissa L. Langhan.

Roskind CG, Leonard K, Baghdassarian A, et al. Pediatric emergency medicine fellows’ milestone evaluations: Do they all meet the targets for graduation?. AEM Educ Train. 2021;5:e10620. 10.1002/aet2.10620