Abstract
Background
In contrast to historical feedback, which was vague or provided residents' numerical scores without clear meaning, milestone-based feedback is focused on specific knowledge, skills, and behaviors that define developmental trajectory. It was anticipated that residents would welcome the more specific and actionable feedback provided by the milestone framework, but this has not been studied.
Objective
We assessed internal medicine (IM) residents' perceptions of receiving feedback in the milestone framework, particularly assessing perception of the utility of milestone-based feedback compared to non–milestone-based feedback.
Methods
We surveyed a total of 510 IM residents from 7 institutions. Survey questions assessed resident perception of milestone feedback in identifying strengths, weaknesses, and trajectory of professional development. Postgraduate years 2 and 3 (PGY-2 and PGY-3) residents were asked to compare milestones with prior methods of feedback.
Results
Of 510 residents, 356 (69.8%) responded. Slightly less than half of the residents found milestone-based feedback “extremely useful” or “very useful” in identifying strengths (44%), weaknesses (43%), specific areas for improvement (45%), and appropriate education progress (48%). Few residents found such feedback “not very useful” or “not at all useful” in these domains. A total of 51% of PGY-2 and PGY-3 residents agreed that receiving milestone-based feedback was more helpful than previous forms of feedback.
Conclusions
IM residents are aware of the concepts of milestones, and half of the residents surveyed found milestone feedback more helpful than previous forms of feedback. More work needs to be done to understand how milestone-based feedback could be delivered more effectively to enhance resident development.
What was known and gap
Residents should welcome specific and actionable feedback provided by the milestone framework, but this has not been studied.
What is new
A slight majority of residents with experience with traditional feedback reported milestone-based feedback was more helpful.
Limitations
Single specialty study, lack of instrument validation, and participating programs that are part of a collaborative reduce generalizability.
Bottom line
Residents' perception of early feedback using the milestones is somewhat positive. More work needs to be done to enhance the utility of milestone-based feedback.
Editor's Note: The online version of this article contains the survey instrument used in the study.
Introduction
Milestones are developmentally based, specialty-specific knowledge, skills, and behaviors that residents are expected to demonstrate as they progress through training.1 They are intended to create a recognizable trajectory in the essential domains of competency. In the new accreditation system, residency programs' clinical competence committees must review each resident's developmental progress in a milestone format semiannually and report their conclusions to the Accreditation Council for Graduate Medical Education. It is expected that based on this process, residents will receive milestone-based feedback to identify strengths and areas for improvement.
Feedback linked to Milestones is based on specifically observable knowledge, skills, and behaviors; this is in contrast to historical feedback, which has often been vague and lacking a specific action plan for performance improvement.2 Predefined milestones could be used to assess and document a resident's developmental progression toward competency.3 Pilot studies involving internal medicine (IM) and surgery faculty have found construct validity in the content and responses on milestone-based evaluations as well as their perceived utility for resident assessment.4,5 It has also been suggested that providing milestone-based feedback would be beneficial to individual residents, and that residents would welcome this more specific and actionable feedback, compared to feedback using a traditional approach.6 To date, little is known about how IM residents perceive receiving milestone-based feedback.
As all specialties move forward with giving feedback in a milestone-based format, it is imperative that we understand the resident perspective on receiving such feedback. This study was designed to determine IM residents' perceptions of receiving biannual feedback in a milestone framework. Specifically, we aimed to determine if IM residents found milestone-based feedback useful as a method of self-assessment. In addition, we hypothesized that residents with previous experience of non–milestone-based biannual feedback would find milestone-based feedback more helpful.
Methods
The Northeast Milestone Collaborative (NEMC), a group of IM residency programs that convened in part due to an Association of Program Directors in Internal Medicine faculty development grant, consists of university and community programs in Connecticut, Massachusetts, and Rhode Island. The NEMC held several faculty development sessions on milestone use and the development of milestone-based evaluation tools. Faculty in the NEMC represent 14 residency programs.
Members of the NEMC were surveyed on whether or not they provided their residents with a biannual evaluation in a milestone format in December and January of the 2013–2014 academic year. The 7 programs that responded positively were included in the study and provided their residents' e-mail addresses so a survey instrument could be sent electronically via SurveyMonkey. The survey was developed by senior faculty with extensive experience in IM development, implementation, and feedback techniques through an iterative process. The e-mail announcement containing the link to the survey informed residents that participation was voluntary and anonymous. No compensation was provided.
Residents were asked if they were informed of milestones and if they received feedback in milestone format at their most recent biannual evaluation. Those who reported having received milestone-based feedback were asked a series of questions evaluating their perception of the usefulness of milestone-based feedback in identifying performance strengths, areas for improvement, and whether or not they were progressing along their anticipated trajectory. For these questions, a usefulness scale ranging from “extremely useful” to “not at all useful” was employed, with the middle option, “somewhat useful,” considered a neutral response. Second- and third-year residents were asked specifically whether receiving biannual feedback in a milestone-based format was more helpful than previous non–milestone-based feedback (survey provided as online supplemental material). For the purposes of analysis, scores on the usefulness scale of 1 and 2 (“extremely useful” and “very useful”) were combined and considered positive responses, scores of 4 and 5 (“not very useful” and “not at all useful”) were combined and considered negative responses, and a score of 3 (“somewhat useful”) was considered a neutral response. The survey was conducted between February 17 and April 7, 2014, with e-mail reminders sent to nonresponders every 2 weeks (with a maximum of 3 reminders). Program leadership was queried separately as to the methods used to deliver resident feedback.
The study protocol was approved by the Institutional Review Board at the University of Connecticut School of Medicine.
Descriptive statistics were used to summarize the data, and the Microsoft Excel Analysis Toolpak and SAS software (SAS Institute Inc) were used in data analysis.
Results
Participating institutions are listed in table 1. The survey instrument was sent to 510 residents, with 356 (69.8%) responding. All institutions had at least a 50% response rate. Chi-square analysis shows no difference when comparing the response rates of residents from the authors' programs with the response rates of residents from other programs (P = .22).
TABLE 1 .
Responders Versus Nonresponders by Institution
Due to the voluntary nature of the survey, not all respondents answered every question. table 2 indicates the distribution of respondents by year of training. Chi-square analysis showed no significant difference between responders and nonresponders based on postgraduate year (PGY; P = .09).
TABLE 2 .
Responders Versus Nonresponders by Postgraduate Year (PGY) Levela
A total of 90% (313 of 349) of respondents stated that they had been informed of the milestones, with resident meetings being the most common platform (79%, 245 of 310 respondents), followed by e-mail notification (51%, 158 of 310 respondents).
Of those who responded, 267 (77.4%) reported that they received feedback in milestone language at their biannual evaluation in December and January of the 2013–2014 academic year, whereas 78 residents (22.6%) responded that they did not, and were not asked to complete additional questions.
table 3 shows how useful respondents (by PGY) felt milestone-based feedback was in helping them identify areas of strength, areas of weaknesses, specific areas in need of improvement, and in providing them a sense of whether or not they were progressing at the expected pace. Logistic regression analysis shows no difference between how different PGY levels answered the questions in table 3, with odds ratios all between 0.71 and 1, and all confidence intervals crossing 1.
TABLE 3 .
Usefulness of Milestone Information in the Following Areasa
table 4 shows that 51% (102 of 200) of respondents agreed or strongly agreed that receiving feedback in a milestone-based format was more helpful than previous non–milestone-based biannual feedback.
TABLE 4 .
Milestone-Based Biannual More Helpful Than Previous Feedbacka
Discussion
Although our study was done prior to implementation of ACGME-required milestone reporting, it is encouraging that the majority of responding residents were familiar with the concept of milestones. Equally reassuring is that 48% (127 of 265) of respondents found milestone-based feedback extremely or very useful in helping them determine if they were progressing at the appropriate pace, with an additional 40% (107 of 265) finding it somewhat useful. Relatively few residents found feedback in a milestone-based format not to be useful in identifying areas of strength, weaknesses, and areas for improvement. It is also reassuring that different PGY levels viewed the usefulness of milestone-based feedback similarly, adding to the generalizability of the results.
A previous study suggested that milestones promoted a more uniform understanding of performance expectations, enhanced self-assessment, and led to the receipt of more specific performance feedback among residents.7 It follows logically that residents would find milestone-based feedback useful in tracking their professional development, a finding also reported in the pediatric literature.8
While we established that IM residents found a milestone-based framework for biannual evaluation at least as helpful as previous evaluations, our group anticipated a more positive response than 51% for strongly agree/agree. We had hypothesized that residents would find the specific, criterion-based narratives of the reporting milestones more helpful when compared to previous feedback. A limited qualitative study with verbal probing to explore narrative themes would clarify the perceived positives and negatives for the milestone format by contrasting it with previous feedback techniques.9 With this information, modifications could potentially improve resident perception of milestone-based feedback. Residents desire feedback for advancement, and it is our responsibility as educators to deliver feedback in a way that enhances education and advancement. Yet, despite the recognized importance of feedback in medical education, feedback practices are felt to be complex and suboptimal.10 Many factors add to the discrepancy between learners' and supervisors' perceptions of feedback and are beyond the scope of our discussion, yet we expect that residents may be less concerned with the format of feedback, and more with its delivery. The programs in our study all delivered feedback to residents in a similar fashion: written/electronic feedback in milestone language was provided to residents, followed by or associated with an individual, face-to-face milestone review and discussion of academic trajectory with either an associate or a program director. While the content of the feedback changed from previous non–milestone-based biannual evaluations, the process for providing that feedback has not changed. The NEMC endorses that there is great educational value in feedback for learners, and that educators do not yet understand completely how to best integrate milestone-based feedback into medical education. This is an area deserving of more study.
There are limitations to our survey-based study. We relied on expert validation of the survey questions by the authors because we wanted the survey to be conducted in a timely manner to reduce the impact recall bias would have on the results as much as possible. As a consequence, the survey questions were not tested with a sample target audience, which may have allowed for misinterpretation of questions. While the sample size and response rate for this study were adequate, we surveyed residents from programs that, by virtue of their participation in our collaborative, may have been more informed and well-versed in the IM milestones.
Studies looking at the effect of feedback have demonstrated variable outcomes with respect to behavior or performance change.10 Future studies should focus on whether the specific feedback provided to residents in a milestone-based format leads to easily recognizable changes in resident behavior and practice habits, which allow residents to demonstrate, and faculty to observe, trainee progression to competence.
Conclusion
Milestones are an integral component of the new accreditation system and may assist programs in determining where on the competency continuum a given resident stands. Our study provides the first glimpse of resident perception of the use of milestones as part of the biannual evaluation process and could be used to drive faculty development and resident needs assessments to improve the utility and effectiveness of milestone-based feedback. Half of the IM residents who were given feedback in a milestone-based format agreed that this type of format was more helpful than feedback given to them in a non–milestone-based manner, and only 18% did not find milestone-based feedback more helpful. Our results are reassuring, yet more work needs to be done to gain a better understanding of how such feedback could be delivered more effectively to enhance resident development.
Supplementary Material
Footnotes
Steven Angus, MD, FACP, is Program Director, Internal Medicine Residency Program, University of Connecticut School of Medicine; John Moriarty, MD, FACP, is Associate Program Director, Primary Care Program, Yale University School of Medicine, Yale-New Haven Medical Center; Robert J. Nardino, MD, FACP, is Program Director, Internal Medicine Residency (Saint Raphael) Program, Yale-New Haven Medical Center; Amy Chmielewski, MS, is Program Coordinator, Internal Medicine Residency Program, University of Connecticut School of Medicine; and Michael J. Rosenblum, MD, FACP, is Program Director, Internal Medicine Residency Program, Baystate Medical Center.
Funding: The authors report no external funding source for this study.
Conflict of interest: The authors declare they have no competing interests.
References
- 1.Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system—rationale and benefits. N Engl J Med. 2012;366(11):1051–1056. doi: 10.1056/NEJMsr1200117. [DOI] [PubMed] [Google Scholar]
- 2.Holmboe ES, Yepes M, Williams F, Huot SJ. Feedback and the mini clinical evaluation exercise. J Gen Intern Med. 2004;19(5, pt 2):558–561. doi: 10.1111/j.1525-1497.2004.30134.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Caverzagie KJ, Iobst WF, Aagaard EM, Hood S, Chick DA, Kane GC, et al. The internal medicine reporting milestones and the next accreditation system. Ann Intern Med. 2013;158(7):557–559. doi: 10.7326/0003-4819-158-7-201304020-00593. [DOI] [PubMed] [Google Scholar]
- 4.Aagaard E, Kane CG, Conforti L, Hood S, Caverzagie KJ, Smith C, et al. Early feedback on the use of the internal medicine reporting milestones in assessment of resident performance. J Grad Med Educ. 2013;5(3):433–438. doi: 10.4300/JGME-D-13-00001.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Borman KR, Augustine R, Leibrandt T, Pezzi CM, Kukora JS. Initial performance of a modified milestones global evaluation tool for semiannual evaluation of residents by faculty. J Surg Educ. 2013;70(6):739–749. doi: 10.1016/j.jsurg.2013.08.004. [DOI] [PubMed] [Google Scholar]
- 6.Green ML, Aagaard EM, Caverzagie KJ, Chick DA, Holmboe E, Kane G, et al. Charting the road to competence: developmental milestones for internal medicine residency training. J Grad Med Educ. 2009;1(1):5–20. doi: 10.4300/01.01.0003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Nabors C, Peterson SJ, Forman L, Stallings GW, Mumtaz A, Sule S, et al. Operationalizing the internal medicine milestones-an early status report. J Grad Med Educ. 2013;5(1):130–137. doi: 10.4300/JGME-D-12-00130.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Schumacher DJ, Lewis KO, Burke AE, Smith ML, Schumacher JB, Pitman MA, et al. The pediatrics milestones: initial evidence for their use as learning road maps for residents. Acad Pediatr. 2013;13(1):40–47. doi: 10.1016/j.acap.2012.09.003. [DOI] [PubMed] [Google Scholar]
- 9.Willis GB, Artino AR. What do our respondents think we're asking?: using cognitive interviewing to improve medical education surveys. J Grad Med Educ. 2013;5(3):353–356. doi: 10.4300/JGME-D-13-00154.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Telio S, Ajjawi R, Regehr G. The “Educational Alliance” as a framework for reconceptualizing feedback in medical education [published online ahead of print November 18, 2014] Acad Med. doi: 10.1097/ACM.0000000000000560. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.