Multisource Feedback in the Ambulatory Setting

Abstract

Background

The Accreditation Council for Graduate Medical Education has mandated multisource feedback (MSF) in the ambulatory setting for internal medicine residents. Few published reports demonstrate actual MSF results for a residency class, and fewer still include clinical quality measures and knowledge-based testing performance in the data set.

Methods

Residents participating in a year-long group practice experience called the “long-block” received MSF that included self, peer, staff, attending physician, and patient evaluations, as well as concomitant clinical quality data and knowledge-based testing scores. Residents were given a rank for each data point compared with peers in the class, and these data were reviewed with the chief resident and program director over the course of the long-block.

Results

Multisource feedback identified residents who performed well on most measures compared with their peers (10%), residents who performed poorly on most measures compared with their peers (10%), and residents who performed well on some measures and poorly on others (80%). Each high-, intermediate-, and low-performing resident had a least one aspect of the MSF that was significantly lower than the other, and this served as the basis of formative feedback during the long-block.

Conclusion

Use of multi-source feedback in the ambulatory setting can identify high-, intermediate-, and low-performing residents and suggest specific formative feedback for each. More research needs to be done on the effect of such feedback, as well as the relationships between each of the components in the MSF data set.

Introduction

Recognizing that the assessment of clinical competence is multifactorial,^1–4 the Accreditation Council for Graduate Medical Education (ACGME) has mandated multisource feedback (MSF) in the ambulatory setting for internal medicine residents.⁵ Multisource feedback, also known as a 360-degree evaluation, was initially developed by the manufacturing industry for quality improvement purposes, with a focus on individuals working in teams.⁶ As medical care has become increasingly team-based, MSF has become an accepted method of feedback for physician performance^4,7–13 and has been used to measure interpersonal communication, professionalism, and teamwork behaviors for residency trainees in multiple settings.^14–22 Yet few studies to date have combined the evaluation of the behaviors noted here with measures of residents' medical knowledge and clinical quality.

The purpose of this study was to describe the assessment and feedback procedures used at the University of Cincinnati to evaluate residents in the ambulatory setting. We were particularly interested in the rankings provided through various modalities, including self, peer, staff, attending physician, and patient evaluations, as well as clinical quality data, in-house testing, and in-training examination testing results. Relative rankings on each modality were used to guide feedback discussions with the program director and chief resident.

Methods

In 2006, the University of Cincinnati internal medicine residency program created the ambulatory long-block²³ as part of the ACGME Educational Innovation Project.²⁴ The long-block occurs from the 17th to the 29th month of residency and is a year-long continuous ambulatory group-practice experience involving a close partnership between the residency and a hospital-based clinical practice. Long-block residents follow approximately 120 to 150 patients each, have office hours 3 half-days per week on average, and are responsive to patient needs (by answering messages, refilling medications, and so forth) every day. Otherwise, long-block residents rotate on electives and research experiences with minimal overnight call.

The ambulatory team is divided into 6 mini-teams with 3 to 4 long-block residents and 1 registered nurse or licensed practical nurse per team. Additional support for the mini-teams includes a nurse practitioner, a social worker, a pharmacotherapy clinician, an anticoagulation clinician, and multiple faculty preceptors. The team uses a disease registry to track quality. Once a week, the entire team meets to review care delivery. The long-block MSF consists of self, peer, staff, attending physician, and patient evaluations, as well as clinical quality data, in-house testing, and in-training examination testing results.

Peer, staff, and attending physician evaluations consisted of 4 questions scored on a 5-point Likert scale. Each member of the care team was asked to rate each resident on 4 dimensions: patient care, teamwork, professionalism, and efficiency. Residents received their score in comparison to the class mean and range, as well as a rank for each question. The data in table 1 represent the final evaluation for this particular long-block; it was collected from 22 residents, 13 staff members (nurses, pharmacists, social workers), and 12 attending physicians. If an evaluator did not have enough contact to rate a resident, he or she abstained. On average, each resident received 28 ratings on each dimension. Written comments were also obtained from the care team but they are not included in this analysis.

Table 1.

Resident Rankings for Peer, Staff, Attending Physician, and Patient Evaluations

Patient evaluations were collected immediately after a clinical encounter using a 10-question adaptation of the Consumer Assessment of Health Plans Study (Clinician and Group Survey–Adult Primary Care)²⁵ (table 1). Patients indicated how often they had seen the physician they were rating (the average for this class was 4 visits by the time of the evaluation) and then they answered 9 questions using a 4-, 5-, or 10-point Likert scale. For this analysis, residents were evaluated by an average of 24 patients each.

Quality data (table 2) were collected using a disease registry. The main clinical data were drawn from the diabetes physician recognition program of the National Committee for Quality Assurance.²⁶ Residents reviewed their personal data monthly and received absolute values on each measure as well as ranks in comparison to their peers and the group as a whole. Residents also received composite rankings on overall outcome measures (A1c, low-density lipoprotein, and blood pressure) and process measures (smoking status documentation, eye examinations, foot examinations, nephropathy checks, and total patient load). Long-block residents managed 35 diabetic patients on average.

Table 2.

Resident Rankings for Knowledge-Based Testing, Clinical Quality, and Self-Evaluation

Information regarding testing came from 2 main sources. The first was the score on the in-training examination taken before the long-block. The second score came from a series of board-preparatory examinations taken over the course of the long-block. These tests covered the core of internal medicine as well as the main subspecialties. Residents received their scores in comparison to the class mean and range, as well as a rank for each test (table 2).

The self-evaluation was given at the start of the long-block and consisted of 45 questions encompassing the 6 core ACGME competencies.²⁷ Each question was scored on a 5-point Likert scale with 1 being an “area where I know that I need improvement,” 3 being an “area where I think that I perform adequately,” and 5 being an “area where I think that I am very skilled.” An average score for each core competency self-evaluation section was generated, and then this score was compared to the group as a whole (table 2). Residents received their score in comparison to the class mean and range, as well as a rank for each section.

table 1 displays the ranking data for each resident in the long-block class for peer, staff, attending physician, and patient evaluations. table 2 contains the ranking data for knowledge-based testing, clinical quality measures, and the self-evaluations. For both tables the Likert scale and class median and ranges are shown where appropriate. For this analysis, residents were sorted by overall performance rank by summing the peer, staff, attending physician, patient, testing, and quality ranks, with resident A being the highest rated and resident V being the lowest rated (tables 1 and 2). Specific rankings are out of 22 for each measure except for the quality measures, which are out of 24 because the data for 2 faculty members were included in the quality data set. We calculated Cronbach alpha for the 360-degree evaluations, the patient evaluations, and the self-evaluations and obtained values of .84, .92, and .89, respectively. figure 1 displays the ranking data from both tables in graphic form. Each column represents a specific resident (A through V). Data points closer to the top of the figure indicate higher relative rankings for a given evaluation entity than data points at the bottom of the figure. Data were also placed into radar graphs (figure 2) to display and compare MSF components for individual residents. Data points closer to the center of the graph represent higher rankings than data points near the edge.

Figure 1.

Spread of Multisource Feedback Data for Each Resident

Each column represents a specific resident (A through V). Data points closer to the top of the figure indicate higher relative rankings for a given evaluation entity than data points at the bottom of the figure.

Figure 2.

Radar Graphs for Overall Rankings for Each of the Major Multisource Feedback Categories

A, Ratings for residents A through C. B, Ratings for residents H, L, and Q. C, Ratings for residents T through V. Data points closer to the center of the graph represent higher rankings than data points closer to the edge.

Residents met with the program director and chief resident in the 4th and 11th months of the long-block to review the global MSF data. Residents were asked to reflect on the relative rankings in each data set and develop a learning plan to improve any deficits. The program director and chief resident used the data to guide their specific formative feedback. If problems were identified in any of the evaluation areas before these meetings, ad hoc conferences were convened.

The study received a waiver from the University of Cincinnati institutional review board.

Results

The MSF identified residents who performed well on most measures compared with their peers (residents A and B), residents who performed poorly on most measures compared with their peers (residents U and V), and residents who performed well on some measures and poorly on others (residents C through T) (tables 1 and 2, figure 1). Each relatively high-, intermediate-, and low-performing resident had a least one aspect of the MSF that was significantly lower than the other.

Examples of focused feedback for relatively higher rated residents included:

• Resident A (figure 2a) scored well in each of the performance categories, but had a low self-evaluation rating. This prompted a discussion regarding stress levels and making sure this resident's internal monologue was not excessively negative.

• Resident B (figure 2a) scored highly in every category except the in-training examination, leading to a plan for improved studying even though test scores were above average.

• Resident C (figure 2a) scored highly in most categories, but was rated lower by patients when compared with peers. This led to a review of the specific components of the patient evaluation that were lower (table 2–physician explains and physician gives instructions) and a plan for each.

Residents in the middle of the class received similar reviews and developed specific plans based on the data. Examples include:

• Resident H (figure 2b) scored highly in the peer/staff/attending physician efficiency category (table 1) and on test scores but was rated last in the class on overall patient evaluations. This led to a reflection that efficiency as seen by the care team could be negatively perceived by patients.

• Resident L (figure 2b) ranked first in the physician listens portion of the patient evaluation (table 2), but 19th in the physician is on time category (table 2) and 21st in the efficiency section of the peer/staff/attending physician evaluation (table1). This prompted a discussion of the potential tradeoffs of patient care: more time spent with patients could have potentially negative consequences on coworker perception within the practice.

• Resident Q's self-evaluation scores were first overall, first in knowledge, and second in patient care, but actual performance was 16th overall, 17th in knowledge, and 20th in patient care (figure 2b). This led the resident to reflect on calibrating self-assessment and performance. Other examples of significant discordance in MSF domains include resident N (2nd in patient evaluations, 18th in peer/staff/attending physician evaluation, and 21st in testing) and resident S (4th in peer/staff/attending physician evaluation, 16th in patient evaluation, and 20th in overall clinical quality). The program director used these data to develop specific plans of reflection and action.

Figure 2.

Radar Graphs for Overall Rankings for Each of the Major Multisource Feedback Categories (Continued)

Relatively lower-performing residents (residents T, U, and V, figure 2c) were ranked lower in almost every category, and the MSF identified a number of opportunities for improvement.

Figure 2.

Radar Graphs for Overall Rankings for Each of the Major Multisource Feedback Categories (Continued)

Discussion

When assessing the performance of a class of 22 residents, one might expect to encounter consistency within the performance measure outcomes. One possibility for data assortment could have been the higher-performing residents being ranked higher in most measures, and the lower-ranked residents ranking lower in most measures (eg, resident A is ranked highest in most categories, resident B is ranked next in most categories, resident C is ranked third in most, and so forth). Our data provide a more heterogeneous set of scores for each resident on the various components of the MSF. Although residents A and B are ranked higher in many components in the MSF and residents U and V are ranked lower, the majority of residents received both high and low rankings. The variation in ranking of the components of the MSF allows for specific, high-value formative feedback for each resident. Inclusion of clinical quality data and testing scores added an important dimension to the MSF. Limiting the evaluation to a single modality or only the behavioral components of a typical 360-degree evaluation (communication, professionalism, and teamwork behaviors) would have missed significant feedback opportunities for many residents.

Single end-of-rotation rating forms typically offer few high-value formative feedback opportunities for residents. Raters often use a global impression of the resident, creating a halo effect that fails to distinguish different dimensions of clinical care.^28,29 This problem exists for poorer performing residents but may actually be accentuated for higher performing residents; stellar evaluations often suggest that the resident should “keep on doing what they are doing” and little else. Use of MSF allows for focused, improvement-oriented feedback for even the highest rated residents. Although there could have been a halo effect for single raters on each type of rating instrument, we did not see this across rating instruments.

This study has several limitations. First, the extensive MSF detailed here was completed for one residency class within a unique ambulatory educational structure and may not be generalizable to other residencies. We included collection of feedback directly in the work flow of our long-block redesign, and the continuous contact between residents, staff, and patients made it relatively easy to collect a significant amount of feedback. Residencies with other ambulatory structures may find this degree of data collection unsustainable. However, use of MSF that includes the behavioral components of a typical 360-degree evaluation as well as clinical quality measures and test scores is generalizable. Second, most of the evaluation tools, with the exception of the Consumer Assessment of Health Plans Study questions and the in-training examination, have not been validated in research studies. However, regarding the peer, staff, attending physician, and patient evaluations, studies have shown an acceptable generalizability coefficient, with a minimum of 25 patient and 10 peer evaluations,⁴ that we were able to achieve for each resident. Third, the rankings presented in the tables represent normative comparisons and not absolute values. It is possible that this residency class as a whole could compare favorably or unfavorably to another residency class, illustrating the danger of using normative comparisons as the standard of measurement. However, the clinical quality measurement for the class as a whole bested many national benchmarks. In addition, the absolute value medians and ranges of the peer, staff, attending physician, and patient ratings were quite high overall, and it is not clear what the minimum rating should be for clinical competence. Rankings were therefore used to identify relative weaknesses in resident performance and to guide improvement strategies. Fourth, we did not show results of the feedback generated by the MSF data. The long-block lasts only 1 year. It takes time for meaningful data to be generated, and the most important evaluation period occurred in the 11th month of the long-block. However, both the residents and program director thought that these evaluations were among the most complete and constructive used in the residency program. Plans are underway to better evaluate resident progress while on the long-block and after returning to the inpatient services.

Conclusion

An ambulatory practice MSF system that includes self, peer, staff, attending physician, and patient evaluations, as well as corresponding clinical quality data and knowledge-based testing performance can identify relatively high-, intermediate-, and low-performing residents, and suggest high-value focused formative feedback for each. More research needs to be done on the effect of such feedback, as well as the relationships between each of the components in the MSF data set.