Setting and Problem
Competency-based medical education requires faculty members to assess clearly defined outcomes of learning over time. Unfortunately, assessment of competence is fraught with many difficulties, including the ability of faculty to accurately translate clinical performance into helpful feedback for the learner.
In 2011, we created an assessment system consisting of entrustment ratings of discrete work-based tasks called observable practice activities. Faculty members are asked to provide written comments justifying a given entrustment level, as well as specific suggestions for improvement. Despite multiple faculty development efforts that include e-mails, videos, narrated PowerPoints, and in-person presentations, a significant number of faculty members still use the assessment system incorrectly.
Intervention
We created a feedback tool for the end-of-rotation assessments that faculty members provide for residents. We began by defining behaviors we desired in our faculty and then generating a rating scale for each behavior (Figure).
Figure.
Feedback on Feedback Tool
Once a month our education team reviews all assessments submitted by faculty members. Each reviewer assesses 7 to 10 assessments monthly. The feedback tool (Figure) consists of 5 columns with numeric values, from 1 (poor) to 5 (excellent), and an average score for each review is calculated. All faculty members receive an e-mail with feedback on their assessments, and those with average scores less than 2 (indicating at least 1 score in the lowest performance column) are invited to an in-person meeting with the program director. All scores are reported to the chair of the department as part of each faculty member's yearly performance review.
Outcomes to Date
We completed 1149 feedback forms for 202 faculty members over 2 years. The average score per faculty assessment episode was 3.28 (median = 3.25). A total of 9% (106 of 1149) of assessments received an average score of less than 2, and 26% (52 of 202) of faculty members received an average score of less than 2 for at least 1 assessment (most faculty had more than 1 assessment).
Typical narrative comments delivered to faculty members included:
“You rated Dr. X's OPAs at a level 4 [entrusted to perform without supervision] throughout the evaluation. In order to justify such a high rating you should specifically note why and how an intern could perform at such a high level.”
“‘Read more' is too vague. What particular diagnoses did you observe Dr. X struggle with?”
“You gave this resident the same score for every question. It is unlikely he is equally skilled in all areas. You have also answered every box: Did you see all the skills? If not, then don't force the answer—leave it blank. We would rather have a few high-quality responses on what you observe, than low-quality responses on things you did not observe. A member of the education team will be contacting you to discuss further.”
Of the 52 faculty members who had at least 1 feedback form with an average score less than 2, 24 had subsequent assessments available for our team to review. Of these, 58% (14 of 24) had no subsequent ratings less than 2, while 42% (10 of 24) had at least 1.
Many faculty development initiatives employ general education sessions, but behavior change requires direct feedback and coaching. We developed a process that identifies faculty members who may benefit most from these interventions with regard to written feedback performance. Further study is needed to understand and optimize this process.