Direct Observation Reassessed

The Challenge

Direct observation is embedded in the clinical learning environment, as faculty physicians supervise trainees to ensure high-quality patient care and assess their readiness for progressive autonomy. Direct observation provides opportunities for the assessment of learning—what a resident does in real time with real patients. It also can be used to assess for learning, to identify gaps between what a resident does well and what he or she needs to improve. As part of direct observation, faculty physicians typically are asked to complete an assessment based on their observation of a resident's performance in a real-life clinical context; faculty “are” the measurement tool.^A

What Is Known

Direct observation is used in 90% of graduate medical education programs,^1,B and the education community can improve its value. Addressing the following common issues^C–E can reduce measurement errors² and enhance the accuracy of direct observation:

1. Rating tools are often too long, include ambiguous items, or do not assess what is intended.

2. The rating process is cumbersome, time consuming, includes too few raters, or is not integrated with the program's larger assessment system.

3. Raters may have bias about a resident's performance based on prior experiences,³ may delay completing or fail to complete ratings, not understand the assessment process, or lack competence themselves in the skill being assessed.^F,G

4. Limited variance in ratings may occur, with overall high ratings or identical scores across different items, due to a lack of rater training or because rating forms are too long.

Effective forms for direct observations can be quite simple. For example, the Zwisch scale, which has 2 questions and a measure of case complexity, correlates well with longer instruments,⁴ and a single-item rating of operative performance is equivalent to longer scales.⁵ Also, narrative descriptors outperform numeric scales.⁶ Sampling the right number of faculty is key: using 7 to 10 different faculty members minimizes rater idiosyncrasies and is feasible.⁷ In sum, it is important to align the direct observation process with how faculty form judgments to simplify the process and to gain as many assessments as possible.^8,H–M

Rip Out Action Items

Program directors should:

1. Educate faculty, residents, and staff on the importance and practice of direct observation.

2. Mentor faculty on their role as assessor and how to best observe, judge, and represent their judgment through narratives and rating scales.

3. Simplify the process of direct observation and associated rating forms.

4. Clarify and distinguish the formative role (“How is the resident doing this activity at this moment?”) from a summative assessment that uses multiple ratings, along with other data, to make judgments about trainee progression in the program.

How You Can Start TODAY^9,N,O

Partner with your faculty to enhance their use of direct observation. Help faculty refine their role of teacher to include performance assessor or judge. Provide intermittent educational boosters and reminders about key features of direct observation.

1. Acknowledge that we all have biases by anticipating the typical sources of bias to mitigate them.

2. Ask faculty to assess the trainee's performance observed during a specific, recent encounter (not the resident's performance over the preceding day/week/month/rotation or a past experience with the trainee).

3. Emphasize frequent, brief assessments using a few items and narrative comments (including strengths) to improve identification of performance issues.

4. Seek assessments from multiple raters for summative judgments in order to limit bias.

5. Leverage opportunities for direct observation when faculty are already directly supervising residents.^P

6. Complete ratings within 72 hours of an encounter as after that time the rating accuracy decreases.^Q

7. Simplify rating forms and associated processes.

What You Can Do LONG TERM⁹

1. Continuously build a culture that values direct observation. Make the process transparent to both residents and faculty. Start by identifying several committed faculty champions.

2. Involve faculty in cooperative discussions to identify opportunities for improving its use, since the faculty are the “instrument” of assessment. Do faculty differentiate between direct observation as formative versus summative assessment, and how does this influence their assessment?

3. Use faculty champions to assess the strengths and opportunities in a real-time process. Do raters assess what they actually observe? Do the residents and Clinical Competency Committee consistently receive direct observation information? If so, how useful is that information?

4. Review direct observation assessment forms. Do other programs in your specialty or other programs at your institution use a simpler form or process? Identify which information is most needed for resident progress decisions, and design the assessment form to capture that information while deleting other items. If 1 type of form is more useful, can you eliminate the others? Always leave space for free text, and ask faculty to describe trainees' strengths and provide suggestions for improvement. When available, incorporate items that correlate with clinical outcomes.

5. Prioritize 1 or 2 direct observation improvements such as to increase its use on a particular rotation, or to simplify a form to capture its assessment. Develop an intervention that outlines goal(s), specific action plan(s), time line(s), and how success will be measured. Monitor to ensure that the intervention meets its goals.

6. Develop a reliable, feasible, simple system to facilitate assessment form completion and to ensure that trainees receive feedback. Apps have increased the number of evaluations, improved trainee satisfaction, and helped faculty learn milestones.^R–V Video may also be a beneficial tool.^W

7. Benchmark assessors' performance and provide feedback on a regular basis.

8. Consider use of direct observation for scholarly work as a key outcome measure for intervention studies.