Cognitive Demands and Bias: Challenges Facing Clinical Competency Committees

The Clinical Competency Committee is discussing what milestone level of professionalism to assign a resident. The discussion is reaching its end, and the Committee is about to designate a strong Level 4. Then 1 faculty member questions the rating, and mentions that the resident was 10 minutes late to clinic last week. Another member states that the resident missed a didactic session 3 months ago. A third faculty member pipes up and recalls that the resident didn't read an assigned article last year. A fourth member says, “Not good. Let's give her a Level 3.” What happened?

In 2013, the Accreditation Council for Graduate Medical Education (ACGME) charged training programs to form a Clinical Competency Committee (CCC) to develop a method to assess residents' progression toward unsupervised practice, using the ACGME Milestones and based on residents' evaluations. The CCCs form 1 element of an integrated assessment system¹ and are an essential component of ACGME's new accreditation system.² The CCCs are expected to review residents' development semiannually and to assign and report milestone levels.²

Milestones are explicit outcome data of developing competencies. In their determination of appropriate, resident-specific milestone levels, CCCs integrate information gleaned from rotation evaluations, specific competency assessments, tests, and other data. The CCC members deliberate and determine the milestone level to report to the ACGME. The goal of this process is to improve feedback to residents, to “enhance credibility of judgments about resident/fellow performance,”¹ and to aid in program improvement and faculty development.

In CCC deliberations, members often need to make more than 20 milestone-level determinations per resident. This represents a high cognitive load, with the potential for decision-making fatigue, which degrades decision-making processes.³

During assessment and decision making, bias can and does occur across settings.⁴ A study of judicial decisions found that after eating, court judges give more lenient sentences.⁵ When asked if this is true, judges denied the tendency. These judges, while striving to be impartial, demonstrate the unconscious operation of biases.⁵ Another example comes from the marketing of wine. The list price of a bottle of wine influences the subjective appraisal of its taste.⁶ Bias also occurs in elections. People correctly predict election winners in fields of unknown candidates, based solely on their appearance.⁷ Even in the appraisal of the scientific merit of journal manuscripts, reviewers display implicit bias.⁸

Bias may be hardwired.^3,6 In the example of wine pricing, the more expensive the wine, the more it is experienced as pleasant, the more there is activation in the orbitofrontal cortex.⁶ Just altering the price of wine alters drinkers' experience and neuronal activity.⁶

Bias is normal and common.⁹ Bias typically has a negative connotation, but it does help us navigate our environment.¹⁰ Constantly monitoring for bias is not easy,¹⁰ especially during CCC meetings. (See the table for examples of bias that can occur during CCC deliberations.)

Table.

Examples of Bias That Can Occur During Clinical Competency Committee (CCC) Deliberations

There are several strategies to help minimize the effect of bias on decision making.¹⁰ Recognition of bias and motivation to change are key initial steps.¹¹ For CCC members, faculty development exercises may help raise awareness and build a shared vocabulary regarding bias (information provided as online supplemental material). Having CCC members role play scenarios written to elicit bias and label the types of bias is an option. This has been used successfully both within a local CCC and at a national workshop.¹²

To optimize the value of the data, many CCCs have turned to commercial resident management systems (RMS) to organize assessments and evaluations. Alternatively, a homegrown RMS can provide a platform tailored for CCC use. One such program, Harvard South Shore–Milestones Dashboard (HSS-MD), is a Microsoft Excel–based dashboard that is highly visual, adaptable, efficient, portable, and free. It graphs competency development over time, which enables committee members to detect problems in residents' progress. It handles all residents' data equitably and transparently. It reduces concerns about summarizing objective observations, cognitive overload, and biased decision making. The HSS-MD generates a mean that serves as an initial milestone level for CCC deliberations.¹

Committee members must give reasons for overriding the precalculated milestone subcompetency level, and, in the course of the discussion, the CCC has time to consider potential bias. For example, if the HSS-MD derived level for a resident's subcompetency Patient Care 1 is “2,” and a faculty member thinks it should be “3.5,” then the faculty member must justify the higher level. If the faculty member says, “Because yesterday I saw Dr. X perform an efficient and compassionate H&P,” then others can ask if availability bias is occurring. The faculty member is disregarding 6 months' worth of data in favor of 1 recent patient interaction. Other committee members might respond with, “It sounds like Dr. X performed a stellar examination yesterday. I wonder if focusing on that examination, rather than including all the data from rotations, might be an example of availability bias. Perhaps ‘2’ is the appropriate level for now, and it is encouraging that Dr. X can also perform at a higher level at times.” By arguing for a different level and labeling the potential bias, committee members can become more aware of their own biases and more thoughtful in their comments. In our experience, CCC members are open to learning and talking about bias, and they appreciate the opportunity to reflect on their statements.

To keep the issue of bias central to the discussions, CCCs can print out copies of examples of bias (table) and refer to them during the meeting. This can help build a shared vocabulary and awareness.

A Clinical Competency Committee cannot avoid the challenges of cognitive demand or bias. No individual or group assessment can be entirely objective. By being more mindful of the potential for bias and by developing a shared vocabulary to describe the bias, CCCs can mitigate its effects. The use of a clear protocol and an easily assimilated database can also help decrease bias.