A Structured Approach to Mitigating Cognitive Bias in Educational Assessment

Abstract

Visual Abstract.

Medical educators encounter numerous competing responsibilities and a high volume of assessments, which often create barriers to providing high-quality, timely feedback.¹ Resident assessments involve increased cognitive load when evaluating clinical reasoning and decision-making, particularly for struggling residents.^2-4 To mitigate cognitive load, faculty often resort to heuristics to facilitate cognitive efficiency.⁵ However, reliance on heuristics can result in increased cognitive bias, leading to inaccurate assessments, missed interventions, and negative educational and patient outcomes.^2,6 To reduce the effect of cognitive biases resulting from educators’ cognitive workload, a systematic approach to debiasing is needed to ensure high-quality resident assessment.

While more than 100 heuristics and cognitive biases exist, recent studies have identified 3 categories: mindware gaps, valuation biases, and anchoring and adjustment biases.^7,8 These categories provide a framework for systematically mapping, evaluating, and mitigating bias on resident assessment (Table). Mindware gaps result from a lack of understanding, logic, or analytic skills, leading to less-than-ideal assessments. For example, if a resident contributed to a poor patient outcome, their faculty may overemphasize this single significant situation when providing a global assessment. Valuation bias refers to a systematic error in assessing value, worth, or an outcome. Faculty may fall susceptible to valuation bias by providing an inflated assessment of procedural skill to a charming or physically attractive resident. Anchoring and adjustment bias refers to the tendency of relying too heavily on initial information when making decisions or quantitative estimates. For example, if a resident showed up late to their first day of a rotation, the faculty would need to ensure they did not anchor on this negative first impression when providing summative feedback. Understanding these categories enables educators to recognize and mitigate bias, leading to more accurate and fair assessments.

Table.

Dimensions of Heuristics and Biases and Common Associated Biases in Assessment

Dimensions of Heuristics and Biases	Heuristics and Biases	Definition and Example
Mindware gaps	Availability bias	Assigning too much attention to memorable events due to frequency, recency, or perceived importance. Example: Despite an average performance throughout a rotation, a faculty provides a positive assessment after a resident discovers an uncommon diagnosis.
	Group attribution error	Falsely attributing one person’s actions to that of a group, or vice versa. Example: Assuming students pursuing surgical specialties are not invested in their education while rotating on medical specialties, or vice versa.
Valuation biases	Bandwagon effect	Adopting the opinions, beliefs, attitudes, or behaviors of the group. Example: A faculty mentions an issue at a meeting. Then others begin sharing other minor concerns that would not have been discussed otherwise.
	Horn and halo biases	Generalizing opinions of a person or group based on a single trait or characteristic, either good or bad. Example: A resident may receive inflated assessments in numerous areas of competency, simply based on excellent communication skills.
	In-group bias	Preferential treatment to those in our group over other groups. Example: Preferring to mentor residents with a similar background or interests.
Anchoring and adjustment	Anchoring bias	Relying too heavily on an initial observation or piece of information without acknowledging new or changing information. Example: After a low PGY-1 in-training examination score, a faculty member might “anchor” on unsatisfactory medical knowledge and assign scores that are lower than deserved later in residency.
	Confirmation bias	Focusing on data to support preconceptions and overlooking evidence to the contrary. Example: A faculty member with a negative opinion of a resident finds errors in documentation and clinical reasoning, failing to acknowledge positive feedback regarding clinical performance.

Abbreviation: PGY, postgraduate year.

Note: Adapted from Dickey et al² and Ceschi et al.⁷

The evidence-based classification of heuristics and biases provides a foundation for developing a systematic approach to cognitive debiasing. Building on this foundation and incorporating additional literature,^7,9-13 we developed a structured approach to minimize the influence of cognitive bias. This approach, known as SKAIR, enables faculty to account for biases to improve resident assessments, without having to identify individual heuristics and biases. The SKAIR Mnemonic consists of 5 key steps:

1. Slow Down. Cognitive biases influence assessment when educators lack time to think analytically. Ensure there is sufficient time to develop an assessment and avoid pressure to “just get it done.”

2. Know Your Bias. The more educators are aware of the 3 dimensions of bias, the more their effects can be minimized without needing to name all 100+ described heuristics and biases.

3. Consider the Alternative. To evaluate an initial impression of a resident, contemplate the possibility that the initial impression is wrong. Develop an educational differential.

4. Get More Information. Corroborate the initial assessment by reviewing objective data, extending resident observations, or comparing the initial assessment to that of a trusted colleague.

5. Reflect. Metacognition—being aware of one’s own thought processes and the factors influencing cognition—is an important tool in cognitive debiasing.

All 5 elements of the SKAIR Mnemonic are designed to encourage educators to reflect on their assessments. However, reflecting on external factors contributing to high cognitive load, such as stress, fatigue, or burnout, can help educators be more aware that cognitive biases may be influencing assessments.

Program leadership is responsible for ensuring faculty have the tools necessary to advance their educational knowledge and skill. Introducing the SKAIR Mnemonic through faculty development could allow program faculty to advance in their own Accreditation Council for Graduate Medical Education Clinician Educator Milestones, particularly Recognition and Mitigation of Bias (Universal Pillar 3) and Learner Assessment (Educational Theory and Practice 3).¹⁴ When presented at national conferences, 97% of family medicine residency educators (128 out of 132) and 98% of multidisciplinary medical educators (46 out of 47) agreed on post-presentation surveys that the SKAIR Mnemonic was a highly valuable approach to mitigating cognitive bias. While programs and faculty have multiple competing priorities, integrating this method of debiasing across specialties could help strengthen the culture of assessment.

Case Example: The Faculty Meeting

Your residency program’s monthly faculty meeting included 20 minutes to review the progress of all 15 residents, and Jeremiah was the first intern discussed. A senior faculty member expressed frustration about a recent precepting encounter when Jeremiah gave a disorganized, rambling presentation of a 90-year-old woman with dizziness. Not only was Jeremiah unable to develop a differential diagnosis, but he also failed to address advance care planning. A junior faculty member then commented about a situation weeks ago when Jeremiah was unable to choose the correct stress test for a patient with exertional chest pain. The residency coordinator mentioned that she saw Jeremiah sitting at a bar drinking over the weekend and hoped he didn’t have a drinking problem. As the newest member of the faculty, you have had several positive encounters with Jeremiah and have seen that he is consistently prepared for clinic, organized, and efficient. You remain silent. The faculty group decides to pursue a remediation plan.

SKAIR in Action

Slow Down. Was Jeremiah well served by limiting the review of his performance to such a brief discussion? Processing too much information without appropriate time can make us vulnerable to cognitive bias.

Know Your Bias. A mindware gap such as availability bias, where too much attention is given to memorable events, may be contributing to the assessment of a recent, frustrating precepting encounter occurring earlier that day. In addition, having the most senior faculty member speak first has led to the bandwagon effect, a valuation bias that reflects the tendency to converge on unanimity to avoid conflict.

Consider the Alternative. Were other factors at play that might have influenced Jeremiah’s performance during these encounters?

Get More Information. Were dissenting opinions and observations adequately explored? Was there a review of written rotation feedback? Perhaps scheduling direct observations would provide more reliable information.

Reflect. What other factors may be contributing to faculty members’ cognitive load, leaving them susceptible to bias? Did faculty rush to the meeting after providing clinical care? Are there other topics on the agenda that are distracting faculty from the discussion of Jeremiah’s performance?

The case example illustrates how cognitive biases commonly influence assessments in medical education. Using the SKAIR Mnemonic can provide faculty a structure for identifying and mitigating biases in a 5-step process. While this structured approach has been rated as valuable for cognitive debiasing, future work should include evaluating the SKAIR Mnemonic to determine its actual impact on debiasing and overall efficacy in reducing bias.

Conclusion

Medical educators have a duty to provide accurate, debiased assessments that are based on objective, reproducible judgments. To ensure residents receive the most accurate and fair assessments, faculty may benefit from employing structured strategies, such as the SKAIR Mnemonic. Introducing this approach into a program can provide a quick reference tool for faculty to use during formal assessments, such as during Clinical Competency Committee meetings, and informal faculty discussions. Through structured debiasing, accurate resident assessment can help avoid unnecessary remediation while also ensuring appropriate resident advancement.