Seen Through Their Eyes: Residents’ Reflections on the Cognitive and Contextual Components of Diagnostic Errors in Medicine

Abstract

Purpose

Diagnostic errors in medicine are common and costly. Cognitive bias causes are increasingly recognized contributors to diagnostic error, but remain difficult targets for medical educators and patient safety experts. The authors explored the cognitive and contextual components of diagnostic errors described by internal medicine resident physicians through the use of an educational intervention.

Method

Forty-one internal medicine residents at University of Pennsylvania participated in an educational intervention in 2010 comprised of reflective writing and facilitated small group discussion about experiences with diagnostic error from cognitive bias. Narratives and discussion were transcribed and analyzed iteratively to identify types of cognitive bias and contextual factors present.

Results

All residents described a personal experience with a case of diagnostic error that contained at least one cognitive bias and one contextual factor that may have influenced the outcome. The most common cognitive biases identified by the residents were anchoring bias (36, or 88%), availability bias (31, 76%), and framing effect (23, 56%). Prominent contextual factors included caring for patients on a subspecialty service (31, or 76%), complex illness (26, 63%), and time pressures (22, 54%). Eighty-five percent of residents described at least one strategy to avoid a similar error in the future.

Conclusions

Residents can easily recall diagnostic errors, analyze the errors for cognitive bias, and richly describe their context. The use of reflective writing and narrative discussion is an educational strategy to teach recognition, analysis, and cognitive bias avoidance strategies for diagnostic error in residency education.

Patient safety has received enormous attention since the 1999 Institute of Medicine Report, “To Err is Human.”¹ Although the target of most patient safety initiatives is the identification and improvement of poorly designed systems of care, the “human” or cognitive side of patient safety has received much less attention.² Diagnostic errors are common and costly, comprising 17% of preventable adverse events analyzed in the landmark Harvard Medical Practice Study³ and frequently ranking as the most common type of error in medical malpractice claims.⁴ Diagnostic errors are sometimes caused by systems errors but are equally likely to be caused by individual cognitive errors.⁵ Cognitive errors have been described in medicine within the context of clinical reasoning,⁶ diagnostic error,^7,8 and the patient safety movement.⁹ Unlike systems errors, which are easily identifiable and actionable, how doctors think remains a more difficult target to understand and influence.

It is difficult to solve the problem of diagnostic error in medicine. Systems improvements, such as the use of information technology to facilitate follow-up and tracking of test results by physicians and patients, have been suggested.^10–12 However, education on the cognitive underpinnings of diagnostic error appears equally important.^13,14 While strategies to teach clinical reasoning in the diagnostic process have been described,¹⁵ less is known about the inherent thinking patterns that humans use when processing information and optimal methods to teach the recognition and avoidance of cognitive bias. Since clinical reasoning habits are developed during medical school and residency, introduction of these concepts into medical training is critical.

In this study, we sought to capture internal medicine residents’ experiences with diagnostic error caused by cognitive bias, and to identify related contextual factors using reflective writing and narrative discussion as an educational strategy.

Method

Study design and participants

Within a 3-part longitudinal curriculum in diagnostic error and cognitive bias, we conducted an educational intervention among all second-year internal medicine residents at the Hospital of the University of Pennsylvania during the 2010–2011 academic year. Part 1 was a one-hour introductory curriculum on cognitive bias and diagnostic error. Part 2 occurred three months later and consisted of a reflective writing and narrative discussion session. Part 3, which occurred six months after Part 2, employed a web-based curriculum including matching exercises, videos with short-answer questions, and a final multiple choice test. In this article, we discuss the results of Part 2 of the longitudinal curriculum. At the beginning of the reflective writing and narrative discussion session, residents were given a five-minute summary of the prior curriculum and then randomly allocated into groups of four to six, each with a faculty facilitator. During the first ten minutes, we instructed them to write about a specific case example of diagnostic error with cognitive bias that they had either witnessed or participated in during their training. They were asked to include all circumstances that contributed to the team’s diagnostic decisions and describe what they learned and how this might change their future practice. Narratives were read aloud sequentially by each resident in the group. After each narrative, discussion was facilitated by the group leader. We audiotaped the entire session and transcribed it verbatim. We then used NVivo 9.0 software (QSR International, Cambridge, MA) for coding and analysis.

Four study investigators (AO, JR, JV, JM), two additional internal medicine residency program faculty, and three chief residents served as faculty facilitators. All facilitators participated in a one-hour training session led by an investigator (JR) in which the goals of the session and facilitation strategies were described. A guide that contained specific examples of how to mediate and guide conversation around sensitive diagnostic and cognitive errors was distributed during the training session.

The study was approved by the Institutional Review Board at the University of Pennsylvania. We obtained residents’ informed consent prior to their participation.

Development of coding scheme

Prior to the curriculum, we piloted the reflective writing exercise with five internal medicine residents who were not members of the participating class in order to develop and refine the coding scheme. To develop the first version of the coding scheme, we generated a list of known cognitive biases and de-biasing strategies (many of which were discussed within the curriculum) based on review of the literature and group discussion.⁷ Next, inductive codes were developed, expanding the preliminary coding scheme, by approaching the text-based sample with three questions: what cognitive biases were identified, in what context do residents report the occurrence of cognitive bias, and what de-biasing strategies did they discuss? An operational definition for each of the codes was entered into a coding dictionary. The resulting code list included cognitive biases, de-biasing strategies, and contextual factors (sorted into patient, system, and provider/team categories). We reviewed an existing taxonomy tool for diagnostic errors but we did not choose it as the primary framework for the coding scheme as this classification system is framed around the sequential steps in the diagnostic process rather than specific types of cognitive bias and contextual factors.^16,17

Analysis

We performed content analysis on all textual data.¹⁸ Two coders (AO and WGP) independently identified and highlighted in the small group transcripts and written narratives every codable unit of text (a statement that conveyed a singular idea) and assigned the codable unit to a code. Inter-rater reliability was first established through coding the pilot transcript and then checked at the midpoint of coding. Inter-rater agreement ranged from 80–95% in all categories. We resolved discrepancies through discussion between the two coders and a third investigator (JR). Simple matrices were generated to characterize which types of biases and de-biasing strategies were most commonly associated with specific contexts. We examined the statistical significance of the associations between biases and contextual factors using the chi-squared test.

Results

Forty-one second-year internal medicine residents participated in the study. Of these, 22 (53.7%) were male. All three tracks within the internal medicine program were represented: 29 (70.7%) categorical, 8 (19.5%) primary care, and 4 (9.8%) physician scientist pathway.

Group discussion results

All residents reported a case in which they cared for a patient and experienced a diagnostic error or delay in diagnosis due to cognitive bias. Eighty-five percent (35 of 41) provided strategies to prevent a similar cognitive error in the future (de-biasing strategies). All residents described stories that occurred in the hospital except for two primary care residents who described outpatient diagnostic errors.

Specific examples of cognitive biases were described in the narratives and most residents identified more than one type of bias. The most commonly described bias was anchoring (staying locked on to an initial diagnostic impression despite disconfirming evidence), which was mentioned in 36 (87.8%) of the narratives. Anchoring bias is closely related to premature closure of the diagnostic process, defined as accepting the diagnosis before full verification.⁷ Premature closure by definition occurs in all cognitive-based misdiagnoses, and could be directly caused not only by anchoring but also by many of the other biases. Therefore, premature closure was not coded as a bias in order to allow for more specific mechanisms of closure (e.g. anchoring bias, availability bias, framing effect) to be identified. Other commonly described biases are shown in Table 1. Several other biases were mentioned in relatively few cases and are not explicitly described.

Table 1.

Cognitive Biases Identified by 41 Internal Medicine Residents in Narratives and Discussion on Diagnostic Error, University of Pennsylvania, 2010^*

Cognitive bias	No. (%) of narratives in which individual biases were reported	Definition†	Illustrative quote
Anchoring	36 (87.8)	Staying locked on to a diagnosis made early on despite disconfirming evidence	“Once she came in, we had an impression of her…. I think that the giant bias in the room … if he’s got a thoracic primary, chest pain in a cancer patient with a thoracic primary is probably going to be cancer pain.”
Availability	31 (75.6)	Jumping to a diagnosis that comes to mind quickly because it is common, serious, recently encountered, or otherwise noteworthy	“The cognitive bias was that this was a guy who was infected and was hypovolemic and hypotensive when he presented and we see [acute tubular necrosis] patients all the time.”
Framing effect	23 (56.1)	Allowing the way the story is framed, whom the story came from, and where the patient is seen to influence diagnostic conclusions	“The patient [was presented as] a lung cancer patient with worsening shortness of breath who had been ruled out for a pulmonary embolism … presenting the story that way indicates that was something that they didn’t need to consider or work up anymore.”
Blind obedience	22 (53.7)	Showing undue deference to authority or technology	“I think I fell into that bias initially in that I deferred to authority probably for too long and I should’ve been more aggressive in pushing for what I felt the patient needed to have done.”
Unpacking	22 (53.7)	Failure to elicit all relevant information in establishing a diagnosis	“I attributed his back pain to just bony metastases. I never did a rectal exam. I sort of—I had mentioned to my intern to do it, but I didn’t follow up with it.”
Confirmation	20 (48.8)	Looking for clues to support the diagnosis rather than working the outliers into the diagnosis	“There was confirmation bias because we kept on studying his urine output and as we looked more into his [genitourinary] system we continued to find problems and treat them.”
Diagnostic momentum	20 (48.8)	Pushing forward diagnoses from previous encounters without evaluation of their accuracy or goodness of fit with current presentation	“But the point is she was hypoxic from heart failure, not from pneumonia, and I missed that even in the CCU because the patient was billed as pneumonia as a MICU admission through the ED.”
Visceral bias	20 (48.8)	Personal feelings toward the patient influence diagnostic conclusions	“Lung cancer is caused by smoking … people can have a tendency to blame the patient for their own disease.”
Overconfidence	19 (46.3)	Belief that one knows more than one actually does, or that a test/procedure is more accurate/effective than it actually is	“I think I know what’s best for this patient [but I did] not really want to admit that maybe I don’t…. I felt like I had to confidently give answers to everything along the way.”

^*Comments were derived from the discussion portion of a curriculum on experiences with diagnostic error that arose from cognitive bias.

^†Derived from Croskerry P. The importance of cognitive errors in diagnosis and strategies to minimize them. Acad Med. 2003; 78(8):775–780.⁷

Group discussion concentrated on the contextual factors and emphasized how they contributed to the error. Within the narratives, contextual factors were identified and classified into three broad categories: patient factors (15 factors reported in 68% of cases), environment factors (19 factors reported in 81% of cases), and team or provider factors (17 factors reported in 90% of cases) (Table 2). The most commonly reported contextual factor was caring for a patient on a specialty service (e.g., oncology, cardiology, pulmonary, gastroenterology, the medical intensive care unit, geriatrics service), as opposed to a hospitalist/general medicine service. This was reported in 31 of 41 narratives (76%). Of the non-specialty narratives, one described an event on a hospitalist service, two were from primary care settings (these were reported by primary care residents), and one from the emergency room. The identity of the clinical service or location could not be ascertained in the remaining six narratives. Other commonly reported contextual factors relating to the care team included a lack of interest in the patient’s case, prominent hierarchy within the team, overreliance on consultants, and the resident’s lack of confidence in his or her diagnostic skills.

Table 2.

Prominent Contextual Factors in 41 Residents’ Narratives and Discussion of Cognitive Bias and Error, University of Pennsylvania, 2010^*

Contextual factor	No. (%) of narratives in which factor was reported †	Illustrative quote
Team and provider factors	38 (92.7)
Specialty service	31 (75.6)	“That’s one of the downsides of specialization, I think, that we sort of get carried away with our specific area of what we’re doing and forget—not forget, but you’re just not attuned to these other things that are going on.”
Lack of interest in the patient’s case‡	13 (31.7)	“It wasn’t urgent or emergent that we sort of find out what was going on because I think there were other patients who were critically ill.”
Hierarchy	13 (31.7)	“It’s uncomfortable to approach an attending as a resident and say, ‘Actually, I don’t agree with that plan. Let’s get more information.’”
Over-reliance on consultants	12 (29.3)	“Sometimes [when you reach out to consultants] we can sit back and say, ‘Well, Heme is on the case and— Heme always looks at smears, and if they didn’t see anything, then it isn’t a problem.’ You can really shut [down] once you say, ‘I have an expert who has said that it is not this problem.’”
Lack of confidence	7 (17.1)	“I think this was maybe my second week So I was very not confident in myself.”
System and environmental factors	33 (80.5)
Time pressure	22 (53.7)	“If we can get to a diagnosis faster, it means that you have more time to take care of other patients.”
Recent transfer§	15 (36.6)	“This problem of outside hospital [transfers] and not looking at records happens to me all the time on call. The patients come in and it just takes a while. You have so much work. It’s hard to figure out what’s in [the records].”
Poor handoff	14 (34.1)	“The patient came straight from the ED and I don’t remember a whole lot of the sign-out that I was given because it was kind of quick.”
Lack of knowledge or guidelines	14 (34.1)	“There was not a standard approach to evaluating fever in someone who’s just returning from a certain part of the world, there’s like no framework.”
Insufficient information¶	14 (34.1)	“Because the patient was not accompanied by any family members, it was hard to determine the nature of his baseline cognitive function.”
Too many patients	13 (31.7)	“Coming up with a brand new differential for each chief complaint is just not feasible when it’s your seventh admission in an ICU.”
Chaotic emergency department (ED) environment	12 (29.3)	“In the ED, you have many patients to see. You’re often required to rely on heuristics because you have limited time.”
Delayed or inaccurate test result	12 (29.3)	“We were trying to wait for the final—like, confirmatory diagnosis, but it was taking so long that we were like, ‘We should start treatment,’ because she was having fevers also.”
Patient factors	28 (68.3)
Complex illness	26 (63.4)	“Patients having multiple problems that account for their symptoms is one that’s really challenging because our tendency and the way that we’re sort of taught to think is that we have this constellation of symptoms and we’re supposed to create a unifying diagnosis.”
Vague history from patient	13 (31.7)	“This was a patient who was not a complainer and so he was very reluctant to sort of tell us as he was declining.”
Chronic illness	10 (24.4)	“The first attending signed out to the second attending, “This is a very unfortunate gentleman. He looks like he’s from Auschwitz and there’s a lot of things that are lacking in his care.’”
Bad reputation or negative connotation	9 (22.0)	“Even before seeing the patient I felt like I had a negative picture of him in my mind.”
History of substance abuse or narcotic- seeking behavior	6 (14.6)	“You see somebody on methadone and immediately you think, ‘Oh, this is going to be a battle.’ It’s going to be a big negotiation about giving pain medications, and it really can cloud your decision.”
Patient refusing services	6 (14.6)	“‘We don’t have time to do all of this if you’re not going to be cooperative. So we’re going to go with this and it seems right,’ and it was working for a period of time.”
Poor prognosis	6 (14.6)	“He was very cachectic … we were surprised to learn that he still was full code status.”

^*Comments were derived from the discussion portion of a curriculum on experiences with diagnostic error that arose from cognitive bias.

^†Subcategory numbers and percentages do not total up to main category numbers or 100% because more than one factor could be reported per narrative.

^‡Lack of interest in the patient’s case either by the entire team or any team members.

^§Recently transferred from another hospital or between levels of care in the same hospital such as intensive care unit to medical floor.

^¶For example, lack of records from a referring hospital, lack of the appropriate test results, or inability to gather the pertinent information from the patient or family.

Environmental or system factors often contributed to the outcome in many of the narratives. These included time/workload imbalance, transfers between hospitals or from the Emergency Department, poor handoffs, lack of knowledge of clinical guidelines, and lack of access to past medical records. The most common patient factors that contributed to the outcome included a high degree of illness complexity, a vague history from the patient, chronic illness, a “bad reputation” from previous admissions, and a history of narcotic-seeking behavior.

Thirty-five (85%) of the residents discussed actions that could prevent similar errors in the future. Establishing a broader differential diagnosis (15, or 36.6%) and being aware of one’s own tendencies or predisposition for bias (15, 36.6%) were most frequently cited, followed by gathering data in a systematic fashion (13, 31.7%). Additionally, seeking other explanations for problems (11, 26.8%), acknowledging one’s “gut feelings” (10, 24.4%) and emotions (9, 22%), and knowing when to “slow down” (9, 22%) were also mentioned as useful de-biasing strategies to employ when making decisions.

Matrix results

Finally, we examined intersections among the biases and contextual factors. The relative frequency of cognitive bias reporting in the setting of individual contextual factors was quantified. Among the 31 narratives taking place on a specialty service, anchoring, availability (propensity for looking to the most cognitively “available” diagnoses, e.g., those seen most frequently or recently), and framing effect (allowing the way the story is framed to influence the diagnosis) were still the most commonly reported biases. Blind obedience (showing undue deference to authority or technology) was associated with caring for a patient on a specialty service. This bias was also reported in the setting of “lack of confidence” on the part of the resident making the diagnosis, cases in which consultants were integral members of the decision-making team, and cases in which team hierarchy or an intimidating attending was present. Framing effect was associated with a vague history from a patient, a report of being “too busy,” having too many patients, working overnight or fulfilling the role of “night float,” and the transfer of a patient from one service to another (or from one hospital to another). The unpacking principle (failure to elicit all relevant information in establishing a diagnosis) was often cited in the setting of a chronic illness and when a handoff occurred. Diagnostic momentum (pushing forward diagnoses from previous encounters without evaluation of their accuracy or goodness of fit with current presentation) was also commonly associated with handoffs. Visceral bias (personal feelings toward the patient which influence diagnostic conclusions) was associated with provider fatigue or provider lack of interest in the case. Finally, confirmation bias (the tendency to preferentially trust data that supports the initial diagnostic impression over data that refutes it) was associated with having too many patients or being “too busy.” These associations were not statistically significant (all chi-square statistics (df = 1) were < 3.84 and P > .05), which was attributed to small sample size.

Discussion and Conclusions

By performing qualitative content analysis on a reflective writing and narrative discussion session within our residency curriculum, we characterized the experiences of internal medicine residents with diagnostic error and cognitive bias. Although this was a relatively small study, it is the first study to document and analyze resident physicians’ experiences with diagnostic error and the context in which the error occurred. Two previous studies have evaluated curricula in cognitive bias,^19,20 but neither used a narrative approach. While the residents’ perceptions do not encompass all of the factors that may contribute to diagnostic error, their narratives provide valuable insight into the types of cognitive bias and associated contextual factors experienced frequently by trainees. Similar to previous research on experience with diagnostic errors among physicians after training,¹⁶ we found that residents easily recalled cases of cognitive error. Furthermore, they demonstrated thoughtful reflection on both the personal, cognitive influences as well as the external, contextual factors of the cases as they shared their experiences.

Certain cognitive biases were manifested more frequently than others in the resident narratives. Identifying a smaller core list of biases that internal medicine residents are most likely to encounter will assist medical educators in teaching about cognitive bias. The two most frequent biases elucidated from the narratives were anchoring and availability, which were identified in 88% and 76% of narratives respectively. This finding is consistent with other studies.⁵ Framing effect is a bias that contributes to error when diagnosticians are strongly influenced by how a problem or diagnosis is framed. This bias was noted in 56.1% of the narratives and is interesting to consider in light of the current resident duty hour restrictions.²¹ These restrictions have resulted in more resident handovers and thus more opportunities to frame a clinical scenario or diagnosis when handing the patient over to the next physician. Since handover training is now recommended for all residency programs,²² we should seize this training opportunity to also teach about the natural tendency for and pitfalls of the framing effect and how to avoid introducing this bias during handovers, particularly in cases of diagnostic uncertainty.

Contextual factors at the level of the patient, the clinical environment, and the healthcare team influence the likelihood of cognitive error.²³ Most of the environmental factors (e.g., time pressures, transfer, and handoffs) and patient-related factors are difficult, if not impossible, to modify in practice. However, it is possible that with instruction, residents can recognize both high-risk contextual factors as well as biased patterns of thought within themselves, in order to influence their decision-making. Indeed, how the resident physician thinks about the patient and the diagnostic process is perhaps the only piece of the clinical encounter that is modifiable in real time. During our facilitated discussions, residents suggested ways to reduce bias in their decision-making including reflective techniques (e.g., “knowing one’s tendencies”) and recommitting to the fundamentals of history-taking and physical examination to generate appropriate differential diagnoses.

Of interest, most of the errors described occurred on specialty services as opposed to general inpatient medicine services. We believe that this finding could represent the large amount of inpatient time that residents spend on specialty services in our institution. However, there may be factors specific to specialty services that facilitate cognitive bias. For example, specialty attending physicians may be prone to availability bias and generate a more limited differential diagnosis when presented with problems outside of their specialty in the same way that a generalist attending may have availability bias for less common subspecialty conditions. For many complex reasons, specialty physicians may focus only on the aspects of the case that fit within their discipline. Attending physicians with higher levels of clinical expertise for specific medical conditions could also contribute to a team environment with a greater degree of vertical hierarchy, even unintentionally, and therefore reduce the potential for the resident to serve as a cognitive “double check” by questioning the attending. This is important to acknowledge when we identify opportunities for improvement in team-based contextual factors. Finally, it is possible that patients on specialty services in our hospital are more likely to possess some of the patient-related contextual factors, including more complex or chronic illnesses, but given our knowledge of the complexity on the general medicine services at our institution, this seems less likely.

Although there is tremendous growth in the field of patient safety in medical education, almost all of these new curricula focus on the systems aspects of medical error^24,25 as opposed to the cognitive aspects and their contributing factors. This neglect may arise from the relative difficulty in identifying and fixing diagnostic errors, but may also stem from difficulty confronting and sharing what are perceived to be highly individualized mistakes.^26,27 We believe that important components of medical education include individual reflection on one’s own diagnostic errors, increased comfort among trainees and faculty around the topic of diagnostic error, and providing a vocabulary and framework for discussion on why these errors occur.

To our knowledge, the use of narrative discussion as a vehicle for education related to diagnostic error is novel. Mamede and colleagues have shown that forced diagnostic reflection improved diagnostic accuracy among internal medicine residents in complex cases.^20,28 The experience of writing facilitates reflection and may tap reservoirs of thought previously inaccessible to the writer.²⁹ The process of reading what one has written in a small group facilitates discussion of (potential) personal failure in a psychologically safe environment. Promoting a culture of psychological safety can be fostered by retrospectively reviewing diagnostic errors and delays during morbidity and mortality conferences. This culture should also be cultivated and encouraged prospectively on teaching rounds in order to confront our cognitive bias tendencies in real time. If a teaching rounds environment is created that is “low-risk” and free from hierarchy, the initial diagnostic impression can be continually challenged and reassessed.

The qualitative nature of our study is most powerful in its ability to generate hypotheses, and its descriptive nature places limits on the conclusions we may draw. Care must be taken not to assign ordinal importance to prominent themes in the study, and the cutoff threshold for inclusion was decided during the retrospective analysis performed by the group. Though the residents described many biases that were not taught in our previous didactic sessions, it is likely that our teaching allowed residents to develop comfort with the biases that we did teach and therefore mention them more frequently in the narrative sessions. Finally, we did not formally assess resident satisfaction at the conclusion of the narrative session. Informally, however, many residents reported that they enjoyed the experience. We plan to continue this activity as part of our patient safety curriculum.

Resident physicians, like all physicians, are prone to the effects of cognitive bias when making diagnoses. We have demonstrated that residents can easily identify these biases in retrospect and are willing to discuss why the error occurred and how to prevent similar errors in the future. We hope that use of reflective writing and narrative discussion will generate interest among educators as a way to engage faculty and trainees in conversations about diagnostic error and to enhance learning from one another’s mistakes.