MIND THE OVERLAP: HOW SYSTEM PROBLEMS CONTRIBUTE TO COGNITIVE FAILURE AND DIAGNOSTIC ERRORS

Abstract

INTRODUCTION:

Traditionally, research has examined systems- and cognitive-based sources of diagnostic error as individual entities. However, half of all errors have origins in both domains.

METHODS:

We conducted a focused ethnography of inpatient physicians at two academic institutions to understand how systems-based problems contribute to cognitive errors in diagnosis. Medicine teams were observed on rounds and during post-round work after which interviews were conducted. Field notes related to the diagnostic process and the work system were recorded, and findings were organized into themes. Using deductive content analysis, themes were categorized based on a published taxonomy to link systems-based contributions and cognitive errors such as faulty data gathering, information processing, data verification and errors associated with multiple domains.

RESULTS:

Observations, focus groups and interviews of 10 teams were conducted between January 2016 and April 2017. The following themes were identified: (1) challenges with interdisciplinary communication and communication within the electronic medical record contributed to faulty data gathering; (2) organizational structures such as the operation of consulting services in silos promoted faulty information processing; (3) care handoffs led to faulty data verification, and; (4) interruptions, time constraints, and a cluttered physical environment negatively influenced multiple cognitive domains.

CONCLUSIONS:

Systems-based factors often facilitate and promote cognitive problems in diagnosis. Linking systems-based contributions to downstream cognitive impacts and intervening on both in tandem may help prevent diagnostic errors.

INTRODUCTION

Diagnostic error, including delayed, wrong, or missed diagnoses has become a central focus within the patient safety movement [1, 2]. Although estimates vary, studies suggest that a diagnosis is wrong between 10–15% of the time [3, 4], and that every human, at least once in their life, will be exposed to a diagnostic error [2], Thus, an urgent and yet unmet need to improve diagnostic reasoning, accuracy, and efficiency exists.

Current research has attempted to explain diagnostic errors in two domains: cognitive or system-based causes. Graber et al. published a useful taxonomy for such errors, separating cognitive errors into those stemming from faulty knowledge, faulty data gathering, faulty information processing, and faulty information verification and systems errors into technical and organizational [5]. Examples of cognitive errors include anchoring bias (where health care providers “lock on” to a diagnosis made early in the diagnostic process despite contrary evidence), or availability bias (where the likelihood of the diagnosis relates to how readily it comes to mind) [6]. Conversely, lack of standardized policy, discontinuous care, and interruptions are often cited as system based problems that contribute to errors in diagnosis [5, 7–9]. While a convenient classification scheme, these two domains never occur in isolation; in fact, one retrospective study of 93 diagnostic errors identified nearly 6 system- or cognitive-based factors per case, with almost half stemming from both domains [5].

While the contextual contributions to cognitive error as well as tools to identify and organize systems and cognitive contributions to error have been previously studied [10, 11], efforts to understand diagnostic errors and interventions to curb them have often treated systems and cognitive errors individually [12, 13]. The drawbacks of this segregated approach (from a human factors perspective) have been recently highlighted in the literature [14]. Despite this, there continues to be a paucity of data exploring the interrelatedness of these domains – or, more specifically, how system factors may contribute to cognitive missteps in diagnosis – exist. This gap is important as some systems-based issues might be more amenable to change, thus preventing cognitive errors downstream. Therefore, we aimed to evaluate systems-based contributions to cognitive error so as to identify high-yield interventions to curb diagnostic errors in patient care.

METHODS

We designed a focused ethnographic study to examine factors contributing to diagnostic decision-making in the inpatient setting. Based on qualitative methods, a focused ethnography is the practice of studying a group of people who have a “specific knowledge about an identified problem,” with the goal of providing useful application of the findings to the setting under study [15].

Settings and Participants

We observed members of inpatient internal medicine teams at two affiliated, academic medical centers. Teaching teams consisted of an attending physician, one senior resident, two interns, and medical students. Teams were selected at random and were followed longitudinally on a weekly basis so that work routines on both call days (i.e., admitting new patients to the hospital) and non-call days (i.e. rounding without admitting) could be observed.

Data Collection

Observations were conducted by a multi-disciplinary team including clinicians (physicians, nurses) and non-clinicians (qualitative researchers, social scientists, and healthcare engineers). Typically, one clinician and one non-clinician were paired to observe each teaching team. Observers were oriented to the field of diagnosis and diagnosis errors through team meetings prior to observations. Team members were observed during pre-round work, rounds, and after rounds in activities such as order entry, documentation, communication, and processes such as admitting and discharging patients. Explanations for observed behaviors were elicited and documented. Observational data were collected using both open field notes as well as templated field notes modeled on components of the National Academy of Science model for diagnosis, which categorizes phases of the diagnostic process (e.g., data gathering and integration, formulation of a working diagnosis, and treatment delivery and outcomes), and components of the work system [2, 16, 17].

Focus Groups and Interviews

To understand participant rationales for diagnostic decisions and factors potentially contributing to errors, qualitative scientists (MH, MQ) and the study PI (VC) conducted focus groups and interviews with interns, resident and attending physicians after team observations. This allowed us to elicit thoughts and opinions on what we observed regarding systems-factors related to diagnosis. Attending physicians were interviewed separately from resident and interns to eliminate any power differentials during discussions. Within focus groups, residents and interns were asked about barriers to diagnosis, systems-based contributions to diagnosis, and challenges in making diagnoses. Attending physicians were also asked questions related to oversight, recognition and management of errors made by the team. All focus groups and interviews were audio recorded and transcribed verbatim for analysis.

Data Analysis

Field notes, focus group and interview transcripts were aggregated and analyzed using inductive content analysis by a qualitative specialists (MH) and verified by the study PI (VC) [18]. Inductive codes were derived from reading a sample of field notes and transcripts, which were then systematically applied to the remaining data. All data were coded using NVivo v10 [19], after which code reports were generated to check for coding consistency and identify themes related systems and cognitive contributions to error. Once these themes were identified, we returned to the literature and identified Graber’s taxonomy of diagnostic error [5] as conceptually useful way of organizing the data, an approach that has been utilized previously in qualitative literature [20, 21]. Consistent with this taxonomy, codes were organized thematically via internal discussion amongst investigators into those representing systems-based contributions to: (a) faulty data gathering (e.g., ineffective or incomplete history and physical, workup, procedures, or techniques), (b) faulty information processing (e.g., failed context generation, overestimating/underestimating usefulness of salient findings, failed heuristics), and (c) faulty information verification (e.g., premature closure). Because additional pertinent codes remained, a separate category, (d) systems-based contributions affecting multiple cognitive processes, was created.

Ethical and Regulatory Oversight

This study was reviewed and approved by the Institutional Review Boards at the University of Michigan Health System (HUM-00106657) and the VA Ann Arbor Healthcare System (1-2016-010040).

RESULTS

Between January 2016 and April 2017, a total of 10 teaching teams were observed during and after rounds for a total duration of 286 hours. A total of 31 residents and interns participated in focus groups and 10 attending physicians participated in interviews.

SYSTEM-BASED CONTRIBUTIONS TO FAULTY DATA GATHERING

A key cognitive problem was the inability to obtain accurate information in a timely fashion to make clinical decisions. Trainees and attending physicians most often highlighted system challenges related to interdisciplinary communication and the electronic medical record (EMR) as key drivers of this problem. For example, despite recognizing the importance of interdisciplinary care, team members reported limited nursing involvement during morning rounds. We observed that nursing had competing priorities during medical rounds, (such as giving hand-off to oncoming nurses or passing medications); yet, the knowledge of real-time information by nursing staff was felt paramount for diagnosis (e.g., overnight response to treatment, family input, concerns regarding behavior or sleep). Furthermore, the inability to communicate with nurses in person was exacerbated by barriers to information gathering. For example, one resident noted that paging character limits often led to incomplete messages. Others cited lack of nurse phones as straining communication, as calls were redirected to a unit clerk who then had to locate the nurse to relay the message.

…even something as simple as, “has the patient peed today?”. It takes 30 minutes to hear back an answer because I can’t actually talk to a nurse.

(Resident; Focus group)

Challenges with communication were also felt to arise from reliance on the EMR. For example, lack of updated vital signs or laboratory tests impaired diagnosis on rounds. Similarly, inability to provide context to orders via the EMR was felt to adversely affect the diagnostic process.

…a lot of times we put the orders in the system…and the assumption is, oh, they’ll look and see it in the system and so they’ll do it. And at times there’s context that needs to be taken into account, like when should I get this? How quickly should I do this? And we don’t always have time to be paging and make those kinds of communications as quickly as we may need to…

(Resident; Focus group)

SYSTEM-BASED CONTRIBUTIONS TO FAULTY INFORMATION PROCESSING

An important theme that linked system-based factor to impaired cognitive processes was problems in functionality within the EMR. In fact, we observed how even physicians highly trained in the EMR frequently sought help from others for some tasks (e.g., locating appropriate imaging studies, data from prior admissions, and certain orders). “Hard stops” in the EMR (i.e., screens that had to be addressed before further activities could be performed), were similarly felt to create interruptions in diagnosis and treatment. Furthermore, navigation of the EMR was felt to be taxing. For example, we observed that residents often spent upwards of thirty minutes combing through charts in search of historical data, imaging, lab results and documentation prior to admitting patients.

Organizational structures of other services were also viewed as limiting data synthesis and leading to delays or inaccurate diagnoses. A commonly cited example was the discrepancy observed between preliminary resident radiology reads and final reports. Final reports were entered several hours after the preliminary read, sometimes after management and treatment decisions were made. Knowing that this delay was possible, team members often deferred care decisions or had to rely on potentially incomplete or inaccurate information. Team members also cited organizational structures of consulting teams as confounding diagnosis, as consultants failed to recognize organ systems outside of their domain and rarely communicated with each another to reach mutual conclusions.

[Patient] had six different teams consulted and all the teams were saying this is a bone marrow problem except Hematology was saying this is not a bone marrow problem. So, it then becomes just everyone fighting against each other and the primary team is just stuck between a rock and a hard place.

(Intern; Focus group)

SYSTEM-BASED CONTRIBUTIONS TO FAULTY DATA VERIFICATION

Handoffs of care between day and night teams, including newly admitted overnight patients, frequently arose as a system-based theme fueling cognitive errors. Similarly, transitions at the end of the month for interns and residents, and every two weeks for attendings facilitated anchoring bias and premature closure (i.e., the failure to consider reasonable alternatives following the initial diagnosis). We observed these biases regularly, as teams accepted emergency room findings, consultant conclusions, or pieces of data without questioning assumptions.

…we are constantly signing out to one another and somebody else is getting the history and the emergency department is getting the history and they call them up and say, oh, this is what’s happening. So it’s easy like to hook onto what they said and just like kind of take it like at face value and not investigate it further.

(Resident; Focus group)

….in July, I picked up service. The patient’s hypercalcemic. The reasoning behind the hypercalcemia was high PTH…. It was a guy getting treated for pneumonia and treated for pneumonia on several occasions. The intern sort of said, “hey, his calcium level is up on the first day. It was like 11.5. That’s kind of weird….” The first couple of days they just gave him fluids, thinking, you know, he was going to get better but not understanding why. And then, probably about day three, we actually went back….and found he had a baseball in his lung. And I said, “That’s not pneumonia; that’s lung cancer.”

(Attending; Interview)

SYSTEM-BASED CONTRIBUTIONS AFFECTING MULTIPLE COGNITIVE PROCESSES

Several system-based factors were noted to affect multiple cognitive domains. These included interruptions, time constraints, and the physical environment.

Interruptions

One of the most cited system barriers to accurate diagnosis was frequent interruptions from pagers. On rounds, residents and interns were typically paged away at least once per morning, often having to play “catch up” upon returning. Teams described pages as “incessant” and “distracting” and stated that the high volume of pages led to cognitive dissociations between tasks. One resident recounted being paged 120 times within a 24-hour shift, dramatically affecting his ability to think critically about patients.

Imagine you are having these sorts of educated discussions about what you want to do for your patient, thinking about mechanisms and lab testing and diagnostics, and right in the middle of it, you have to keep closing out the chart of that patient to go order somebody else’s bowel regimen or to follow up on the family wanting to know about why the patient is on Vitamin D supplements.

(Intern; Focus group)

Time Constraints

Nearly all team members reported a lack of time to think critically about diagnosis. This was particularly notable on call days, when senior residents were trying to admit patients amidst responsibilities such as managing patients on the floor and discharging existing patients. On top of this, residents were expected to attend educational conferences, leading to being rushed and fueling errors in decisions.

I would consolidate my thoughts and my thinking at the end of the day when I finish writing my note and I’ve actually written everything down and then like I said, 20 percent, I find 20 percent new problems. And now it’s 7 or 8 PM. It’s too late to deal with these. If I had actually taken—if I had actually pushed that process up a few hours… and found maybe these few problems, I could have a consult that day instead of the next day. I could have gotten that test done today as opposed to tomorrow morning and something could have happened overnight.

(Intern; Focus group)

Physical Space

Many team members described the physical space as “suboptimal” for diagnosis. For example, team rooms were noted to be cluttered and disorganized. A lack of dedicated space for every team member was commonly observed, leading to an insufficient number of computers. Team rooms were noisy and shared amongst multiple medicine teams and nursing staff. In fact, some resorted to donning headphones to drown out ambient noise and allow them to think clearly.

…we share our team room with another [team] and they’re doing their kind of table rounds and we’re trying to do our table rounds and it can get loud and cacophonous and I lose my train of thought…

(Intern; Focus group)

DISCUSSION

In this focused ethnography, we identified and categorized how systems-based factors contribute to cognitive error. First, trainees and attending physicians noted that challenges with interdisciplinary communication, including in-person communication and communication within the EMR contributed to faulty data gathering. Second, inefficiencies within the EMR (such as “hard stops”), as well as operations within the hospital (such as timing and siloed role of consulting services), contributed to faulty data processing. Third, frequent handoffs between outgoing and incoming physicians provided opportunities for premature closure and anchoring, resulting in faulty data verification. Finally, constant interruptions, time constraints, and a cluttered physical environment impacted multiple cognitive processes.

Interventions aimed at curbing cognitive contributions to diagnostic error include those aimed at increasing knowledge and improving clinical reasoning [12]. Such efforts have focused on Kahneman and Tversky’s System 1 (intuitive) and System 2 (effortful) domains of reasoning [22]. System 1-based interventions include such aspects as cognitive de-biasing, metacognition, and use of checklists to encourage stopping and thinking. In contrast, system 2-based interventions emphasize principles of clinical reasoning, evidence-based medicine and knowledge regarding specific disease conditions [12]. Importantly, studies testing these principles to prevent errors are often devised and tested in non-clinical settings, where factors such as interruptions, time constraints, physical environment, and the quality and accessibility of necessary information are tightly controlled [23]. Similarly, they assume a puritan view when it comes to cognitive processes and system-based factors. In accordance with Henriksen et al.’s human factors perspective, our findings suggest that separating the two – as has been done in many studies to date – may not be wise [14]. Rather, targeting both system and cognitive factors concurrently may provide greater opportunities to implement sustainable changes.

Our ‘real-world’ study, performed on the wards and outside the confines of a simulated environment, suggests that interventions targeting systems-factors and cognitive factors simultaneously should be prioritized. One example of such an approach are cancer tumor boards, implementation of which has resulted in marked change in diagnoses, evaluation, and treatment strategies in a variety of malignancies [24–26]. Despite necessitating upfront time from physicians across specialties, these boards are successful in part due to the time-savings from care coordination and improvements in interdisciplinary communication, allowing for improved data gathering and processing. In similar fashion, “diagnosis boards” -- aimed at improving diagnosis by conglomerating multiple specialties -- may yield benefit by minimizing information silos and maximizing communication – a classic system and cognitive trap observed in this study.

Targeted interventions aimed at curbing the number of interruptions may also be important in preventing diagnostic errors [27]. For instance, separating admitting from rounding teams may minimize interruptions and improve rounding efficiency by uncoupling pages on existing patients from new admits, allowing more time to synthesize and interpret data. Similarly, rethinking timing and delivery of didactic medical education, (e.g., consolidation during one afternoon rather than every day), may help alleviate time constraints that prevent critical thinking. Finally, “intersectional innovations” [28] that include rethinking the design of physical space for diagnoses (e.g., ergonomically designed workrooms with more table and computer space) and collaboration with information technology professionals to improve EMR functionality would be welcomed. These examples are but a few of how system and cognitive processes might be linked to help improve diagnostic decisions.

Our study has limitations. First, the study was performed at two academic centers, which may limit the generalizability of our findings. Second, our study did not specifically identify cases of diagnostic error; rather, we report on observations and views of trainees and attending physicians on contributors to these events. Third, teams may have modified aspects of their behavior in the setting of being observed, leading to a potential Hawthorne effect. This effect may be limited, however, because people do not act in ways that deviate from habits for long [29]. Fourth, although we specifically examined for overlap between systems and cognitive factors, there may be some aspects that we could not observe. For example, the association between inadequate knowledge and systems-based factors and reasons underlying these relationships could not be explored using ethnographic techniques. Additionally, we acknowledge that lack of key data in the EMR is a system-based problem that may limit cognitive processing; however, it was not possible for us to evaluate this through our observational approach. These limitations were necessary given the broad scope of this work. Finally, while we aggregated data from observations and followed a methodologically rigorous approach to link system and cognitive processes, our findings should be viewed as hypothesis generating. Future studies are needed to assess the veracity of our conclusions.

Our study also has important strengths. First, data were collected over 280 hours, spanning 14 months, with physicians of varying backgrounds and experiences – lending a high degree of internal validity. Second, data collected in real-time through observations were triangulated with focus groups and interviews, leading to a better understanding of workflow and the diagnostic process. Third, our team was highly multidisciplinary and included clinicians and experts trained in ethnographic methods. Finally, we utilized a published framework to organize our findings, allowing for identification of interventions using these data. Aligning systems with cognitive-based processes is novel and offers new insights into tackling diagnostic errors.

Efforts to address diagnostic error have yielded limited results to date. Recognizing systems- and cognitive factors as overlapping processes and designing interventions aimed at concurrently addressing both domains may provide physicians with the time and tools necessary to improve diagnosis.