Exploring Situational Awareness in Diagnostic Errors in Primary Care

Abstract

Objective

Diagnostic errors in primary care are harmful but poorly studied. To facilitate understanding of diagnostic errors in real-world primary care settings using electronic health records (EHRs), this study explored the use of the Situational Awareness (SA) framework from aviation human factors research.

Methods

A mixed-methods study was conducted involving reviews of EHR data followed by semi-structured interviews of selected providers from two institutions in the US. The study population included 380 consecutive patients with colorectal and lung cancers diagnosed between February 2008 and January 2009. Using a pre-tested data collection instrument, trained physicians identified diagnostic errors, defined as lack of timely action on one or more established indications for diagnostic work-up for lung and colorectal cancers. Twenty-six providers involved in cases with and without errors were interviewed. Interviews probed for providers' lack of SA and how this may have influenced the diagnostic process.

Results

Of 254 cases meeting inclusion criteria, errors were found in 30 (32.6%) of 92 lung cancer cases and 56 (33.5%) of 167 colorectal cancer cases. Analysis of interviews related to error cases revealed evidence of lack of one of four levels of SA applicable to primary care practice: information perception, information comprehension, forecasting future events, and choosing appropriate action based on the first three levels. In cases without error, the application of the SA framework provided insight into processes involved in attention management.

Conclusions

A framework of SA can help analyze and understand diagnostic errors in primary care settings that use EHRs.

Introduction

Diagnostic errors are among the most common, expensive, and harmful types of errors in outpatient settings,[1-6] and were recently recognized by WHO as a priority area.[7] Despite their global significance,[8-14] these errors are challenging to study and have received inadequate attention.[15-18] Additionally, the science of understanding and analyzing outpatient diagnostic errors is poorly developed.[19-21] Although patient and system factors are known to contribute to diagnostic error,[22] little is known about how diagnostic decision-making errors occur in routine primary care settings.[23] Preventive strategies are thus relatively underdeveloped.[24]

The diagnostic process (i.e. detecting signs/symptoms and coordinating diagnostic procedures) is dependent on several external factors (work-system factors) and extends beyond the cognition of a single provider.[22] Thus, cognition in the primary care environment has become increasingly “distributed” and dependent on evolving situations and work-system factors such as teamwork, technology and organizational factors.[25-29] However, the role of these work-system factors and how they affect cognition has not been adequately addressed. For instance, there is little understanding of how attention is managed and diagnostic plans are created and executed in the context of longitudinal outpatient care that spans weeks or months.

We previously described the application of the Situational Awareness (SA) conceptual framework from aviation human factors research to advance the science of diagnostic decision-making in primary care.[30] In aviation, SA is defined as a person's “perception of the elements in the environment within a volume of space and time, the comprehension of their meaning, and the projection of their status in the near future”.[31] Loss of or failure to gain SA is commonly implicated in aviation accidents related to pilot errors and may be precipitated by system-related and cognitive factors.[32-35] Maximizing SA has been an important focus of aviation safety research.[36] An application of four levels of SA to the primary care diagnostic process (Figure 1) [30] illustrates the potential of this framework to improve understanding of diagnostic errors. The four levels include information perception, information comprehension, forecasting future events, and choosing appropriate action (resolution) based on the first three levels. SA has also been defined as a “non-technical skill” in anesthesia.[37,38] In this context, SA components include (1) detection and interpretation of situational clues from multiple, dynamically changing sources (including from patients, other members of the care teams, and from displays and equipment); (2) thinking about what might happen in the future and adapting to evolving situations; and (3) keeping track of and utilizing special elements of knowledge.[37] Thus, the unit of analysis is not only the provider but the interaction of the provider within the environment.[38]

Figure 1. An adaptation of Endsley's model of situational awareness to medical diagnosis.

Missed and delayed cancer diagnosis is a leading type of diagnostic error.[2,4,39] In previous work, we found that substantial delays in cancer diagnosis often resulted from failure to take appropriate action on obvious alarm features in clinical presentation [including diagnostic information documented in the electronic health record (EHR)],[40-42] a scenario that strongly suggests lack of SA.[40,41] In the present study, we defined diagnostic errors as missed opportunities to make a timely diagnosis and explored the relationship between the lack of SA and diagnostic errors in two common cancers (colorectal and lung). Our aim was to evaluate whether the SA framework can be used to analyze outpatient diagnostic errors using a “distributed cognition”[25-27] approach that included provider-work system interaction, including interaction with the EHR.

Methods

We used a mixed-method approach to explore application of the SA framework to the understanding of diagnostic errors. We first identified cases of diagnostic errors through record reviews using previously developed data collection methods.[40,41] Because decision-making processes are not always documented clearly, record reviews were followed by qualitative semi-structured interviews to better understand providers' thoughts and actions in selected cases.

Setting

Sites included two large integrated health-systems in the United States, each with almost 2 million outpatient visits annually. Both sites used EHRs. Primary care was delivered by about 175 staff (both academic and non-academic) primary care providers (PCPs) and residents. Local institutional review boards approved the study.

Inclusion/Exclusion Criteria

Using tumor registry data, we identified new cases of primary colorectal cancer (CRC) and lung cancer diagnosed at the study sites between February 2008 and January 2009 (N= 380). We excluded cancer recurrences, cases diagnosed at other institutions, and cases diagnosed in the inpatient setting with no previous primary care contact (e.g., when the patient's first presentation to the emergency room led to hospitalization).

Data Collection

Chart reviews to identify errors

In absence of formal definitions for missed and delayed cancer diagnosis, we defined diagnostic error as lack of timely action on one or more of established indications for diagnostic work-up in CRC[43] and lung cancer (Table 1).[40,41] For example, failure to initiate colonoscopy despite documented presence of iron deficiency anemia was considered an error. “Timeliness” was defined based on expert opinion and selected literature.[41,44,45] Justified decisions to not pursue further diagnostic work-up after recognition of alarm symptoms (e.g., for patients with reduced life expectancy) were not considered errors.

Table 1. Established Indications for Cancer Related Diagnostic Work-Up and Defined Time Periods.

Colorectal Cancer	Time to initiate diagnostic evaluation	Lung Cancer	Time to initiate diagnostic evaluation*
Confirmed iron deficiency anemia	7 days	Blood in sputum/hemoptysis	7 days
Screen positive for fecal occult blood test	7 days	Hoarseness that lasts more than 2 weeks	7 days
Screen positive for barium enema	7 days	Recurrent bronchitis or pneumonia	7 days
Abnormal imaging suspicious for colorectal cancer	7 days	Abnormal chest x-ray	7 days
Hematochezia in the absence of negative colonoscopy within previous 1 year	7 days	Abnormal chest CT scan	7 days
Suspected rectal or abdominal mass on physical exam	7 days	Abnormal abdomen CT scan	7 days
Change in bowel habit	60 days	Serial abnormal imaging	7 days
Due for endoscopic surveillance based on previous test	90 days	Abnormal sputum examination/sputum cytology	7 days
Obstruction	7 days	Unexplained effusion	7 days
Microcytic anemia	30 days	Clubbing	7 days
		New onset Cushing's symptoms/syndrome	7 days
		New onset of hypercalcemia symptoms/syndrome	7 days
		New onset of syndrome of inappropriate antidiuretic hormone secretion	7 days
		Superior vena cava obstruction	7 days
		Worsening persistent cough/bronchitis OR new description of chronic cough lasting > 8 weeks	7 days
		Provider acknowledged unexplained weight loss or other unexplained weight loss >10lbs in addition to respiratory symptoms	7 days

^*We also tested a more lenient criteria of 14 days for appropriate diagnostic evaluation (rather than 7 days), but it did not substantially reduce the number of missed opportunities. [41]

A trained physician used a pre-tested data collection instrument to identify diagnostic errors through detailed review of the EHR.[40,41] We did not collect information on patient harm to reduce hindsight bias.[46] After chart review, two physician experts discussed all identified cases of diagnostic error to confirm the presence or absence of error. Cases unanimously judged to contain errors were considered for interviews. We also interviewed selected PCPs in the non-error group to assess their SA.

Interviews

A social scientist conducted semi-structured, open-ended interviews of PCPs between June 2008 and June 2009. To facilitate recall, we excluded PCPs involved in errors >1 year prior to diagnosis. We selected cases in which PCPs were closely involved and excluded those in which PCPs were residents. We invited PCPs to participate in an interview about the diagnostic process concerning specific patients of theirs who had recently been diagnosed with cancer. The identity of the patient was not specified at the invitation, and we refrained from discussing the purposes of the study in terms of “errors.” To increase likelihood of finding new meaningful information across individuals and diagnostic scenarios, we used maximum variation sampling[47] to select cases with different types of established indications. Selection decisions were made by team consensus. Potential participants were recruited through an email invitation, followed by a reminder email and phone call. PCPs were offered a $50 gift certificate as an incentive to participate.

The interview guide focused on discussing details of the case (see Table 2). PCPs had the EHR available to them to prompt their memories during interviews. The interviewer asked providers to give an account of patient care from initial presentation to the time of diagnosis. Providers were prompted to give details about their diagnostic decision-making, such as their perception of clinical clues and reasons for particular actions. Interviews were conducted in the PCPs' offices and lasted between 15 and 60 minutes. Content was audio-recorded and later transcribed and de-identified.

Table 2. Sample Interview Protocol Questions.

1. Patient History/Presentation

I'd like to specifically focus on the patient presentation. S/he was diagnosed with cancer on date XX/XX/XX. Take me through the process that led to his diagnosis.

Probe Examples

Can you talk about some of the reasons the patient came in?

Was there anything specific that caught your attention?

Talk about your plan of action.

2. Specific Events

On this date XX/XX/XX, ______ happened. Can you take me through that event?

Probe Examples

What is the general procedure?

What do you usually do?

Talk about the factors led to (or could lead to) the delay?

After going through the case, is there anything you think could have prevented the delay (example from case)?

3. General Questions

In general, can you talk about factors that you think might affect clinical decision making? (Example: factors that help/facilitate or impede/delay the process.)

Analysis

We analyzed data using a framework-analysis approach,[48] a qualitative research method that allows for the inclusion of existing concepts as well as emergent themes.[49] Based on previous work,[30,40,41] we anticipated that chart review and interview data would suggest failure to act on established indications for work-up in diagnostic error cases.

Framework analysis consists of 5 stages: familiarization, thematic analysis (thematic framework), indexing (coding), charting, and mapping and interpretation. First, the qualitative team (four study-investigators) familiarized themselves with all data by reading, rereading, and summarizing each case. Multiple readings allowed investigators to gain an overall impression of the interviews and chart reviews and to begin creating a thematic framework incorporating both emergent and a priori themes. From this framework the qualitative team created a coding scheme to summarize recurrent observations and themes in the data. Two investigators independently applied the coding scheme to all interview transcripts using the software package Atlas.ti 5.0.[50] Coders subsequently met to reach consensus about discrepant coding decisions.

The research team reviewed the EHR and interview transcript for each case and collaboratively determined whether there was evidence for lack of SA. If so, the team defined the lowest level at which this occurred (higher levels depend on the success of the lower levels, consistent with the SA literature).[51] For example, lack of SA at Level 1 occurred when there was failure of perception of some information by the provider (see Appendix). Finally, the coded data and levels of SA were analyzed across cases and by themes to explore emergent patterns. All conclusions were drawn from multiple team meetings and group consensus.

Results

Sample Size

Across both sites, 254 cancer cases met inclusion criteria. On chart review, 30 of 92 lung cancer cases (32.6%) and 56 of 167 CRC cases (33.5%) met criteria for error. Table 3 shows the distribution of types of cancers and respective indications for work-up. Of 55 PCPs invited to interview, 26 agreed to participate, 10 declined (most often due to lack of time), 5 left the institution before contact, and 14 did not respond to multiple interview requests. Providers had been in practice for a median of 12 (range 2-44) years. The final sample consisted of 31 cases (17 cases containing 18 errors and 14 with no errors) discussed with 26 PCPs (3 PCPs discussed 2 cases, and 1 PCP discussed 3 cases). There was no statistical difference in median years in practice between providers in error versus no-error groups.

Table 3. Types of Established Indications in Cases of Diagnostic Error^*.

Established indication	Total
Colorectal Cancer (n=56)
Fecal Occult Blood Test positive	33
Bright red blood per rectum	21
Microcytic anemia	15
Iron deficiency anemia	10
Change in bowel habit	8
Abnormal Barium Enema (BE)	2
Abnormal imaging other than BE	2
Lung Cancer (n=30)
Abnormal chest x-ray	18
Abnormal chest CT scan	8
Chronic cough	3
Hemoptysis	2
Weight loss >10lbs	2

^*Some cases have more than one missed clue

SA and Error

In cases of diagnostic error, the model was useful to understand how SA was lacking in certain aspects of the dynamic provider-work system interaction.

Perception: Level 1 SA (n = 5)

All cases associated with delayed response or lack of response to predefined established indications were categorized as lack of Level 1 SA. The EHR notification system, the primary means by which providers received abnormal test results, featured prominently in these cases. For instance, a provider who did not order colonoscopy until 6 months after an abnormal hemoccult discussed how he might have missed the test result:

“It may have been an error, I got it [test result alert] but I just wasn't able to remember to put the colonoscopy in…But it would have not missed my eye when he [the patient] came the next time that's why I gave it [colonoscopy referral]. There is no defense other than between then and now he didn't see me…It could have come on a day when I was off and it was somebody doing an alert for me. I really cannot tell.” (Provider 2)

Certain providers did not perceive alarm features recorded in the EHR. For instance, one provider acknowledged the importance of a patient's documented weight loss over several months but failed to see this clue until the patient brought it to his attention:

“They [x-rays] were all normal. That was up to March, until March and then July he came in and said ‘ok I am losing weight.’ Ok so CAT Scan on you because…it's a little bit worrisome now.” (Provider 13)

Comprehension: Level 2 SA (n = 4)

Cases judged as lacking Level 2 SA were those in which comprehension of the current situation was affected; they led to delayed action, inaction, or inappropriate action. In one such case a provider recognized rectal bleeding in an older patient but attributed this to hemorrhoids and referred the patient to a proctologist without performing additional work-up. In another case with rectal bleeding, another provider did not take any follow-up action because he attributed this to prostate cancer radiation:

“Bright-red blood per rectum on occasion but as far as I knew it was due to his radiation. So everything that he had, kind of leaned toward prostate cancer and things you get from prostate cancer and radiation and metastatic to bone and all … in most patients I take care of who have metastatic prostate cancer all these symptoms are related to that. And not a new cancer…” (Provider 6)

Forecasting: Level 3 SA (n = 5)

When providers' thinking about what could happen in the near future was affected, we categorized these cases as lacking Level 3 SA. Forecasting was thus applied not only in terms of disease progression and outcomes, but also in terms of how the work-system [28,29] in which the provider was practicing in might operate. Most of the providers involved took inappropriate actions, rather than delayed or no actions. For instance, one provider, knowing he would be out-of-office for few months, failed to hand off a patient being followed for two previous abnormal x-rays. Instead, he assumed that the patient would return to complete the third x-ray prior to his upcoming leave:

“And so what happened was, I went out and there was a discontinuation in care. …when you're talking to patients and you give them follow up appointments you do expect them to keep them although sometimes they don't for whatever reason…but in hindsight, I guess with two abnormal, we probably should've just went on and got the CAT scan so that was an error in my judgment…” (Provider 3)

Choosing appropriate action: Level 4 SA (n = 4)

Delayed or no response tended to occur when providers lacked a sense of urgency related to the clue. For instance, despite knowledge of microcytic anemia, a provider did not pursue actions such as ordering iron studies. In other cases, two providers with knowledge of diagnostic clues waited until their patients' return visits to order the subsequent diagnostic work-up.

“Well usually I'll want to see the patient back so I can explain to them what the problem is and that may sometimes be a delay. … my preference is to talk to them [in person] because sometimes it may be that they didn't do the diet or you know something like, they're having rectal bleeding and I need to get them in quicker. So I prefer to see them…” (Provider 26)

Lack of Level 4 SA also led to inappropriate responses. For example, one provider ordered a CT scan of the abdomen and tumor markers to work up rectal bleeding.

SA in Cases without Errors (n = 14)

We also evaluated application of the SA model in cases without errors. These cases involved patients who were symptomatic as well as those where only the cancer screening test was abnormal. In all asymptomatic cases (n=5; all CRC), providers immediately ordered the next appropriate step and notified patients as soon as the screen was found abnormal. For symptomatic patients (n = 9), we noticed that providers not only perceived and comprehended symptoms as suspicious and took immediate action, but also took additional steps to ensure that patients maintained future follow-up (Table 4), a potential strategy to combat diagnostic error.[52] Thus, the model was applicable and showed the dynamic nature of the interaction of the providers and their environments (including patients) and provided insight into processes involved in attention management.

Table 4. Additional steps taken by primary care providers to ensure that patients maintained future follow-up.

Cancer type	Example and quote
Took additional steps to ensure that patients maintained future follow-up.
Colorectal	“Ok in March 2008 let me see…I saw a drop in his hemoglobin hematocrit. I did a guaiac, which was positive… And then, at that time, I ordered him a colonoscopy. I called the colonoscopy unit right away and they did the colonoscopy the same [month]…” (Provider 8)
Made additional efforts to express their concerns to patients, emphasizing the need for further testing in an attempt to ensure follow-up
Lung	“They didn't want to do anything about it [abnormal chest x-ray]. So I put in a consult for him to Pulmonary again. I spend a lot of time with him that day kind of breaking down their myths.” (Provider 9)
Persistently contacted a patient's family member to express concern for follow-up
Lung	“And luckily I did have the daughter's contact number. And left her several messages until I got a hold of her and kind of impressed upon her that his chest x-ray looked really abnormal, he's been a smoker, and he has trouble breathing. I'm concerned there's some kind of tumor going on.”(Provider 1)

Discussion

We found that the SA framework can be used to analyze outpatient diagnostic errors in the context of a distributed cognition approach that includes provider-work system interaction, including interaction with the EHR. The SA framework has already proven useful in other complex distributed environments such as aviation and, more recently, in anesthesia. Because the diagnostic process in the longitudinal, distributed environment of primary care is poorly understood, application of SA framework might help overcome some of the challenges faced in the application of current diagnostic decision-making models.[53-56]

Previous work on diagnostic error has emphasized concepts related to medical problem-solving and normative decision-making.[55,57-62] Much of this literature has focused on unusual scenarios (uncommon diseases or atypical presentations of common diseases)[63] and clinical problem-solving under experimental conditions.[53,54,64] Models of deliberate decision-making processes, though useful, do not fully account for the reality that providers frequently miss more typical diagnostic “alarm features” in real-world practice environments. Human factors-based frameworks for addressing safety in complex systems accommodate both higher-level decision-making processes and “simple” risks such as forgetting or overlooking important information. One finding of this exploratory study was that providers implicitly depend on “prompts” within the system (of which EHR alerts are but one example) to remember when to perform required actions. This is an example of distributed cognition (information processing distributed within the overall social-technical system rather than being held solely within the mind of the individual provider). When these expected prompts fail to occur for various reasons, the provider may fail to act. Thus, whereas many errors manifest in an apparently straightforward manner (e.g., “dropping the ball” on a missed test result), the circumstances that influence their frequency or consequences may be complex, and potentially modifiable. Application of the SA framework may help in better understanding the nature and circumstances around which errors occur in naturalistic socio-technical environments.[29,65]

Retrieval of diagnostic information from several distributed sources[66] will become increasingly challenging as healthcare becomes more integrated through EHRs or health information exchanges.[67,68] Thus, measuring and maximizing various components of providers' SA should be an essential consideration to reduce diagnostic errors. Because SA has been studied within teams in aviation, it can also offer useful insights for primary care team interaction.[69] The framework also encompasses cognitive activities suggested by experimental work involving diagnosis, including “generating/triggering a diagnostic hypothesis” or “formulating a working-diagnosis”.[53,57] However, by defining these activities as a series of cognitive processes involving errors in real-world practice, the framework lays a foundation for understanding contributory factors that might affect them.

The SA framework has features similar to previous taxonomies of data gathering and interpretation[22] and other diagnostic care processes[39]. Use of the SA framework might also help understand how both intuitive and more deliberate processing are involved in the context of the provider-work system interaction, tying well to the dual-process model of diagnostic reasoning.[55] For instance, later stages of the SA model reflect the more deliberative processing that must take place for successful diagnosis. Because of the emphasis on distributed cognition and provider-work system interaction, analysis using the SA approach could be complementary to other taxonomies. For instance, it could provide insight by distinguishing among levels of SA failures, an important concept because there may be different strategies to address risks (i.e., through training or system redesign) depending on the nature of the failure.

We see two potential advantages for applying the SA framework alone or in combination with other analytical approaches. First, EHRs will house much of the diagnostic data providers will use to make diagnosis in the future and more automation is inevitable, hence this type of human factors perspective is essential to make further advances in the area. Second, SA has been measured in simulated settings in other fields and we envision that innovative mechanisms to measure levels of SA in simulated and live health care settings (e.g., using standardized patients)[70] can help advance the understanding of breakdowns in the diagnostic process in real-world settings. The current analytic frameworks do not allow that.

For optimal performance, certain aspects of the socio-technical work system need to be anticipated. In this context, we found forecasting to be of particular interest in diagnostic error scenarios. For instance, not being able to predict system vulnerabilities in health care could increase the risk that patients will “fall through the cracks.” Anecdotally, we noted that that in non-error cases providers mentioned certain proactive strategies to minimize risk of loss of follow-up (e.g., personally scheduling colonoscopies by calling gastroenterology, reinforcing need for follow-up to patients and/or family members).[71] Cognitive and behavioral scientists should further explore these concepts.[72] In aviation, high levels of SA might also be associated with “dynamic skepticism,” an attitude of questioning the validity of previous assumptions by constantly evaluating incoming data.[73] This concept also merits further exploration. For instance, PCPs might become less sensitive to subtle clues because much of what they deal with is management of many routine or chronic conditions. Dynamic skepticism could be taught as a countermeasure providers might take to help anticipate and prevent problems.

Ongoing research in aviation and other industries about improving SA might be useful to design interventions to reduce diagnostic error.[33,74] For instance, system design could have a powerful influence on how information is presented (for instance, automated graphical display of weight if the EHR detects weight-loss) to maximize perception. Level I SA failures, for instance, could be averted through improved alerting mechanisms, display designs, and task management strategiecs. Similarly, future research can identify cognitive constructs, mental models, and other EHR-related determinants (such as decision support systems) that facilitate comprehension.[75] Novel training strategies using simulation could be implemented to maximize forecasting, which we found to be of particular concern.[70,76,77] Measurement of SA could be incorporated in the design and evaluation of information systems so that provider SA can be maximized to reduce diagnostic error.

Our results have several limitations. First, our results do not show any causal relationship between SA and diagnostic error or diagnostic performance. Failure of SA does not imply a provider error and it is possible that factors unrelated to SA lead to problems in error cases and to successful outcomes in non-error cases. Underlying work-system problems might also be responsible for both loss of SA and errors. At present, we can only posit that loss of SA might be a risk factor of certain diagnostic errors. Second, as SA models have mostly been applied to more volatile situations, such as in aviation and anesthesia, its applicability to primary care needs to be further validated. Third, we did not use any experimental techniques to measure SA but used consensus to determine lack of SA. Fourth, despite a large sample for chart reviews, the interview sample was small. Though adequate for concept exploration, our sample might not have represented PCPs and diagnostic scenarios optimally and may not be generalizable outside our study settings. Lastly, we could not collect reasons why providers did not choose to participate despite multiple invitations. Despite these limitations, because diagnostic errors occur universally, these concepts and methodologies might be useful to others.[78]

In summary, we found that a human factors-based framework can help analyze and understand outpatient diagnostic errors using a distributed cognition approach that includes provider-work system interaction, including providers' interaction with the EHR. Future research should explore strategies to measure and improve SA through multidisciplinary interventions.