A structured approach to EHR surveillance of diagnostic error in acute care: an exploratory analysis of two institutionally-defined case cohorts

Abstract

Objectives

To test a structured electronic health record (EHR) case review process to identify diagnostic errors (DE) and diagnostic process failures (DPFs) in acute care.

Methods

We adapted validated tools (Safer Dx, Diagnostic Error Evaluation Research [DEER] Taxonomy) to assess the diagnostic process during the hospital encounter and categorized 13 postulated e-triggers. We created two test cohorts of all preventable cases (n=28) and an equal number of randomly sampled non-preventable cases (n=28) from 365 adult general medicine patients who expired and underwent our institution’s mortality case review process. After excluding patients with a length of stay of more than one month, each case was reviewed by two blinded clinicians trained in our process and by an expert panel. Inter-rater reliability was assessed. We compared the frequency of DE contributing to death in both cohorts, as well as mean DPFs and e-triggers for DE positive and negative cases within each cohort.

Results

Twenty-seven (96.4%) preventable and 24 (85.7%) non-preventable cases underwent our review process. Inter-rater reliability was moderate between individual reviewers (Cohen’s kappa 0.41) and substantial with the expert panel (Cohen’s kappa 0.74). The frequency of DE contributing to death was significantly higher for the preventable compared to the non-preventable cohort (56% vs. 17%, OR 6.25 [1.68, 23.27], p<0.01). Mean DPFs and e-triggers were significantly and non-significantly higher for DE positive compared to DE negative cases in each cohort, respectively.

Conclusions

We observed substantial agreement among final consensus and expert panel reviews using our structured EHR case review process. DEs contributing to death associated with DPFs were identified in institutionally designated preventable and non-preventable cases. While e-triggers may be useful for discriminating DE positive from DE negative cases, larger studies are required for validation. Our approach has potential to augment institutional mortality case review processes with respect to DE surveillance.

Introduction

Diagnostic errors (DE), defined as missed opportunities to make a correct or timely diagnosis based on available evidence, are frequently implicated in cases of preventable harm due to medical error [1]. Unfortunately, these errors are difficult to detect and characterize, and consequently, remain underrecognized in patient safety research [2], [3], [4], [5]. Most research on DEs has been conducted in ambulatory settings, including pediatric facilities, primary care clinics, and emergency departments, and these data suggest that at least 1 in 20 adults in the United States are affected annually [6], [7], [8], [9], [10], [11], [12], [13]. When DE occurs in hospitalized patients, it has potential to result in considerable harm.

A recent meta-analysis estimates that at least 0.7% of adult hospitalizations involve a harmful DE [14]. Autopsy studies suggest variable rates (4.1–49.8%), with rates of preventable deaths (class I errors) ranging from 0 to 20.7% [2]. Emerging data using the validated Safer Dx instrument suggests that the rate of harmful DEs is between 5 and 7% in medical patients hospitalized in the intensive care unit or who are readmitted [4, 15]. The variability in reported rates is in part due to the methods used to assess the likelihood of DE (autopsy vs. chart review), as well as the variable pre-test probability of detecting DE in specific cohorts – patients who expire or undergo autopsy are much more likely to have experienced a consequential error [16]. A variety of cognitive, systems, and clinical factors may also be implicated in this variability, though these factors may not necessarily lead to harm in patients who experience a DE [4, 17], [18], [19], [20]. Such factors may not be routinely detected by traditional institutional mortality case review processes which are more focused on identifying medical errors in general (not specifically DE) and determining whether harm caused by these errors was preventable.

In contrast to other hospital quality and safety improvement efforts, efforts to systematically identify and assess the impact of DE in the inpatient setting have been limited, in part due to the complexity of hospital care [16]. First, few institutions have a standard process for assessing the likelihood of DE; determinations based on retrospective case reviews via institutional safety reporting systems and morbidity and mortality forums can be subjective, and medical errors that contribute to harm may more often be reported compared to those that do not. Second, while traditional case review processes that incorporate validated instruments to determine the likelihood of DE (Safer Dx) or pinpoint failures in the diagnostic process (Diagnostic Error Evaluation Research [DEER] Taxonomy) might be useful for augmenting traditional mortality case review processes with regard to identifying DEs, this has not yet been validated [19, 21, 22]. As we recently reported, a process driven approach for pinpointing failures in the diagnostic process may lead to increased detection of both consequential and non-consequential DE, as well as targets for intervention [15]. Third, while certain events recorded in the EHR, such as unplanned encounters or readmissions, have been used to identify enriched cohorts of potential DE cases in ambulatory patients, leveraging electronic triggers (e-triggers) to identify potential cases of DE in hospitalized patients is less well understood [4, 8, 23], [24], [25]. While preliminary work has suggested that specific events routinely captured by the EHR (e.g., multiple consultants, codes, unexpected surgery, new or worsening oxygen requirement, etc.) may be associated with DEs, such correlations have not yet been validated [23, 26].

The objectives of this study were to apply a structured EHR case review process for assessing the likelihood of DEs, characterizing their clinical impact, and quantifying the number and types of diagnostic process failures in two cohorts of hospitalized patients who expired on the general medicine service. All cases underwent our institution’s gold standard mortality case review process to determine whether the outcome (death) was preventable or non-preventable. Our secondary objective was to explore whether certain categories of e-triggers were associated with DE for further validation prior to use for electronic case surveillance.

Materials and methods

Setting, subjects, study design

This study was approved by the Mass General Brigham (MGB) Institutional Review Board. We used our institutional Enterprise Data Warehouse (EDW) to identify adult patients (>18 years) who expired between 1/2016 and 12/2018 while hospitalized on the general medicine service at a major academic medical center in Boston, MA. All cases previously underwent a formal mortality case review, described by Mendu et al. [27], which utilized a four step process (SEEK) that: (1) Surveys clinicians using an electronic mandated mortality review tool capturing complications, communication issues, timeliness of interventions, end-of-life information, presence of medical errors, presence of systems issues, and suggestions for quality improvement; (2) Evaluates completed surveys and obtains additional information from respondents (performed by a trained clinician); (3) Escalates the case to a multidisciplinary quality committee; and (4) Keeps track of cases in a central database. Under the initial review process, each death received a preventability rating (1–5); those likely to have had medical error (a rating of 3 or more) were designated preventable and investigated further under the SEEK schema. Approximately 5–6% of cases that have undergone this mortality case review process are typically deemed preventable, a rate that has remained stable over time [27].

Case cohorts

As part of an exploratory study to validate our structured EHR case review process and assess its potential utility for DE surveillance, we constructed two cohorts using all patients who expired on the general medicine service and underwent our institutional mortality case reivew process (n=365), our current “gold standard” for identifying medical errors contributing to death. The first cohort included all institutionally designated preventable cases (n=28). The second cohort included an equal number of randomly selected non-preventable cases (n=28). Since many of the factors contributing to preventable cases of death (e.g., miscommunication, care delivery delays) also underlie DEs [28], we hypothesized that cases of hospitalized patients who expired with medical error (considered preventable by the institutional mortality case review process, rating of 3 or more) would have different rates of DEs compared to those considered not preventable (rating of 2 or less) [27]. We excluded cases with length of stays (LOS) greater than 1 month from both groups to minimize the complexity and volume of records for adjudicators to review.

Structured EHR review process for determining diagnostic error in acute care

The structured EHR case review process for assessing DEs during acute care (Table 1) was designed and managed in REDCap [29]. Our chart abstraction instrument (Supplementary Figure S1) was based on validated instruments (Safer Dx, DEER Taxonomy) to assess the likelihood of DE and presence of diagnostic process failures (DPFs) [3, 30, 31]. We adapted individual Safer Dx items to reflect the diagnostic process during the acute episode of care under review per guidelines reported by Singh and similar to the experience of Bergl [15, 21]. Additionally, we included sections to determine the presence of associated therapeutic error (TE) and to characterize the preventability, ameliorability, and severity of DEs based on a widely used scale [32, 33]. Lastly, reviewers were instructed to identify failures in the diagnostic process using the modified DEER Taxonomy adapted for acute care as we previously reported [30]. Branching logic was used to guide the reviewer at each step of the process: case information review, diagnostic timeline evaluation, Safer Dx item completion, diagnostic error determination; adverse event and harm evaluation; and DEER failure point assessment. The EHR case review process and abstraction instrument were iteratively refined based on feedback from DE experts and clinician adjudicators participating in reviews of test cases.

Table 1:

Structured EHR case review process for assessing diagnostic error in acute care.

Step	Section	Description
1	Case information	–Review admission timeline (admit time, transfer location, admission from ED status)
		–Provide brief case summary based on review of discharge summary
2	Diagnostic timeline evaluation	–Evaluate chief complaint, timeline of diagnosis and management and treatment, discrepancies between the main diagnoses documented at admission and discharge, and whether there were secondary diagnoses that were not appropriately addressed
3	Safer Dx instrument	–Enter responses to all items across using a 6-point Likert scale based on chart review
		–Use responses to aid in determining the likelihood of diagnostic error during the encounter
4	Diagnostic error determination	–Record the number of diagnostic errors and describe each error
		–Consider possible biases and improper weighing of evidence that may have led to DE (assess the presence of “hypothesis generating” process failures)
5	Harm assessment	–Determine if an adverse event (actual harm) occurred, or if there was potential for harm
		–Rate the severity, preventability, and ameliorability of the adverse event(s)
6	DEER taxonomy: Diagnostic failure point classification	–Assess presence of diagnostic process failures across each diagnostic dimension [33]: patient-provider encounter and initial assessment (access/presentation, history, physical exam); diagnostic test ordering, performance, and interpretation; diagnostic information and patient follow-up; subspecialty consultation or referral; healthcare team communication and collaboration; and patient experience
		–Identify the most significant failures leading to diagnostic error(s)

^aBranching logic directs adjudicators to review specific parts of EHR based on admission and discharge diagnoses, and appropriate treatment documentation. For instance, if there is preliminary evidence of a delayed diagnosis (discharge diagnosis substantially different than the admission diagnosis), the tool suggests that the reviewer examine timestamps for diagnostic tests, labs, medications, and consultation orders relevant to the diagnostic process. ^bModified diagnostic error evaluation and research (DEER) taxonomy was previously adapted for acute care by mapping individual diagnostic process failure points to the original six Safer Dx domains and including failure points for two additional dimensions (patient experience; healthcare team communication and collaboration) [37, 38]. The final tool included a total of 41 failure points mapped to eight diagnostic process dimensions, plus the three “hypothesis generation” failure points that were required to be completed in all cases (Step 4) in which DE was confirmed [37, 38]. ^cThe adjudication procedure requires clinicians to independently conduct case reviews in the EHR using our chart abstraction instrument.

Hospital-based physicians who worked on the general medicine service were trained to evaluate the diagnostic process over the acute episode of care. Specifically, they were trained to review the discharge summary (including primary and secondary diagnoses); emergency room course; initial history and physical note by the primary team; initial consultant notes; objective data (lab results, timing of treatments, consults, procedures, etc.); progress notes (primary team, consultants, nurses, ancillary staff, etc.), and other relevant documentation (significant event notes). Clinician reviewers, blinded to the institutional mortality case preventability determination, were randomly assigned sampled cases to co-adjudicate in pairs. Adjudicators independently reviewed each case and recorded all responses in the REDCap abstraction form (Supplementary Figure S1), including their DE determination. Adjudicators met to resolve discrepancies during a final consensus review session. A panel of experienced hospitalist physicians (AKD, JLS) reviewed all cases, which included a review of the chart abstraction forms completed by the adjudicator pair, as well as the relevant ED-to-hospital encounter in the EHR. DE was confirmed if a missed opportunity to make a correct or timely diagnosis within the context of an evolving diagnostic process was evident [34].

Electronic triggers

To identify the types of hospital events that could be used to detect DE, we surveyed the literature and engaged with subject matter experts to explore potential e-trigger criteria associated with DE. Based on work by Shenvi and El-Kareh (2014), we defined the following e-trigger categories (Table 2): multiple teams (≥3 transfers or ≥3 consultations); unexpected events (a code, rapid response, unexpected surgery, or cancelled procedure); diagnostic documentation (a symptom-oriented principal problem at discharge, a discrepancy between encounter diagnoses upon admission and discharge, or a change in principal problem on problem list); and clinical deterioration (suboptimal lactic acid clearance, acute kidney injury, persistent or increasing hypoxia, or sustained fever) [23, 26]. We worked with a data analyst to code specific queries corresponding to individual e-triggers based on these criteria. For instance, our diagnostic documentation-related e-triggers emulated previously reported methods for defining discrepancies in documentation at various points of the care timeline, such as ICD code mismatches between the admission diagnosis and the hospital principal problem [35]. Similarly, we defined symptom-oriented diagnoses as those with ICD-10-CM codes between R00 and R99, capturing general symptoms, signs, and abnormal clinical and laboratory findings, not elsewhere classified. For all cases in our cohort, we queried the EDW to determine if these specific e-triggers were present.

Table 2:

E-trigger categories and definitions.

Multiple teams:

(1) 3 or more consultations; OR

(2) 3 or more transfers between teams / services (excluding transfers to critical care)

Unexpected events:

(1) Code or rapid response; OR

(2) ICU transfer after 24 h; OR

(3) Cancelled procedure; OR

(4) Unexpected surgery

Diagnostic documentation:

(1) A symptom-oriented problem entered as the hospital principal problem at discharge; OR

(2) A discrepancy between the primary encounter diagnosis upon admission and discharge; OR

(3) A change in principal problem on the hospital problem list

Clinical deterioration:

(1) Sustained fever defined as two temperature readings of >100°F occurring in a rolling 24 h period ≥48 h after admission); OR

(2) Poor lactic acid clearance defined as the occurrence of a repeat lactic acid value that is ≥10% higher than the previous value and is also ≥2.0; or a repeat lactic acid value, taken within 12 h of the first reading, that is ≥90% of the first value if the first value was ≥2.0; OR

(3) Acute kidney injury (AKI) defined as creatinine (CR) values within 48 h of admission that are ≥1.5× the initial creatinine CR value; a high CR value that fails to resolve (must be <1.44 or 50% lower than the triggering value) within 72 h for patients without chronic kidney disease (CKD) or end stage renal disease (ESRD) on the problem list; a low glomerular filtration rate (GFR) (<48) that fails to improve (>48 or 2× the triggering value) within 72 h for patients without CKD or ESRD on the problem list; or a high CR value that fails to resolve (<1.44 or 50% lower than the admission value) or a low GFR value that fails to improve (>48 or 2× admission value) within 72 h for patients with AKI on following 7 days of that lowest value; OR

(4) Persistent or increasing oxygen requirement defined for patients requiring a nasal canula as an average O2 flow rate for any 24 h period after the first 2 calendar days post admission that is greater than the average O2 flow rate during those first 2 days; a change in O2 device (from nasal cannula to any other device OR from none to any device) at least 2 times in a 24 h period >48 h post admission; or the average O2 flow rate for any 24 period is >2L or the patient requires an O2 device other than none or a nasal cannula at least twice in 24 h period >72 h after admission

Measures

Our main measures included the proportion of cases with one or more DEs, DEs contributing to death, mean DPFs per case, and mean e-triggers per case. Secondary measures included mean DPFs by dimension per case and mean e-triggers by category per case.

Statistical analysis

We used descriptive statistics to report demographics and main measures for each test cohort. We calculated Cohen’s kappa for agreement on DE determination between adjudicators based on their independent reviews, and between adjudicators’ final consensus review and the tertiary panel review. We compared the frequencies of our measures in each cohort using Fisher’s exact test or the means (SD) using a two tailed t-test as appropriate. Within each test cohort, we compared the mean (SD) number of DPFs by dimension and the mean (SD) number of e-triggers by category per case in DE positive vs. DE negative cases using a two tailed t-test.

Results

Of 365 patients who expired on the general medicine service, 28 (7.7%) and 337 (92.3%) were designated preventable and non-preventable, respectively, by our institutional mortality case review process (Figure 1). After excluding cases with LOS>1 month, 27 preventable and 24 non-preventable cases were reviewed independently by two trained adjudicators using our structured EHR case review process. Most demographic variables (Table 3) were balanced, though LOS was longer in the non-preventable cohort. Inter-rater reliability for DE determination was moderate between individual reviewers (Cohen’s kappa 0.41), and substantial between final consensus reviews and tertiary review by the expert panel (Cohen’s kappa 0.74).

Figure 1:

Consort diagram.

Table 3:

Demographics of preventable and non-preventable expired patient cohort determined by formal institutional mortality case review.

Characteristic	Preventable cohort, n=27	Non-preventable cohort, n=24	p-Value
Age, years, mean (SD)b	77.1 (11.8)	79.9 (9.0)	0.13
Female gender, n (%)c	13 (48)	12 (50)	>0.99
Race, n (%)c
Caucasian	20 (74)	21 (88)	0.30
Non–caucasian	7 (26)	3 (1)
Ethnicity, n (%)c
Non-hispanic	22 (81)	21 (88)	0.59
Hispanic	2 (7)	0 (0)
Missing	3 (11)	3 (13)
Primary language English, n (%)b	23 (85)	20 (83)	>0.99
Socioeconomic status (median income by zip code)–no (%)a
Less than or equal to $47,000	5 (19)	3 (13)	0.48
$47,001–$63,000	4 (15)	7 (29)
Greater than $63,000	18 (67)	14 (58)
Insurance status, n (%)a
Private (commercial)	5 (19)	3 (13)	0.85
Public (medicaid, medicare)	21 (78)	20 (83)
Other (self pay-free care)	1 (4)	1 (4)
Length of stay – mean (SD)b	8.1 (7.4)	12.2 (9.5)	0.06
Admission service (%)c
Medicine	21 (78)	17 (71)	0.75
Non-medicine	6 (22)	7 (29)
Primary care physician, n (%)c
Network	10 (37)	8 (33)	>0.99
Non-network	17 (63)	16 (67)
Admission diagnosis, n (%)c
Symptom or undifferentiated	8 (30)	7 (29)	0.97
Disease specific	19 (79)	17 (71)
Primary diagnosis at death, n (%)c
Symptom or undifferentiated	8 (30)	3 (13)	0.14
Disease specific	19 (79)	21 (88)
Van-walraven elixhauser comorbidity score, %a
1–5	13 (48)	8 (33)	0.53
6–10	13 (48)	14 (58)
11 or more	1 (4)	2 (8)
Elixhauser comorbidities–mean, SDb	6.0 (2.8)	6.6 (2.4)	0.22
MSDRG weight–mean, SDb	2.3 (1.3)	2.9 (2.4)	0.22
ADRDRG weight–mean, SDb	3.6 (2.1)	3.5 (2.9)	0.22
Inpatient encounters within prior 6 months (%)a
0	15 (56)	10 (42)	0.60
1–2	8 (30)	10 (42)
3 or more	4 (15)	4 (17)
Ambulatory encounters (ED, urgent care, PCP) within prior 6 months, n (%)a
0	15 (56)	16 (67)	0.44
1–2	9 (33)	5 (21)
3 or more	3 (11)	3 (0.13)

^aChi-square test, ^bWilcoxon rank sum, and ^cFisher’s exact test for ordinal, continuous, and dichotomous variables, respectively.

Main measures for the preventable and non-preventable cohorts are reported in Table 4. There were no significant differences for overall frequency of DEs, mean DPFs per case, and mean e-triggers per case. The frequency of DE contributing to death was significantly higher in the preventable compared to the non-preventable test cohort (55.6% vs. 16.7%, OR [95% CI] 6.25 [1.68, 23.27], p<0.01). Descriptions of four non-preventable cases with DEs that contributed to death based on our structured EHR case review process, as well as associated DPFs and e-triggers, are provided in Table 5.

Table 4:

Main measures of diagnostic error from structured EHR case review in two test cohorts.

Main measures	Preventable cohort, n=27	Non-preventable cohort, n=24	OR [95% CI]	p-Valuea
DE, 1 or more – no (%)	18 (66.7)	14 (58.3)	1.43 [0.46,4.47]	0.57
DE contributing to death – no (%)	15 (55.6)	4 (16.7)	6.25 [1.68, 23.27]	<0.01
Diagnostic process failures per case – mean (SD)	6.22 (3.56)	4.96 (4.80)	1.26 [−1.10, 3.62]	0.16
E-triggers per case – mean (SD)	4.93 (2.26)	4.96 (1.93)	−0.03 [−1.22, 1.16]	0.59

^aFisher’s exact test or ^at-test as appropriate. Forty (78.4%) cases were found to have one or more associated TEs (mean (SD) of 1.6 (0.68) TEs per case) without significant associations between the preventable and non-preventable cohorts (20 (74.1) vs. 20 (83.3), OR 0.57 [0.14, 2.26], p=0.5¹. Bold value signifies statistically significant results.

Table 5:

Non-preventable cases determined by institutional mortality case review with a diagnostic error contributing to death detected by structured EHR review.

Case, description of diagnostic error, tertiary panel comments	Diagnostic process failures
	E-triggers
Case 1. A 91 y/o female presented with cough and lower extremity swelling after a recent hospitalization. In the ED, the patient was diagnosed with pneumonia based on CXR and started on broad spectrum antibiotics. The admitting team’s initial working diagnosis was pneumonia and heart failure. Lower extremity Dopplers ordered non-urgently showed acute and chronic clots bilaterally. After reconsidering the working diagnosis, a heparin drip was started for presumed pulmonary emboli. The patient clinically deteriorated, requiring escalating oxygen for respiratory failure. After addressing goal of care, the patient was transitioned to comfort measures and expired	Failure or delay in recognizing or acting upon urgent condition or complications
A missed opportunity to diagnose acute pulmonary embolism at admission. The admitting team anchored on the diagnosis of pneumonia, though procalcitonin was negative (0.1 ng/mL). Venous-thromboembolic disease was not considered in the initial differential diagnosis despite evidence of tachycardia, hypoxia, asymmetric lower extremity swelling, and suggestive ECG findings	Clinical deterioration (O2, AKI)
Case 2. An 87 y/o female with CAD s/p PCI was transferred from a referring hospital for medical optimization prior to hip surgery after evaluation of a fall revealed right femoral neck fracture. Elevated troponins, hypoxemia, and high NT-pro-BNP were attributed to demand ischemia and this was communicated to the patient. During surgery, the patient had a PEA arrest, was resuscitated, and emergently taken for cardiac catheterization which showed a severe right ventricular infarct and failure. She was placed on an intra-aortic balloon pump but developed ventricular fibrillation and expired	Erroneous clinician interpretation of test
A missed opportunity to diagnose ACS and right heart failure . The elevated troponins and initial ECGs were incorrectly attributed to demand ischemia. The patient had increasing troponins, evolving ECG changes, and echo findings suggesting RV infarct in the 24 h prior to surgery, but these findings were also not correctly interpreted. While the primary team did communicate their concerns as part of peri-operative cardiac risk assessment, cardiology was never consulted	Multiple teams (transfers). Unexpected events (ICU transfer, code). Diagnostic documentation (discrepant diagnosis). Clinical deterioration (AKI)
Case 3. An 87 y/o female was transferred from OSH after fall with new tib-fib fracture. Due to altered mental status, the history was primarily collected through review of OSH records. The patient was noted to be hypoxemic and volume overloaded on initial exam, but the team was primarily focused on managing atrial fibrillation and rapid ventricular rate. As the hospital course progressed, the patient developed new AKI and sepsis for which antibiotics were initiated. The patient declined rapidly, was transitioned to comfort measures, and expired	Suboptimal weighing of a physical examination finding. Failure or delay in performing needed test(s)
A missed opportunity to diagnose SBP upon admission . The admitting clinician commented on a nodular liver and ascites noted on OSH imaging, but sub-optimally weighed the possibility of SBP, deferred a decision to obtain a diagnostic paracentesis and initiate antibiotics. Later during the hospital course, the team became more concerned about SBP and HRS. While attempting to get a paracentesis, the patient became progressively hypotensive, and goals of care were addressed	Unexpected events (code, cancelled procedure). Clinical deterioration (lactate, O2, AKI)
Case 4. A 92 y/o male with heart failure with preserved ejection fraction, atrial fibrillation, and recently diagnosed pancytopenia (MDS), was admitted with dyspnea and confusion concerning for pneumonia, and was treated with broad spectrum antibiotics. The hospital course was complicated by tachycardia and volume overload. After a rapid response/code, the patient was transferred to cardiac intensive care for worsening respiratory distress and pulmonary edema. He underwent attempts at diuresis assisted with pressor support but became progressively oliguric. The family elected to advance code status and withdraw care	Erroneous clinician interpretation of test. Failure or delay in recognizing or acting upon urgent condition or complications. Failure or delay in ordering a referral or consult
A missed opportunity to diagnose heart failure decompensation on admission as etiology of hypoxia, symptoms, elevated BNP, and worsening pulmonary infiltrates . Infectious etiologies were over-prioritized early during hospitalization despite clear clinical evidence of cardiopulmonary dysfunction. While clinical signs of decompensated heart failure were evident upon admission, the patient received intravenous fluids instead of aggressive diuresis, leading to worsening tachycardia, hypotension, and hypoxia for 3 days prior to transferring to cardiology	Multiple teams (consults). Unexpected events (ICU, code). Diagnostic documentation (undifferentiated diagnosis at discharge). Deterioration (lactate, fever, O2, AKI)

O2, persistent or increasing oxygen requirement; AKI, acute kidney injury; Lactate, poor lactic acid clearance; Fever, sustained fever; Consults, multiple consultations. Bold and italicized text signify the description of the diagnostic error that occured in the case. CAD, coronary artery disease; PCI, percutaneous coronary intervention; PEA, pulse-less electrical activity; ECG, electro-cardiogram; RV, right ventricular; OSH, outside hospital; SBP, spontaneous bacterial peritonitis; HRS, hepato-renal syndrome; MDS, myelodysplastic syndrome.

Of the 27 cases in the preventable cohort, 18 (66.7%) were confirmed to have one or more DEs (mean (SD) of 1.72 (0.66) DEs per error-positive case) by our process. Of the 18 DE positive cases, 15 (83.3%) were associated with failure or delay in considering the correct diagnosis; 15 (83.3%) were associated with suboptimal weighing or prioritization of a primary or secondary diagnosis; 3 (16.7%) were associated with too much weight to a lower probability or priority diagnosis; and DE was determined to have contributed to death in 15 (83.3%).

Of the 24 cases in the non-preventable cohort, 14 (58.3%) were confirmed to have one or more DEs (mean (SD) of 1.29 (0.47) DEs per error-positive case) by our process. Of the 14 DE positive cases, 12 (85.7%) were associated with failure or delay in considering the correct diagnosis; 9 (64.3%) were associated with suboptimal weighing or prioritization of a primary or secondary diagnosis; none were associated with too much weight to a lower probability or priority diagnosis; and DE was determined to have contributed to death in 4 (28.5%).

Secondary measures for both test cohorts are reported in Table 6. For DE positive compared to DE negative cases within each cohort, the mean number of DPFs per case was significantly higher overall, and either significantly or non-significantly higher in six process dimensions: history, physical exam and assessment; diagnostic test ordering, performance, and interpretation; diagnostic information and patient follow-up; subspecialty consultation/referral; and healthcare team communication. For DE positive compared to DE negative cases within each cohort, the mean number of e-triggers per case was non-significantly higher overall, and in three categories: multiple teams, unexpected events, and clinical deterioration.

Table 6:

Secondary measures.

	Preventable cohort, n=27				Non-preventable cohort, n=24
	DE+ n=18	DE− n=9	Effect sizea [95% CI]	p-Value	DE+ n=14	DE− n=10	Effect sizea [95% CI]	p-Value
Diagnostic process failures per case – mean (SD)
Access/presentation	0.22 (0.43)	0.56 (0.53)	−0.34 [−0.73, 0.05]	0.09	0.5 (0.65)	0.3 (0.67)	0.20 [−0.37, 0.77]	0.47
History	1.22 (1.17)	0.56 (1.13)	0.66 [−0.31, 1.63]	0.17	1.29 (1.14)	0.3 (0.95)	0.99 [0.07, 1.91]	0.04
Physical exam/assessment	0.89 (1.37)	0.11 (0.33)	0.78 [−0.18, 1.74]	0.11	1 (1.18)	0.1 (0.32)	0.90 [0.10, 1.70]	0.03
Diagnostic test ordering, performance, interpretation	1.56 (1.10)	0.33 (0.5)	1.23 [0.43, 2.03]	<0.01	1 (1.04)	0.1 (0.32)	0.90 [0.19, 1.61]	0.02
Diagnostic information, follow-up	1.94 (1.39)	1.22 (1.20)	0.72 [−0.40, 1.84]	0.20	1.5 (1.56)	0.5 (1.58)	1.00 [−0.35, 2.35]	0.14
Subspecialty consultation	0.83 (0.79)	0.33 (0.71)	0.50 [−0.14, 1.14]	0.12	0.64 (0.84)	0.2 (0.42)	0.44 [−0.16, 1.04]	0.14
Healthcare team communication	0.78 (1.06)	0.11 (0.33)	0.67 [−0.08, 1.42]	0.08	0.64 (1.01)	0.4 (0.70)	0.24 [−0.53, 1.01]	0.52
Patient experience	0.17 (0.38)	0.22 (0.44)	−0.05 [−0.39, 0.29]	0.76	0.36 (0.63)	0.3 (0.67)	0.06 [−0.50, 0.62]	0.82
DPFs per case	7.61 (3.18)	3.44 (2.55)	4.17 [1.65, 6.69]	<0.01	6.93 (4.83)	2.2 (3.26)	4.73 [1.07, 8.39]	0.01
E-triggers per case – mean (SD)
Multiple teams	1.06 (0.80)	0.56 (0.89)	0.50 [−0.20, 1.20]	0.15	1.07 (0.83)	0.9 (0.99)	0.17 [−0.60, 0.94]	0.65
Unexpected events	1.33 (1.03)	0.67 (0.87)	0.66 [−0.17, 1.49]	0.11	1.43 (0.94)	1 (0.82)	0.43 [−0.34, 1.20]	0.26
Diagnostic documentation	0.61 (0.70)	0.78 (0.67)	−0.17 [−0.75, 0.41]	0.55	0.5 (0.65)	0.7 (0.67)	−0.20 [−0.77, 0.37]	0.47
Clinical deterioration	2.33 (0.88)	2.11 (0.87)	0.22 [−0.52, 0.96]	0.54	2.29 (0.88)	1.9 (1.04)	0.39 [−0.42, 1.20]	0.33
E-triggers per case	5.33 (1.97)	4.11 (2.56)	1.22 [−0.61, 3.05]	0.18	5.29 (1.67)	4.5 (2.16)	0.79 [−0.83, 2.41]	0.32

^aEffect size reported as a difference in means. DE+, diagnostic error positive case; DE-, diagnostic error negative case. Bold values in the last row “DPFs per case” and “E-triggers per case” signify summative values of the corresponding rows above.

Discussion

We applied a structured EHR case review process using validated instruments (Safer Dx, DEER Taxonomy) to assess the presence of DE(s) during the hospital encounters of patients who expired on the general medicine service and whose deaths were determined to be preventable or non-preventable by our “gold standard” institutional mortality case review process. Inter-rater reliability was moderate between individual case reviews, but substantial between the final consensus and expert panel reviews. The frequency of DE overall in each cohort was high, accounting for both harmful and non-harmful errors in primary and secondary diagnoses. Importantly, while cases with DEs contributing to death were more frequently identified in the preventable cohort, they were also present in the non-preventable cohort. Within each cohort, mean DPFs per case were significantly higher for DE positive compared to DE negative cases, with contributions from 6 of the 8 process dimensions. Lastly, the mean number of e-triggers per case was non-significantly higher for DE positive compared to DE negative cases in the preventable cohort, mostly driven by the multiple teams, unexpected events, and clinical deterioration categories.

Our findings may have several explanations. First, our laborious EHR case review process was designed to allow clinician adjudicators to evaluate the diagnostic process for both primary and secondary diagnoses, with adjudicators initiating case reviews starting from the time that the patient arrived at our institution (i.e., emergency department or direct transfer into our hospital) until expiration on the general medicine service. Second, adjudicators who reviewed orders, medication administration times, laboratory tests, study results, and daily visit notes by the primary team, consultants, nurses, and other health professionals considered the impact of delayed diagnoses spanning the course of hours to a few days as having much greater consequence in acute care. For example, adjudicators often detected less impactful DEs (a delay in considering pulmonary edema early during hospitalization, leading to inadequate diuresis in a complex patient with multiple chronic conditions) as suggested in the 10 cases of DE in which the death was rated as non-preventable by the institutional mortality case review process. These cases (Supplementary Table S1) had noticeable delays in establishing a key diagnosis, but the impact of the harm attributed to DE was variable (no harm, moderate harm, major harm), and ultimately not implicated in death. Third, the threshold for calling a DE leading to preventable harm was likely lower in high-morbidity conditions (sepsis, mesenteric ischemia), especially when corresponding failures in the diagnostic process were identified using the modified DEER Taxonomy or led to harm due to delayed administration of appropriate therapies. Fourth, by incorporating standardized harm assessment (none, mild, major, death) criteria [36], individual adjudicators were able to distinguish between DEs that were implicated in death with DEs that were not implicated in death (but which may have contributed to non-death harm). Finally, all cases were deliberated by adjudicators during a crucial consensus review to resolve discrepancies in DE determination, harm designation, and DPF identification, which likely led to substantial agreement with tertiary case reviews conducted by our expert panel.

The frequency of DE contributing to death in the preventable cohort was higher than rates reported in autopsy studies which tend to detect cases in which the main diagnosis was either missed or incorrect, but not delayed [2, 37, 38]. These prior studies may underestimate DE rates based on contemporary definitions which characterize DEs as ‘missed opportunities’ to establish a timely primary diagnosis for the encounter [34]. In contrast, our adjudicators were trained to assess the evolving and dynamic nature of the diagnostic process and also scrutinized clinically important secondary diagnoses that contributed to harm in the case (e.g., failure to diagnose metabolic alkalosis contributing to altered mental status and an aspiration event in a patient hospitalized for a heart failure exacerbation requiring prolonged diuresis). In the context of hospital care, establishing a timely diagnosis (establishing the diagnosis of sepsis or pulmonary embolism at the time admission with supporting clinical data) for both primary and secondary diagnoses is crucial to instituting appropriate goal directed therapy independent of whether the patient experienced a harmful outcome. Thus, institutional processes that use arbitrary ratings to characterize the preventability of overall harm in a specific case risk missing such clinically meaningful delays and potentially overlooking faulty processes that would eventually lead to a harmful outcome in other cases if not corrected [27]. While our approach did not incorporate direct input from clinicians via an electronic survey administered at the time of death, our use of the modified DEER Taxonomy adapted for acute care did empower adjudicators to critique the diagnostic process to identify one or more of the 41 potential failures on a case-by-case basis [30]. Furthermore, by enumerating these DPFs within each diagnostic process dimension, our process yielded more quantitative data that could be used for identifying potential targets for safety improvement activities than currently provided by the institutional mortality case review process [27, 30].

As part of our exploratory analysis, we also quantified the presence of 13 e-triggers, postulated to identify enriched cohorts with DE [26], within four clinical categories: multiple teams, unexpected events, diagnostic documentation, and clinical deterioration. Our observation of a trend towards a higher number of e-triggers in DE positive compared to DE negative cases appeared to be driven by three categories (multiple teams, unexpected events, clinical deterioration) in both cohorts, a finding that merits additional evaluation prior to use for DE case surveillance. Such larger validation studies might entail iterative development and validation of individual or combinations of these e-triggers per established frameworks such as the Safer Dx Trigger Tools Framework [24, 25, 39], [40], [41].

Our study has several limitations. First, our sample was small and limited to expired patients with a length of stay <1 month; however, as an exploratory study, a comparison of the performance of our EHR review process in two institutionally defined cohorts enabled us to explore potential utility of our approach for case surveillance in acute care where there is a dearth of data. Second, our EHR case review process was retrospective, and thus, did not consider information provided by clinicians involved in cases, which likely limited our ability to identify cognitive factors and communication issues, which may not be well documented in the EHR. Third, though clinician adjudicators were instructed to assess the likelihood of DE in context of the evolving diagnostic process, the risk of “playing Monday morning quarterback” may have inflated error rates, especially in patients who had poor outcomes (death). However, tertiary review by our expert panel served to provide a level of quality control. Fourth, DE assessment and harm determination may have been affected by subjectivity: the optimal “window of opportunity” for what constitutes a timely and accurate diagnosis in acute care may vary by both diagnosis (e.g., acute pulmonary embolus vs. eosinophilic pneumonia) and individual adjudicator experience. However, we aimed to improve consistency by providing additional guidance to reviewers through extensive training and tertiary review by an expert panel. Finally, while we used a modification of the original Safer Dx instrument validated for the ambulatory setting, we incorporated guidance on how to revise it for different settings [3, 21, 22]. We offer specific lessons and recommendations (Table 7) for applying the Safer Dx instrument. A key recommendation based on our experience is that the threshold for calling a delayed diagnosis a DE varies according to the diagnosis (e.g., sepsis) under scrutiny and the potential clinical impact of that diagnosis if not made in a timely manner (e.g., delay in care escalation, including ordering lactic acid levels and blood cultures, initiating broad-spectrum antibiotics, and restoring tissue perfusion within 6 h). Given the nuances of this review process, we underscore a tertiary review by an expert panel to ensure consistency of error characterization within context of the evolving diagnostic process, and whether it was implicated in harm.

Table 7:

Common pitfalls and strategies for determining diagnostic error in acute care.

Example of common pitfall	Strategies for addressing pitfalls
Hindsight bias; playing “Monday morning quarterback”	–Evaluate the diagnostic process during the hospital encounter as evolving rather than as individual events
	–Under ideal conditions, ask whether something could have been done differently during the diagnostic window of opportunity, with the information available at that time
Not calling delayed diagnosis a diagnostic error	–Recognize that the optimal window of opportunity for considering a delayed diagnosis an error varies by diagnosis: sepsis (6 h), STEMI (90 min), ischemic stroke (3 h)
	–Be cognizant of the ideal time frames when a high-risk diagnosis must be recognized to prevent the worst outcome
Under-calling diagnostic error due to overweighing subjective data and underestimating objective data	–Irrespective of the primary working diagnosis that is documented, review and interpret objective data such as vital signs, imaging, and labs
	–Focus not just on isolated values but on trends over the encounter
In-group favoritism: Under-calling or overcalling a diagnostic error when a familiar clinician cared for the patient	–Do not assess a chart if you were involved in the case
	–Understand that diagnostic errors are not caused by an individual but are the results of multiple system failures and cognitive biases
	–Ensure that the case review process is a psychologically safe activity free of blame or accusation
Over emphasizing errors not related to the primary working diagnosis at admission.	–Apply the Safer Dx tool to the primary working diagnosis at time of admission for the hospital encounter
	–Provide clinician reviewers a clear process for assessing diagnostic errors for secondary diagnoses identified during the chart review (focus on diagnoses with harmful consequences)
Diagnoses are not clear at discharge, Safer dx determination is equivocal, no further documentation available	–If many errors in the diagnostic process are identified via the DEER taxonomy, consider the presence of diagnostic error, even if the “true” diagnosis is not known
Subjectivity in classification of error type	–Differentiate diagnostic from therapeutic error through training and examples (e.g., antibiotics given before assessing infection using blood cultures is better classified as a therapeutic error)

Conclusions

We developed a structured EHR case review process using validated instruments for assessing the likelihood of DEs and identifying DPFs during the hospital encounter, and assessed its performance compared to our gold standard institutional mortality case review process. While our experience suggests potential for improved detection of DEs compared to traditional methods, our process should be further validated in larger cohorts and at other institutions. Similarly, while our analysis of e-triggers provides early data suggesting potential utility for DE case surveillance, larger studies are required to validate e-triggers individually or within clinical categories. Such studies will serve as crucial steps for outcomes measurement in trials of interventions to promote diagnostic safety.

Supplementary Material

Supplementary Material

Supplementary Material

Supplementary Material

The online version of this article offers supplementary material (https://doi.org/10.1515/dx-2022-0032).