A Critical Appraisal of AHRQ’s “Diagnostic Errors” Report

Abstract

The Agency for Healthcare Research and Quality (AHRQ) report “Diagnostic Errors in the Emergency Department” generated significant mass media interest with negative implications for the safety of contemporary emergency care. The assumptions and methodology underlying this report are problematic, while multiple ongoing efforts to improve the quality and quantity of diagnostic research are missed, neglected, or ignored. The AHRQ report identifies reasonable target diseases for targeting diagnostic quality improvement efforts, as well as viable methods by which to measure any initiatives impact on diagnostic error. We note additional opportunities to improve the status quo by funding emergency department-based diagnostic research and healthcare system-level patient safety research and highlighting innovative approaches to diagnostic science within emergency medicine.

Introduction

In December 2022, the Agency for Healthcare Research and Quality (AHRQ) released a comparative effectiveness review entitled “Diagnostic Errors in the Emergency Department: A Systematic Review” that received unprecedented levels of mass media coverage and swift condemnation from emergency medicine (EM) professional organizations.^1,2 To briefly summarize, the 744-page AHRQ report extrapolated from a total of three prospective studies external to the United States (U.S.) an overall diagnostic error rate of 5.7% (95% CI 4.4–7.1) and a misdiagnosis-related harm rate of 2.0% (95% CI 1.0–3.6) per U.S. emergency department (ED) visit.³ The report actually identified 279 studies, including 41 listing the distribution of diseases associated with diagnostic error or misdiagnosis-related harms, 160 reporting diagnostic error rates, and 185 describing the root causes of diagnostic error, yet they extrapolated from three studies based on their assumptions and exclusion criteria. While we applaud those seeking to define and quantify EM diagnostic error rates, we are disappointed in this incomplete selection of studies, the authors’ illogical extrapolation of outcomes, and underdeveloped contemplation of consequences or productive next steps.

The first study, a case control study from the Canary Islands off the coast of Spain published in 2006, identified ED returns as the adverse consequence of “diagnostic errors.”⁴ The study reviewed unscheduled returns to the ED within 72 hours. Of 32,523 eligible patients, there were 250 unscheduled returns (0.77%). Among these, 54% were again discharged from the ED, while very few were admitted to the intensive care unit or died (2% and 1%, respectively of the 250 ED returns which would represent 0.015% and 0.009% of the total eligible population, or 15 and 9 per 100,000 patient encounters). Additionally, the issue identified in 26% of these returns related to problems with post-ED follow-up care, which is typically beyond the control of EM physicians in the U.S., limiting the generalizability of this finding.

The second study, a prospective observational study from Switzerland of 755 consecutive patients compared the ED diagnosis with admitted patients with their hospital discharge diagnosis.⁵ Diagnostic discrepancies between ED and hospital diagnoses occurred in 12.3% and were associated with an increased hospital length of stay (10 vs. 7 days) and mortality (8.6% vs. 3.8%). Not surprisingly, diagnostic discrepancies were associated with atypical patient presentations. While ED physicians certainly aim to make a correct diagnosis, patients often appropriately receive a preliminary diagnosis pending further work-up in inpatient or outpatient settings because further diagnostics are required to confirm a diagnosis. A patient admitted with the diagnosis of abdominal pain, who has no time-critical findings and ultimately receives a more specific diagnosis is not a victim of diagnostic error. The symptoms of some disease processes like appendicitis or cholecystitis often evolve over hours or days, so reducing diagnostic errors to 0% in ED settings is pathophysiologically unrealistic. Importantly, the authors did not evaluate any associated morbidity or mortality attributable to delays in diagnosis.

The third study, a prospective study of 518 patient visits in 2005 at one of two university-affiliated EDs in Canada, found that 2% (10/503, 95% CI 1%–3.6%) included diagnostic adverse events.⁶ This study focused on high-acuity patients. To extrapolate from these studies that an ED with 25,000 annual visits will have 50 deaths due to error is inappropriate. Instead of essential and humanistic discussions around system issues that impact patient care on a daily basis, the oversimplistic, inaccurate, and sensationalistic headlines definitively declaring that 7 million people are misdiagnosed in EDs annually serves to catalyze further mistrust in the patient-physician relationship.⁷ A different spin on the overall findings of this study actually support the commendable job achieved by EM, despite the healthcare system in which we practice.

The practice of medicine in diagnosis-heavy specialties such as EM is imperfect yet both humbling and rewarding.⁸ Reducing diagnostic error is at one extreme an imperative and at the other an aspirational goal for improvement.⁹ Research in diagnostic error is notoriously challenging. This is due in part to lack of standard definitions and the retrospective nature of its determination, often based on limited documentation and suffering from a fundamental inability to know the thought process and considerations of a provider and absence of information on framing and context as relates to medical decision-making.

Problematic Components of Report

1. Definition of Diagnostic Error or Diagnostic Delay

Obtaining consensus on cases representing “misdiagnosis” is challenging. Inclusion of a study that specifically targets cases with vague presentations, and changes in diagnosis of admitted patients are not the rightful target of inquiries into diagnostic error. Diagnosis is not just an outcome but is a process. A working diagnosis is not a final diagnosis regardless of venue or provider. Patient signs and symptoms evolve and deductive reasoning is assisted by serial examinations, laboratory and imaging studies that progress over time. This report used the National Academy of Medicine definition for diagnostic error as “the failure to (a) establish an accurate and timely explanation of the patient’s health problem or (b) communicate that explanation to the patient.”³

As society’s medical safety net for all unscheduled acute care, whether pediatric, obstetric, geriatric, surgical, dental, trauma, psychiatric emergencies, or more, the ED clinical encounter too often occurs with constraints of space, time, and resources. Hospital crowding and boarding of admitted patients in the ED combined with production pressures and hospital’s priorities to minimize their patients “leaving without being seen” metrics heightens the diagnostic focus on excluding life-threatening conditions over other priorities.¹⁰ Depending on the interpretation of “accurate and timely,” the National Academy of Medicine’s definition of diagnostic error might be unrealistic if not unsafe for the ED setting with the current state of the healthcare system. While conceptually feasible, satisfying such an expectation would require expansion of current EDs, construction of additional EDs, 24/7/365 availability of advanced diagnostic and consultative resources, and anticipation of increased length-of-stays in pursuit of uniform and definitive diagnoses.

In addition, exploring the second component of the National Academy of Medicine definition around communication, efforts to improve the transfer of medical information (between physician and patient or between healthcare teams) in ED settings continue, but strategies endorsed by AHRQ like “teach back” do not effectively improve patient comprehension of diagnosis and test results in ED settings.¹¹ Furthermore, evidence to support alternative communication strategies with patients and families is often sparse.¹² The AHRQ report does not address the relevance of the National Academy of Medicine definition for EM, nor does it sufficiently contemplate the unintended consequences of carelessly adapting ED processes to reduce diagnostic error to zero or the communication obstacles that exist.

2. Diagnostic Priorities of the ED

EM’s role is the rapid evaluation, stabilization, and disposition of undifferentiated patients. While a working diagnosis is inherent to this mission, a final diagnosis is not. Diagnostic approaches are an inherent part of EM training with hypothetico-deductive, Bayesian and ‘rule out worst case scenario’ approaches all taught and used to achieve the above goal.^13,14 These cognitive approaches strive to efficiently identify time-dependent emergencies in which minutes or hours matter, but to ensure reasonable access to emergency care in an era of daily ED crowding the diagnostic approach simply does not include precision for all potential diagnoses.

The AHRQ report could be interpreted by physicians and malpractice attorneys as justification for an exhaustive approach to EM diagnostic testing, so as to reduce miss rates closer to 0%.¹⁵ The drive for specificity in the ED is not feasible financially or from a patient flow perspective and it is not rational. More scientific approaches could be employed. For example, one approach to identifying subsets of patients most likely to benefit from less available or more invasive diagnostic testing is the test-treatment threshold.¹⁶ The Academic Emergency Medicine Evidence Based Diagnostics series provides multiple quantitative examples of test-treatment thresholds for potentially life- or limb-threatening conditions like subarachnoid hemorrhage,¹⁷ septic arthritis (Figure 1),¹⁸ mesenteric ischemia,¹⁹ and scaphoid fractures.²⁰ The harms of ED over-testing and associated overdiagnosis and over-treatment are quantifiable, yet this report does not acknowledge efforts aligned with Preventing Overdiagnosis or the Lown Institute to ensure that diagnostic testing occurs for patients most likely to benefit from that testing.^21,22

Figure 1.

reprinted with permission

3. Inadequate Quantity and Relevance of Included Studies

All were non-U.S. settings. One specifically included cases with vague complaints. Two studies occur in countries where EM is not a specialty with dedicated training and certification, whereas in the U.S. EM board certification has existed for over 40 years, raising valid questions about differences in care.²³ Canada has a very different system for follow up compared to that in the U.S.

The report downplays estimates of per-visit ED misdiagnosis-related harm rates from multiple retrospective studies that identify diagnostic errors based on some triggering event, such as a return visit or hospital admission,^24–27 arguing that these were single-institution studies, that insurance claims-based follow-up to ensure capture of medical events was not pursued, and because “this group of studies systematically underestimates harms, and likely does so by a wide margin.”³ Interestingly, these are observed to identify diagnostic errors resulting in harm and to underestimate those that result in minimal or minor harm. The one U.S.-based study among these had low overall risk of bias, and reported the lowest misdiagnosis-related adverse event rate (0.012% or 12 per 100,000) and misdiagnosis-related death rate (0.00073% or 0.73 per 100,000) of any of these studies.²⁴ Alarmingly, additional U.S.-based ED research exploring diagnostic error was neither included nor excluded in this AHRQ report, missing an opportunity for additional insight.²⁸

4. Questionable Conclusions About the Root Cause of Diagnostic Errors

Malpractice claim reviews are the basis for this contention. However, any physician who has had a malpractice claim or served on a hospital credentialing committee reviewing those with a claim history knows that claims data is hardly concrete scientific evidence nor conclusive proof of error.²⁹ Concurrently, preventability assessments are notoriously subjective and retrospectively determined, fraught with hindsight and outcome biases.^30,31 Closed claim evaluations might arguably represent “tip of the iceberg” cases but are arrived at with a different agenda.

The science of cognitive decision-making is perplexing to study and effective interventions to reduce cognitive errors (or biases) remain elusive.^13,32 Nonetheless, patient safety experts in EM have acknowledged the importance of recognizing decision-making errors associated with premature closure, confirmation bias, search satisficing, and availability bias.¹⁴ Yet, the 744-page report neglected to even acknowledge the progress of EM in understanding and avoiding cognitive errors.

5. Face Validity and Transparency of External Review

The lead author of the report is a talented researcher but is a Neurologist, not an EM physician. As physicians working in EDs for decades, we would not presume to understand or opine on the complexities of diagnostic errors in the realm of inpatient or outpatient Neurology. Admittedly, the authorship team included two ED physicians. However, based on published reports, some of these reviewers believe their voices of caution related to the validity of these findings were neglected, and the extent to which EM opinions were incorporated into the report is uncertain.³³

Additional Criticism

The AHRQ report also overlooks the reality that direct evidence for many diagnostic “tests” (which includes history, physical exam, labs, and imaging) that might be felt to be standard of care is often sparse,³⁴ biased,³⁵ and limited to studies quantifying accuracy without attaining higher levels of diagnostic proof such as patient-centered or society-centered outcomes consequential to those “tests.”³⁶ Diagnostic approaches can sometimes be achieved with alternative (more available and/or less expensive) strategies and these strategies evaluated using randomized trial designs just like interventions, but as noted by the AHRQ authors too often those comparative effectiveness diagnostic trials are nonexistent or assess less relevant outcomes.³⁷ Figure 2 depicts the diagnostic hierarchy of evidence that identifies subsets of patients more likely to benefit from testing when randomized controlled trials are appropriately designed.^36,38 EM-derived clinical practice guidelines for chest pain and abdominal pain recently rendered weak or no recommendations for diagnostic approaches based upon the absence of compelling research.^39,40 Similarly, 91% of American College of Emergency Physician’s Clinical Policy recommendations are not Level A (the highest category) and 50% are Level C.⁴¹ The AHRQ report consistently neglects to highlight the paucity of diagnostic research upon which efforts to reduce diagnostic error could be built.

Figure 2.

Diagnostic Hierarchy of Evidence

Reprinted with permission.

Despite these criticisms, and despite the media spin on this report, its Executive Summary begins with the statement that “overall diagnostic accuracy in the ED is high.”³ The AHRQ report also provides tangible next steps worth considering to further improve the effectiveness of ED diagnostic decisionmaking, including Symptom-Disease Pair Analysis of Diagnostic Error for critical conditions associated with higher short-term risk if undiagnosed and untreated such as stroke or meningitis.⁴² In addition, they propose the concept of “secret shoppers” or fake patients that could be deployed as real-world quality assessment tests across individual EDs.⁴³ They also review other opportunities to identify diagnostic errors including incident reports, automatic triggers for patients with unplanned ED revisits or escalations of care,^44,45 and Leveraging Patient’s Experience to improve Diagnosis.⁴⁶ They also highlight important pitfalls in measuring diagnostic error, including differentiating discrepant diagnoses from errant processes, controlling for presenting disease severity, and over-estimating the preventability of harms.

Conclusion

As a specialty, EM continuously strives to optimize the balance between patient safety and pragmatic patient care in an increasingly challenging clinical milieu. For example, the Society for Academic Emergency Medicine Guidelines for Reasonable and Appropriate Care in the Emergency Department (https://www.saem.org/publications/grace), which are explicitly mentioned in the AHRQ report, apply rigorous GRADE methods and include patient perspectives to derive clinical recommendations for today’s ED while also highlighting prominent diagnostic knowledge gaps for funders and researchers to consider.^47,48 Other Society for Academic Emergency Medicine efforts unacknowledged by the AHRQ report include AHRQ (and other agency) funded Consensus Conferences deriving hundreds of pages of research priorities for shared decision-making and advanced imaging in ED settings.^49,50 A diagnostic accuracy textbook synthesizing ED research for history, physical exam, clinical decision aids, labs, and imaging also exists.⁵¹

Undoubtedly, evidence-based pathways to improve the process, safety, outcomes, and costs of ED diagnostics exist. EM physicians seek to improve the care they deliver. However, funders must support EM-relevant, feasible, and impactful diagnostic science research opportunities that address the enormous knowledge gaps that currently exist.⁵²

This report is significantly flawed and regrettably neglected opportunities to engage ED diagnostics experts to design, interpret, and synthesize existing research on diagnostic errors in EM. On behalf of ED physicians committed to our specialty and the patients for which we care, we invite AHRQ and other research funders to join and support us in taking the next steps to advance EM’s capacity to diagnosis in the most accurate, efficient, cost-effective, and pragmatic manner possible.