The Diagnostic Performance Feedback “Calibration Gap”: Why Clinical Experience Alone Is Not Enough to Prevent Serious Diagnostic Errors

Medical diagnostic errors can be thought of as the bottom of the iceberg of patient safety—a hidden yet vast source of morbidity and mortality.1 According to the U.S. National Academy of Medicine, diagnostic errors represent a major public health problem likely to affect each of us in our lifetime.2 Diagnostic errors contribute to approximately 10% of deaths and 6% to 17% of hospital adverse events and are the leading cause of medical malpractice claims.3 Although typically multifactorial, the majority of diagnostic errors can be traced back to failures in bedside examination skills and clinical reasoning;4 knowledge and skill gaps appear to play important roles that have been underestimated in the context of an overemphasis on cognitive bias as a cause.5

A critical unanswered question for educational strategies to improve diagnosis is how diagnostic errors could remain so common, even with clinical presentations seen daily in clinical practice. In theory, accumulated clinical experience gained over time should be an “antidote” that gradually eliminates misdiagnosis, but experimental studies suggest that more years of experience does not necessarily confer greater diagnostic accuracy.6 Part of the problem is that feedback is essential for improved diagnostic performance, but is often lacking.7 For example, it has been shown that some short‐term deaths after discharge from the emergency department (ED) likely reflect missed diagnoses of life‐threatening illnesses.8 Unfortunately, such feedback rarely returns to individual ED clinicians, which prevents “recalibration” (i.e., adjusting one's own mental models for diagnosis based on real‐world accuracy in prior similar cases) that would otherwise improve diagnostic performance.9 In this article we argue that, absent systematic feedback, even years of sustained clinical practice may not produce the necessary experiential learning to prevent critical diagnostic errors, particularly for high‐risk, low‐frequency conditions.

As a representative case, take the known public health problem of missed stroke in patients presenting with acute dizziness to the ED, where 45,000 to 75,000 strokes are missed at first contact each year10 and an estimated 10,000 to 25,000 serious preventable harms result from missed opportunities for early treatment.10 Dizziness and vertigo are common problems, accounting for approximately 3% of all ED visits, but only approximately 3% to 5% of these are due to stroke.10 A typical full‐time clinical ED physician at a medium‐sized hospital ED (~60,000 visits per year) will see about 3,000 patients with dizziness each decade, 120 due to stroke (Table 1, Figure 1). Roughly 80% will have isolated vestibular symptoms without general neurologic signs12 and a large fraction of these will be misdiagnosed with inner ear problems (or some other form of presumptively “benign” dizziness) and discharged. Only approximately 15% to 20% will go on to have major stroke in the days and weeks after minor stroke or transient ischemic attack13 and roughly 40% of these will return to a different ED in the region14 (assuming that they do not die before making it to the hospital). So, absent systematic follow‐up, an ED physician with a 40% miss rate11 might find out about only approximately one dizzy stroke patient per decade they personally called “benign,” despite having missed 50 such strokes. Miscalibration occurs because that physician's perceived experience is of having recognized 70 (obvious) strokes and only missing one (not‐so‐obvious) stroke, giving an internal mental‐model estimate of 99% sensitivity, despite an actual sensitivity of 60%. Varying the estimated diagnostic accuracy from 20% to 80% for dizzy strokes does not change the bottom line—everyone overestimates their accuracy when they do not get feedback about the cases they miss.15 This leads to a massive “calibration gap” that likely keeps diagnostic performance stagnant over time (and to a general misconception on the part of clinicians that such errors are very rare events).

Table 1.

Calibration Gap for Missed Stroke in Dizziness for an Average Full‐time Clinical ED Physician

Parameter	No. per year	No. per decade
ED visits	60,000	600,000
Dizziness visits (~3%)	1,800	18,000
Dizziness patients while ‘on shift’ (~16%–18%)	300	3,000
Dizziness while on shift due to stroke (~3%–5%)	12	120
Obvious strokes (e.g., with neurologic signs) recognized (~60%a)	7	70
Subtle strokes (e.g., isolated vertigo with only eye/gait signs) missed (~40%a)	5	50
Untreated missed strokes returning with major stroke < 90 days (~15%–20%)	1	10
Missed strokes returning with major stroke to the same hospital (~60%)	< 1	6
Missed strokes returning with major stroke while “on shift” (~16%–18%)	~0	1
Total missed strokes (actual count)	5	50
Total missed strokes w/direct feedback for calibration (perceived count)	~0	1

^aThe precise frequency of ED stroke misdiagnosis in dizziness is unknown, but best estimates suggest it is ~40%11

Figure 1.

Leaky pipeline of feedback on diagnostic performance. There is an enormous gap between actual diagnostic performance and perceived diagnostic performance, because of loss to follow‐up. Initial artwork modified from http://www.techvision21.com/the-bachelors-to-ph-d-pipeline-is-not-leaking-women-and-underrepresented-minorities/). Dx = diagnosis.

This calibration gap could obviously be narrowed by “closing the loop” on diagnostic performance through systematic, prompt, personalized feedback on diagnostic process, accuracy, error, and harms for all clinical cases. As yet, however, no effective mechanism exists for providing such routine, operational feedback at the individual provider level.16 Furthermore, the relative infrequency of cases due to dangerous disorders such as stroke would likely produce very slow learning, over years or decades. While live patient experiences with full, immediate follow‐up on diagnostic performance (both process and outcomes) are probably the most potent for achieving learning and expertise,17 the overall low prevalence of dangerous disorders makes it nearly impossible (even with perfect feedback) for such expertise to develop efficiently.

An alternative would be to leverage novel simulation‐based educational approaches in a blended learning format to achieve expertise through deliberate (topic‐focused) and mixed (epidemiologically valid) practice.18, 19 The theory of deliberate practice states that improved performance emerges when one engages with the content, using deliberate effort, followed by feedback on diagnostic performance.20 Simulation‐based medical education with deliberate practice has been demonstrated to be superior to traditional clinical education.21 Case‐based computer simulations (often known as “virtual patients”22) are increasingly being used to enhance clinical education, and libraries of such patients could be used as a resource to support both deliberate and mixed practice with a goal of improving diagnostic expertise.17 Many virtual patients are “fake” patients developed as educationally idealized versions of specific disorders, but useful learning is enhanced when cases and associated diagnostic decisions are closer to what is encountered in clinical practice,23 so virtual patient libraries should be as “real” as possible. Although difficult to create, such libraries would be readily scalable after initial construction.

One mechanism to develop “real” virtual patient libraries would be to use digital sources of real‐world data, such as those systematically obtained as part of standardized clinical examinations and test batteries (e.g., device‐enhanced “tele‐dizzy” consultations in the ED24) or diagnostic clinical trials (e.g., Acute Video‐oculography for Vertigo in Emergency Rooms for Rapid Triage [AVERT], NCT02483429). Structured clinical histories and sensor‐derived physical examinations (e.g., eye movement recordings using video‐oculography25) could be deployed as virtual patients in existing virtual case platforms (e.g., the Virtual Interactive Case System26 or i‐Human27). Furthermore, “digital” partial task trainers for use on mobile devices (e.g., aVOR28) could be linked to these virtual patient cases to develop the psychomotor aspects of fundamental physical examination skills in parallel to critical cognitive skills of interpreting and integrating findings.

While this may sound futuristic, it may not be far off for some clinical presentations. The above‐mentioned clinical trial of video‐oculography to diagnose acute dizziness and vertigo in the ED (AVERT) is now collecting structured history and physical examination findings from patients who present to the ED with symptom of dizziness vertigo and have examination findings suggestive of inner ear or brain dysfunction. Trial materials include actual patient videos of eye movements and neurologic examinations, as well as fully vetted final diagnoses adjudicated by a multidisciplinary team using strong criterion standard assessments. The authors have already begun to create a library of virtual patient training cases that will then be tested first for their impact on simulated performance using similar cases and, eventually, real‐world diagnostic performance.

Similar approaches could be taken for other presentations known to be associated with high rates of diagnostic error and harm (e.g., chest pain/pulmonary embolus, fever/sepsis, back pain/spinal epidural abscess). Because most of the misdiagnosis‐related harms in the ED are likely due to a relatively small number of high‐risk clinical presentations and diseases (mostly vascular events and bacterial infections29), simulation‐based training for such presentations is probably realistic. Developing case materials may take time and effort, but demonstration projects such as the one we propose for dizziness and stroke could guide the way.

Awareness of diagnostic error frequency, preventability of harms, and the “calibration gap” problem is an essential first step in moving clinicians from competence toward excellence in diagnosis, a goal valued by the entire health care team.30 Relying on “accumulated years of clinical experience” is not currently meeting this need, and novel educational approaches are likely needed to achieve diagnostic excellence in everyday clinical practice.

AEM Education and Training 2018;2:339–342