The frequency of diagnostic errors in outpatient care: estimations from three large observational studies involving US adult populations

Abstract

Background

The frequency of outpatient diagnostic errors is challenging to determine due to varying error definitions and the need to review data across multiple providers and care settings over time. We estimated the frequency of diagnostic errors in the US adult population by synthesising data from three previous studies of clinic-based populations that used conceptually similar definitions of diagnostic error.

Methods

Data sources included two previous studies that used electronic triggers, or algorithms, to detect unusual patterns of return visits after an initial primary care visit or lack of follow-up of abnormal clinical findings related to colorectal cancer, both suggestive of diagnostic errors. A third study examined consecutive cases of lung cancer. In all three studies, diagnostic errors were confirmed through chart review and defined as missed opportunities to make a timely or correct diagnosis based on available evidence. We extrapolated the frequency of diagnostic error obtained from our studies to the US adult population, using the primary care study to estimate rates of diagnostic error for acute conditions (and exacerbations of existing conditions) and the two cancer studies to conservatively estimate rates of missed diagnosis of colorectal and lung cancer (as proxies for other serious chronic conditions).

Results

Combining estimates from the three studies yielded a rate of outpatient diagnostic errors of 5.08%, or approximately 12 million US adults every year. Based upon previous work, we estimate that about half of these errors could potentially be harmful.

Conclusions

Our population-based estimate suggests that diagnostic errors affect at least 1 in 20 US adults. This foundational evidence should encourage policymakers, healthcare organisations and researchers to start measuring and reducing diagnostic errors.

Background

Diagnostic errors pose an important threat to healthcare quality and safety in outpatient settings.^{1 2} The frequency of outpatient diagnostic errors is unknown,³ due in part to varying definitions across studies and to the challenge of detecting these errors, which typically emerge across multiple episodes of care over time. A recent review by Graber discussed the use of several methods to study diagnostic errors, including autopsies, case reviews, surveys, incident reporting, standardised patients, second reviews and malpractice claims.³ However, most of these methods have limited use in determining the frequency of diagnostic error in outpatient medical practice.

Population-based estimates of diagnostic error could help prioritise ongoing patient safety efforts. We aimed to estimate the frequency of outpatient diagnostic errors in the US adult population by synthesising data from three previous studies of clinic-based populations that used conceptually similar definitions of diagnostic error.^4–6

Methods

Data sources included two previous studies that used electronic triggers, or algorithms, to detect unusual patterns of return visits (primary care study) or lack of follow-up of abnormal clinical findings for colorectal cancer (CRC) (colon cancer study), both suggestive of diagnostic errors.^{4 6} A third study examined consecutive cases of lung cancer in two institutions (lung cancer study).⁵ In all three studies, diagnostic errors were confirmed through chart review and defined as missed opportunities to make a timely or correct diagnosis based on available evidence. The criteria for diagnostic errors were comparable across the three studies and excluded atypical presentations and appropriate decisions to watch and wait.

The triggers in the primary care study⁶ were based on the occurrence of unexpected return visits and were therefore selective for misdiagnosis of acute conditions and exacerbations of chronic conditions. Trigger 1 was a primary care index visit followed by unplanned hospitalisation within 14 days; trigger 2 was a primary care index visit followed by one or more unscheduled visit(s) within 14 days. Control visits did not meet either criterion. We considered for inclusion all primary care visits (n=212 165 visits and 81 483 unique patients) in two large health systems between 1 October 2006 and 30 September 2007. For the record review, we reviewed all trigger 1 records from one of the sites (n=220 records) and randomly chose the rest from the second site, yielding a reviewed sample of 674 records out of the total of 1086 visits selected. Maintaining a record sampling ratio from each health system that was consistent with trigger 1, we reviewed 669 of 14 777 visits selected by trigger 2 and 614 of 193 810 control non-triggered visits. Errors in triggered and non-triggered (control) cases were judged to have occurred when adequate data to suggest the final, correct diagnosis existed at an earlier primary care visit, or if documented abnormal findings at the visit should have prompted additional evaluation that would have revealed the correct diagnosis. Determination of errors was made on retrospective review by two independent raters. In extrapolating error rates to all primary care patients at our study sites, we assumed that rates of error in randomly selected samples of triggered and non-triggered cases were applicable to the entire cohort of triggered and non-triggered records in each clinic population.

Although we presumed that the types of errors detected in the primary care study were likely to account for the majority of outpatient diagnostic errors, we also aimed to estimate the rate of errors related to initial diagnosis of less common but serious chronic conditions. Unlike misdiagnosis of acute conditions, these errors may not become apparent for weeks or months. As a proxy for such errors in outpatient care, we obtained frequency estimates for missed and delayed diagnosis of CRC and lung cancer, represented in the two cancer studies,^{4 5} respectively. These conditions were selected due to their relative frequency in several types of prior observational studies related to diagnostic errors.^{5 7–12} In both of our studies, diagnostic errors were characterised by lack of timely follow-up on specific ‘red flags’ or ‘alarm’ features^{5 10 13} of cancer in patients who were eventually diagnosed with either cancer. For instance, for CRC this was operationally defined as lack of follow-up 60 days after documentation of hematochezia (bright red blood per rectum), positive faecal occult blood test and iron deficiency anaemia. For lung cancer, we defined lack of follow-up as failure to initiate further investigation or an appropriate action within 7 days of a documented ‘red flag’ such as an abnormal chest X-ray. None of these data overlapped with the primary care study. We assumed that failure to follow-up on these ‘red flags’ accounted for all diagnostic errors associated with these types of cancer.

We extrapolated the frequency of diagnostic error obtained from our studies to the US adult population,¹⁴ using the primary care study as a proxy to estimate rates of diagnostic error for acute conditions (and exacerbations of existing conditions) and the two cancer studies to conservatively estimate rates of missed diagnosis of CRC and lung cancer (as proxies for other serious chronic conditions). Because of the heterogeneity of the conditions we included, we assumed that they represented the spectrum of diseases at risk for outpatient diagnostic errors.

Results

In the primary care study, samples of triggered visits and control visits were reviewed to determine the percentages of visits that contained errors. These percentages were extrapolated to the larger triggered and non-triggered samples. For trigger 1, 141 errors were found in 674 visits reviewed, yielding an error rate of 20.9%. Extrapolating to all 1086 trigger 1 visits yielded an estimate of 227.2 errors. For trigger 2, 36 errors were found in 669 visits reviewed, yielding an error rate of 5.4%. Extrapolating to all 14 777 trigger 2 visits yielded an estimate of 795.2 errors. Finally, for the control visits, 13 errors were found in 614 visits reviewed, yielding an error rate of 2.1%. Extrapolating to all 193 810 control visits yielded an estimate of 4103.5 errors. Thus, we estimated that 5126 errors would have occurred across the three groups. We then divided this figure by the number of unique primary care patients in the initial cohort (81 483) and arrived at an estimated error rate of 6.29%. Because approximately 80.5% of US adults seek outpatient care annually,¹⁵ the same rate when applied to all US adults gives an estimate of 5.06% (see figure 1).

Figure 1.

Schematic of synthesis of diagnostic error frequency data from three studies.

The colon cancer study estimated errors in CRC diagnosis in 26 out of 291 773 patients seen in outpatient care (0.009%). Extrapolating to the US adult population that seeks outpatient care, we estimated 0.007% of all adults would have errors related to CRC per year. In the lung cancer study, we found 127 errors in 587 cases of lung cancer, yielding a 21.64% error rate. Extrapolating to the 80.5% of the US adult population that seeks outpatient care and the 0.072% of the population that becomes newly diagnosed with lung cancer each year,¹⁶ we estimate 0.013% of all adults would have errors related to lung cancer per year.

Combining estimates from the three studies yields a rate of outpatient diagnostic errors of 5.08%, or approximately 12 million US adults every year. Based on our previous work,¹⁷ we estimated that about one-half of errors would have the potential to lead to severe harm.

Discussion

Although diagnostic errors are difficult to define and measure, there is mounting evidence of their importance and frequency across several types of conditions.^{3 7 11 18} Applying aggregate diagnostic error rates from three previous studies conducted by our group, we estimate that approximately 12 million adults in the USA could experience outpatient diagnostic errors each year. Our estimates are generally consistent with data from the general public about diagnostic errors. A recent telephone survey of a random probability sample of over 2000 US adults suggested that more than one in 10 respondents reported a ‘diagnostic mishap’.¹⁹ Similarly, a survey of 726 paediatricians revealed that more than half (54%) reported making a diagnostic error at least once or twice per month.²⁰ Thus, other recent data corroborate our finding that diagnostic errors are common.

Previous estimations of the frequency of diagnostic error largely relied on expert opinion. For example, Elstein estimated errors occur in about 10–15% of diagnoses.²¹ As noted by Graber, many methods to study diagnostic errors have limited use in calculating the frequency of the problem. For example, autopsy rates have declined,²² the number of self-reports remains low and malpractice claims data are prone to bias and are non-representative.^{11 20 23–25} On the other hand, population-based studies have relied on medical record reviews, which are generally a gold standard for determining diagnostic error.²⁶ Our electronic trigger approach facilitated selective record reviews, thus making the error determination process more efficient than either consecutive or random record reviews. Our estimate, based on a large sample size, is likely to be the most robust one thus far to address the frequency of diagnostic error in routine outpatient practice.

Although it is unknown how many patients will be harmed from diagnostic errors, our previous work¹⁷ suggests that about one-half of diagnostic errors have the potential to lead to severe harm. While this is only an estimate and does not imply all those affected will actually have harm, this risk potentially translates to about 6 million outpatients per year. Additionally, while the contribution of the two cancers to the overall estimation was small, we believed it was important to include them to show their relative contribution. This is because delayed cancer diagnosis is believed to be one of the most harmful and costly types of diagnostic error in the outpatient setting and its significance has become apparent not only in malpractice claims but also in retrospective studies of consecutive cancer cases, surveys and studies of failures to follow-up abnormal test results.^{5 7–12} In many of these studies, lung cancer and CRC are the most common cancers for which diagnosis is delayed.

Our estimates are limited by our assumptions of generalisability. To keep the definition of diagnostic error uniform across studies, we only used data from our own work. Other studies have used different methods and definitions of diagnostic errors and we believed that this would make our extrapolations weaker. Similar to other methods to study medical errors, our trigger was not sensitive enough to capture all types of diagnostic errors in outpatient settings. Our triggers did not adequately capture errors in diagnosis of uncommon diseases, some of which might evolve over a prolonged period of time in the outpatient setting. Our studies also might not have adequately addressed other types of common chronic conditions that could be vulnerable to diagnostic error.²⁷ For example, a recent study from the UK suggests that opportunities to diagnose chronic obstructive pulmonary disease at an earlier stage are being missed in the majority of cases.²⁸ While it is tempting to do so, we did not extrapolate our chronic condition proxy estimates to all types of cancer or other types of chronic diseases because it would not be possible to do this accurately. Lastly, chart reviews inherently would miss errors due to documentation-related issues and this false negative rate is largely unknown. All of these limitations thus suggest that we underestimated the frequency of diagnostic errors and frequency could be well over 5%.

In conclusion, we estimate the frequency of diagnostic error to be at least 5% in US outpatient adults, a number that entails a substantial patient safety risk. This population-based estimate should provide a foundation for policymakers, healthcare organisations and researchers to strengthen efforts to measure and reduce diagnostic errors.